Convolutional Neural Networks (CNNs) have become the main solution for image classification tasks in different applications. Although several CNN architectures are available, there is no best architecture regardless the problem at hand. The selection of the most suitable CNN architecture is usually performed by trial and error, which may take much time and computational resources. Meta-learning (MtL) is a framework developed in machine learning to perform algorithm selection based on the meta-features of each task being solved. Such meta-features are usually descriptive characteristics extracted from the training dataset available in the task at hand. Despite the increasing attention of MtL for algorithm selection, its success strongly depends on defining relevant meta-features to represent the classification tasks of interest. This paper proposes the ImageDataset2Vec method for extracting meta-features to describe image classification datasets. ImageDataset2Vec adopts a pretrained deep neural network to extract features from images datasets, embedding them in a single feature vector. The derived meta-features are employed by MtL to select CNN architectures for image classification. The proposed approach was evaluated for selecting among six CNN algorithms in 45 two-classes image datasets. The results showed that MtL using ImageDataset2Vec overcame different baseline methods, selecting the best possible CNN algorithm in 84.45% of the datasets. Furthermore, the proposal was statistically equivalent to the ground truth when the best CNN is recommended, i.e., when MtL does not select the best CNN, it selects a competitive algorithm. These results show that the proposal was able to extract representative features from image datasets.