This website offers images/video files labeled according to their visual features. The novelty is systematized and easily searchable database that covers many aspects of image labeling from data collection to applications. Such labeled data can be used to train recognition algorithms, estimate statistical properties of images, or just explore and play with image processing. We provide many different training sets that include such features as lines, regions, objects, motion patterns etc.

Classes assigned to those features include both discrete and continues labels. The examples of discrete labels/classes are ground versus figure and close vs. far objects; the example of continues labels are 3D range or motion direction. We try to organize and classify formats for input data and labels, as well as categorize different frameworks for statistical summaries, post-processing, and inference. This would make search of databases easier and applications more consistent.

In providing this information we encourage variability of approaches, free access as well as free contributions to this website. We emphasize public benefits of promoting convenient standards for image/video labeling, building a strong community of enthusiasts, and making image processing more accessible and efficient.

Labeling is an important part of our everyday thinking and understanding things via classification. It encompasses numerous aspects of our life from advertisement to family relationships. Obviously not everything is black and white but that’s why we might have more than two types of labels (classes). In general, picking up the number of classes is a compromise between accuracy and simplicity. It helps to reduce our cognitive load while paying attention to the classes of features that are important to us.

While a number of classes is one aspect of labeling a specific set of features or what to label is another important aspect. Applied to images, two extreme labeling schemas would be assigning a single label to the whole image (e.g. color vs. black-and-white images) or a one label per every single pixel (e.g. background vs. foreground pixels). In most cases, however, we want to label a visual feature that belongs to a level somewhere in between pixels and images, for example, an image part or even a simple contour line. Thus we first have to extract those features via some image processing.

One way to handle (standardize) a processing part is to supply image files with axillary files that specify what features to label. Below is a simple example of input files - an original image with interlaced blue lines (left column) and output files where each line (feature) is assigned a class. In the labeled image a red line stands for foreground while a green one denotes background. This assignment is recorded in the auxiliary file by adding 0 or 1 label to the rows that represent lines.

Input	Output
10, 10, 200, 20 202, 27, 300, 29	10, 10, 200, 20, 1 202, 27, 300, 29, 0	Table 1. Input and output of labeling process. Note that image doesn't change but the feature file is appended by a class label at the end of each row.

At this point you might have a question about how to interpret line coordinates in text files above and how coordinate axis are directed with respect to an image. Thus a separate file that provides this info is needed (e.g. row consists of x1, y1, x2, y2 coordinates of line endpoints; x axis points to the right and y axis points down, origin is an upper left corner). Having text files with features, formats and corresponding labels is a good way to keep track of specific image information. Such files are easy to read and interpret. There are many other formats for labels that include XML (e.g. with behaviors in video frames), image maps (e.g. with segmentation), etc.

However, in order to add more images or update old ones one has to run some programming code and extracts features again. Depending on how close this code corresponds to the original one a set of features and corresponding labels can be quite different from the previous one. Thus a complete data set should include a link to the code used in feature extraction or at least a raw algorithm that can be implemented in some programing language. Whenever this information is absent we supply references to image processing methods that can be used to reproduce a feature extraction process. To summarize, the ideal input should include images/video data, feature files (and/or code for their extraction), and description of feature formats.

So far we talked about feature formats and labels, but the question is really how one can use those labels for, say, recognition of new images. Potential applications along with label statistics represent an output of a good comprehensive data set. To give an example of application, consider a bar plot that reflects a number of occurrences of a certain feature that falls in a specific range. Another name for such a graph is a histogram. Let's build hypothetical histograms for the height of a person where we will put all possible heights in 5 intervals. Depending on a person's gender, the distribution of bars can be quite different:

Class 1	Class 2
		Table 2. Hypothetical histograms of human height. The distribution of heights (feature) is indicative of gender (class).

This represent yet another (statistical) aspect of labeling that typically done during post-processing of labeled files. Now to put it all together, just look at the two histograms above and try to figure out a gender of a person that has height 6.0. You might come to a conclusion that it is rather be a man (higher number of occurrences for 6.0) than a woman. Well all this quite simple but it means you already classifying new data! Now imagine that you are using multiple visual features at multiple locations in the image along with powerful statistical methods. As a result, classification or recognition of new images can be quite accurate, provided your initial training images and selected features were representative of your classes. To summarize, the overall process of gathering and using labeled data includes following stages:

Stages of data set processing
1. Providing representative images and features you want to label
2. Deciding what classes you want to use and assigning them to chosen features
3. Statistical post-processing of labeled files and reasoning about new data

In order to classify and search labeled data sets we decided to impose a very simple subdivision of material. We want to know at least three things about a particular data set: (i) what images, features, classes, and formats it uses; (ii) what are the conditions under which data were recoded; for example, are there several images per each illumination l evel or viewpoint; (iii) what is the range of possible applications and how well they perform. Such subdivision makes a search convenient especially for those who wants to use labeled data. For example, in order to find a data set with desired features one might first search a first section on certain type of images and features; to narrow down the search and to see how well the data would generalize one may check invariant properties of the features and then consult a condition section that lists a range of image variations. If still not sure what data set to choose, one may look into application section and also read comments posted there by other researches.

Below is a table with links to image/video data sets that were painstakingly gathered by numerous groups of scientists and engineers. We also included some unlabeled data that were traditionally used in demos. In order to avoid broken links we try to maintain a copy of a data set but sometimes size restrictions or proprietary reasons force us to provide a link only.

Images unlabeled
Data set	Features/Classes	Conditions	Applications
"Standard" test images frequently used in literature: Lena, Camera man, etc., TIFF 512 x 512, 1.5Mb	N/A	single	many
Binary shapes, GIF approx. 300x300 3.5Mb	figure/ground	several shapes per object	???
Amsterdam Library of Object Images (ALOI) 110,250 images, high resolution	N/A	viewing angle, illumination angle, illumination color	many, PDF
4000 Natural images (Groningen) grey, calibrated, 1536x1024, some urban elements	N/A	single	ICA, etc., PDF

Images labeled
Data set	Features/Classes	Conditions	Applications
MNIST handwritten images 60,000 training, 10,000 testing scaled, grey	digits 0-9	handwritings of diferent people	handwritten digit recognition Link

Image sequences
Data set	Features/Classes	Conditions	Applications

Video unlabeled
Data set	Features/Classes	Conditions	Applications

Video labeled
Data set	Features/Classes	Conditions	Applications
CAVIAR video sequences of mall and public space behavior - 90K frames in 90 sequences, MPEG2	bounding box, behavior	single	many, PDF
Computer generated video (from Object Video)	3D target centroid, bounding box, segmentation	unlimited!	preliminary testing Link