VGGFace2: A dataset for recognising faces across pose and age

Qiong Cao, Li Shen, Weidi Xie, Omkar M. Parkhi and Andrew Zisserman
Visual Geometry Group, Department of Engineering Science, University of Oxford
{qiong,lishen,weidi,omkar,az}@robots.ox.ac.uk

Abstract

Abstract—In this paper, we introduce a new large-scale face dataset named VGGFace2. The dataset contains 3.31 million images of 9131 subjects, with an average of 362.6 images for each subject. Images are downloaded from Google Image Search and have large variations in pose, age, illumination, ethnicity and profession (e.g. actors, athletes, politicians).

The dataset was collected with three goals in mind: (i) to have both a large number of identities and also a large number of images for each identity; (ii) to cover a large range of pose, age and ethnicity; and (iii) to minimise the label noise. We describe how the dataset was collected, in particular the automated and manual filtering stages to ensure a high accuracy for the images of each identity.

To assess face recognition performance using the new dataset, we train ResNet-50 (with and without Squeeze-and-Excitation blocks) Convolutional Neural Networks on VGGFace2, on MS-Celeb-1M, and on their union, and show that training on VGGFace2 leads to improved recognition performance over pose and age. Finally, using the models trained on these datasets, we demonstrate state-of-the-art performance on the face recognition of IJB datasets, exceeding the previous state-of-the-art by a large margin. The dataset and models are publicly available$^{1}$.

_Keywords—_face dataset; face recognition; convolutional neural networks

$^{1}$http://www.robots.ox.ac.uk/~vgg/data/vgg_face2/

Executive Summary: The VGGFace2 dataset was created to address shortcomings in existing public face recognition collections, which often lack sufficient depth per identity, controlled variation in pose and age, or reliable labels. These gaps limit the ability of deep convolutional networks to generalize across real-world conditions such as viewpoint changes, aging, and diverse demographics, while also hindering fair comparison on challenging benchmarks.

The work set out to produce a high-quality training resource that balances scale, intra-subject diversity, and label accuracy, then to demonstrate its value by training and evaluating standard recognition models on established test sets.

Images were gathered through targeted Google Image Search queries for thousands of public figures, followed by automated face detection, classification-based outlier removal, duplicate elimination, and targeted manual review. The final collection contains 3.31 million images across 9,131 identities, with explicit pose and age template annotations provided for a held-out evaluation subset. ResNet-50 and Squeeze-and-Excitation ResNet-50 networks were trained from scratch or after pre-training on MS-Celeb-1M, and performance was measured on the authors’ own test split as well as the IJB-A, IJB-B, and IJB-C benchmarks.

Training on VGGFace2 produced markedly lower top-1 identification error (3.9 %) on the authors’ test set than training on VGGFace or MS-Celeb-1M. On the IJB-A benchmark the same models raised verification true-accept rate at a false-accept rate of 0.001 from roughly 0.85 to 0.90–0.92 and improved identification rates at low false-positive identification rates by 3–4 percentage points. Gains were even larger on the more recent IJB-B and IJB-C sets, where the best VGGFace2-trained SENet exceeded prior published results by substantial margins. Models also showed higher template similarity scores when matching across large pose or age differences, confirming improved robustness.

These results indicate that a dataset emphasizing both breadth and controlled intra-class variation can deliver stronger recognition performance than collections that prioritize only the number of identities. Organizations relying on face recognition for security, access control, or media search can therefore expect lower error rates and reduced sensitivity to viewpoint or age shifts by retraining on this resource.

The authors release both the dataset and trained models for public use. Further gains are likely from combining VGGFace2 with even larger breadth-focused collections and from continued architecture improvements; organizations should therefore consider a staged training approach—pre-train on the largest available set, then fine-tune on VGGFace2—while monitoring performance on their own demographic and imaging conditions.

The reported improvements rest on the IJB evaluation protocols and the authors’ internal test split; generalization to other domains or to video streams with different characteristics remains to be verified at scale.

1. Introduction

Section Summary: Recent efforts to build large face datasets for training deep neural networks have focused on either gathering many images of the same people or images across many different people, but have overlooked systematic variation in pose and age. To fill this gap, the authors introduce VGGFace2, a new public dataset containing more than 9,000 identities and 3 million images, collected via a pipeline that deliberately encourages diversity in pose, age, lighting, and ethnicity while using automated and manual checks to limit labeling mistakes. They also supply test annotations for pose and age performance and show that networks trained on the new data surpass prior results on major face-recognition benchmarks.

Concurrent with the rapid development of deep Convolutional Neural Networks (CNNs), there has been much recent effort in collecting large scale datasets to feed these data-hungry models. In general, recent datasets (see Table 1) have explored the importance of intra- and inter-class variations. The former focuses on depth (many images of one subject) and the latter on breadth (many subjects with limited images per subject). However, none of these datasets was specifically designed to explore pose and age variation. We address that here by designing a dataset generation pipeline to explicitly collect images with a wide range of pose, age, illumination and ethnicity variations of human faces.

We make the following four contributions: first, we have collected a new large scale dataset, VGGFace2, for public release. It includes over nine thousand identities with between 80 and 800 images for each identity, and more than 3M images in total; second, a dataset generation pipeline is proposed that encourages pose and age diversity for each subject, and also involves multiple stages of automatic and manual filtering in order to minimise label noise; third, we provide template annotation for the test set to explicitly explore pose and age recognition performance; and, finally, we show that training deep CNNs on the new dataset substantially exceeds the state-of-the-art performance on the IJB benchmark datasets [1, 2, 3]. In particular, we experiment with the recent Squeeze and Excitation network [4], and also investigate the benefits of first pre-training on a dataset with breadth (MS-Celeb-1M [5]) and then fine tuning on VGGFace2.

The rest of the paper is organised as follows: We review previous dataset in Section 2, and give a summary of existing public dataset in Table 1. Section 3 gives an overview of the new dataset, and describes the template annotation for recognition over pose and age. Section 4 describes the dataset collection process. Section 5 reports state-of-the-art performance of several different architectures on the IJB-A [1], IJB-B [2] and IJB-C [3] benchmarks.

2. Dataset Review

Section Summary: The section reviews the main public "in the wild" face recognition datasets released from 2007 onward, including LFW, CelebFaces+, CASIA-WebFace, VGGFace, MegaFace, Ms-Celeb-1M, the IJB series, and video-focused sets such as Youtube Face. It describes their scale in terms of images and identities, notes which ones were intended for training versus evaluation, and highlights common limitations such as label noise, restricted variation per person, or limited pose and age diversity. The review also briefly mentions much larger private datasets held by companies like Facebook and Google.

::: {caption="Table 1: Two example templates of front-to-profile matching. The three entries in the 'per subject' column are the minimum/average/maximum per subject."}

:::

In this section we briefly review the principal "in the wild" datasets that have appeared recently. In 2007, the Labelled Faces in the Wild (LFW) dataset [6] was released, containing $5,749$ identities with $13,000$ images.

The CelebFaces+ dataset [8] was released in 2014, with $202,599$ images of $10,177$ celebrities. The CASIA-WebFace dataset [9] released the same year that has $494,414$ images of $10,575$ people. The VGGFace dataset [10] released in 2015 has $2.6$ million images covering $2,622$ people, making it amongst the largest publicly available datasets. The curated version, where label noise is removed by human annotators, has $800,000$ images with approximately $305$ images per identity. Both the CASIA-WebFace and VGGFace datasets were released for training purposes only.

MegaFace dataset [11] was released in 2016 to evaluate face recognition methods with up to a million distractors in the gallery image set. It contains $4.7$ million images of $672,057$ identities as the training set. However, an average of only $7$ images per identity makes it restricted in its per identity face variation. In order to study the effect of pose and age variations in recognising faces, the MegaFace challenge [11] uses the subsets of FaceScrub [14] containing $4,000$ images from $80$ identities and FG-NET [15] containing $975$ images from $82$ identities for evaluation.

Microsoft released the large Ms-Celeb-1M dataset [5] in 2016 with $10$ million images from $100$k celebrities for training and testing. This is a very useful dataset, and we employ it for pre-training in this paper. However, it has two limitations: (i) while it has the largest number of training images, the intra-identity variation is somewhat restricted due to an average of $81$ images per person; (ii) images in the training set were directly retrieved from a search engine without manual filtering, and consequently there is label noise. The IARPA Janus Benchmark-A (IJB-A) [1], Benchmark-B (IJB-B) [2] and Benchmark-C (IJB-C) [3] datasets were released as evaluation benchmarks (only test) for face detection, recognition and clustering in images and videos.

Unlike the above datasets which are geared towards image-based face recognition, the Youtube Face (YTF) [7] and UMDFaces-Videos [13] datasets aim to recognise faces in unconstrained videos. YTF contains $1,595$ identities and $3,425$ videos, whilst UMDFaces-Videos is larger with $3,107$ identities and $22,075$ videos (the identities are a subset of those in UMDFaces [12]).

Apart form these public datasets, Facebook and Google have large in-house datasets. For instance, Facebook [16] trained a face identification model using $500$ million images of over $10$ million subjects. The face recognition model by Google [17] was trained using $200$ million images of $8$ million identities.

3. An Overview of The VGGFace2

Section Summary: The VGGFace2 dataset holds 3.31 million images of 9,131 celebrities drawn from web searches, showing large differences in pose, age, lighting, background, and ethnicity while remaining roughly balanced between men and women. Each identity comes with verified face boxes, estimated head orientation, and apparent age, and the collection is divided into a large training split and a smaller evaluation split. Separate template sets are also supplied so researchers can test matching performance specifically across frontal-to-profile views or across younger versus older appearances.

3.1 Dataset Statistics

The VGGFace2 dataset contains $3.31$ million images from $9131$ celebrities spanning a wide range of ethnicities, e.g. it includes more Chinese and Indian faces than VGGFace (though, the ethnic balance is still limited by the distribution of celebrities and public figures), and professions (e.g. politicians and athletes). The Images were downloaded from Google Image Search and show large variations in pose, age, lighting and background. The dataset is approximately gender-balanced, with $59.3$% males, varying between $80$ and $843$ images for each identity, with $362.6$ images on average. It includes human verified bounding boxes around faces, and five fiducial keypoints predicted by the model of [18]. In addition, pose (yaw, pitch and roll) and apparent age information are estimated by our pre-trained pose and age classifiers (Pose, age statistics and example images are shown in Figure 1).

The dataset is divided into two splits: one for training having $8631$ classes, and one for evaluation (test) with $500$ classes.

: Table 2: Two example templates of front-to-profile matching.

:::: {.figure cols="2"} {width=100%}

{width=100%}

Figure 2: Two example templates of front-to-profile matching. ::::

3.2 Pose and Age Annotations

The VGGFace2 provides annotation to enable evaluation on two scenarios: face matching across different poses, and face matching across different ages.

Pose templates. A template here consists of five faces from the same subject with a consistent pose. This pose can be frontal, three-quarter or profile view. For a subset of 300 subjects of the evaluation set, two templates ($5$ images per template) are provided for each pose view. Consequently there are $1.8$K templates with $9$K images in total. An examples is shown in Figure 2 (left).

Age templates. A template here consists of five faces from the same subject with either an apparent age below 34 (deemed young), or 34 or above (deemed mature). These are provided for a subset of $100$ subjects from the evaluation set with two templates for each age period, therefore, there are $400$ templates with a total of $2$K images. Examples are show in Figure 2 (right).

4. DATASET COLLECTION

Section Summary: The dataset collection begins by assembling a list of roughly 500,000 public figures from Freebase, then using Google Image Search and human review to narrow it to about 9,200 identities that each have many distinct photos. For each person, around 1,400 images are downloaded, faces are detected and aligned, and a series of automatic steps remove duplicates, low-quality matches, and obvious outliers through classifiers and clustering. Overlapping or noisy identities are further identified and cleaned with a mix of model-based confusion checks and targeted manual review, ultimately yielding a final set of 9,131 people with high label purity.

In this section, we describe the dataset collection process, including: how a list of candidate identities was obtained; how candidate images were collected; and, how the dataset was cleaned up both automatically and manually. The process is summarised in Table 2.

4.1 Stage 1: Obtaining and selecting a name list

We use a similar strategy to that proposed by [10]. The first stage is to find as many subjects as possible that have a sufficiently distinct set of images available, for example, celebrities and public figures (e.g. actors, politicians and athletes). An initial list of $500$k public figures is obtained from the Freebase knowledge graph [19].

An annotator team is then used to remove identities from the candidate list that do not have sufficient distinct images. To this end, for each of the $500$K names, $100$ images are downloaded using Google Image Search and human annotators are instructed to retain subjects for which approximately $90$% or more of the $100$ images belong to a single identity. This removes candidates who do not have sufficient images or for which Google Image Search returns a mix of people for a single name. In this manner, we reduce the candidates to only $9244$ names. Attribute information such as ethnicity and kinship is obtained from DBPedia [20].

4.2 Stage 2: Obtaining images for each identity

We query in Google Image Search and download $1000$ images for each subject. To obtain images with large pose and age variations, we then append the keyword 'sideview' and 'very young' to each name and download $200$ images for each. This results in $1400$ images for each identity.

4.3 Stage 3: Face detection

Faces are detected using the model provided by [18]. We use the hyper-parameters recommended in that work to favor a good trade-off between precision and recall. The face bounding box is then extended by a factor of $0.3$ to include the whole head. Moreover, five facial landmarks are predicted by the same model.

4.4 Stage 4: Automatic filtering by classification

The aim of this stage is to remove outlier faces for each identity automatically. This is achieved by learning a classifier to identify the faces, and removing possible erroneous faces below a classification score. To this end, 1-vs-rest classifiers are trained to discriminate between the $9244$ subjects. Specifically, faces from the top $100$ retrieved images of each identity are used as positives, and the top $100$ of all other identities are used as negative for training. The face descriptor features are obtained from the VGGFace [10] model. Then, the scores (between $0$ and $1$) from the trained model is used to sort images for each subject from most likely to least likely. By manually checking through images from a random $500$ subjects, we choose a threshold of $0.5$ and remove any faces below this.

4.5 Stage 5: Near duplicate removal

The downloaded images also contain exact or near duplicates due to the same images being found at different internet locations, or images differing only slightly in colour balance or JPEG artifacts for example. To alleviate this, duplicate images are removed by clustering VLAD descriptors for all images remaining at stage $4$ and only retaining one image per cluster [21, 22].

4.6 Stage 6: Final automatic and manual filtering

At this point, two types of error may still remain: first, some classes still have outliers (i.e. images that do not belong to the person); and second, some classes contain a mixture of faces of more than one person, or they overlap with another class in the dataset. This stage addresses these two types of errors with a mix of manual and automated algorithms.

Detecting overlapped subjects. Subjects may overlap with other subjects. For instance, 'Will I Am' and 'William James Adams' in the candidate list refer to the same person. To detect confusions for each class, we randomly split the data for each class in half: half for training and the other for testing. Then, we train a ResNet-50 [23] and generate a confusion matrix by calculating top-1 error on the test samples. In this manner, we find $20$ subjects confused with others. In this stage, we removed $19$ noisy classes. In addition, we remove $94$ subjects with samples less than $80$ images, which results in a final list of $9131$ identities.

Removing outlier images for a subject. The aim of this filtering, which is partly manual, is to achieve a purity greater than $96$%. We found that for some subjects, images with very high classifier scores at stage $4$ can also be noisy. This happens when the downloaded images contain couples or band members who always appear together in public. In this case, the classifiers trained with these mixed examples at stage $4$ tend to fail.

We retrain the model based on the current dataset, and for each identity the classifier score is used to divide the images into $3$ sets: H (i.e. high score range [$1$, $0.95$]), I (i.e. intermediate score range ($0.95$, $0.8$]) and L (i.e. low score range ($0.8$, $0.5$]). Human annotators clean up the images for each subject based on their scores, and the actions they carry out depends on whether the set H is noisy or not. If the set (H) contains several different people (noise) in a single identity folder, then set I and L (which have lower confidence scores), will undoubtedly be noisy as well, so all three sets are cleaned manually. In contrast, if set H is clean, then only set L (the lowest scores which is supposed to be the most noisy set) is cleaned up. After this, a new model is trained on the cleaned set H and L, and set I (intermediate scores, noise level is also intermediate) is then cleaned by model prediction. This procedure achieves very low label noise without requiring manual checking of every image.

4.7 Pose and age annotations

We train two networks to obtain the pose and age information for the dataset. To obtain head pose (roll, pitch, yaw), a $5$-way classification ResNet-$50$ [23] is trained on the CASIA-WebFace dataset [9]. Then, this trained model is used to predict pose for all the images in the dataset.

Similarly, to estimate the apparent age, a $8$-way classification ResNet-$50$ [23] is trained on IMDB-WIKI - $500$k$+$ dataset [24]. Ages of faces are then predicted by this model.

5. EXPERIMENTS

Section Summary: The experiments evaluate the quality of the VGGFace2 dataset by training ResNet-50 and SENet models on it under several setups, including training from scratch or fine-tuning after pre-training on MS-Celeb-1M, and comparing results to models trained on VGGFace or MS-Celeb-1M. Performance is measured via face identification on the VGGFace2 test set plus verification on the IJB benchmarks, using averaged face descriptors and cosine similarity, with care taken to avoid subject overlap between training and test data. Results indicate that VGGFace2 produces lower error rates than the other datasets, confirming its greater diversity and lower label noise, especially for variations in pose and age.

In this section, we evaluate the quality of the VGGFace2 dataset by conducting a number of baseline experiments. We report the results on the VGGFace2 test set, and evaluate on the public benchmarks IJB datasets [1, 2, 3]. The subjects in our training dataset are disjoint with the ones in benchmark datasets. We also remove the overlap between MS-Celeb-1M and the two benchmarks when training the networks.

5.1 Experimental setup

Architecture. ResNet-50 [23] and SE-ResNet-50 [4] (SENet for short) are used as the backbone architectures for the comparison amongst training datasets. The Squeeze-and-Excitation (SE) blocks [4] adaptively recalibrate channel-wise feature responses by explicitly modelling channel relationships. They can be integrated with modern architectures, such as ResNet, and improve its representational power. This has been demonstrated for object and scene classification, with a Squeeze-and-Excitation network winning the ILSVRC 2017 classification competition.

The following experiments are developed under four settings: (a) networks are learned from scratch on VGGFace [10] (VF for short); (b) networks are learned from scratch on MS-Celeb-1M (MS1M for short) [5]; (c) networks are learned from scratch on VGGFace2 (VF2 for short); and, (d) networks are first pre-trained on MS1M, and then fine-tuned on VGGFace2 (VF2_ft for short).

Similarity computation. In all the experiments (i.e. for both verification and identification), we need to compute the similarity between subject templates. A template is represented by a single vector computed by aggregating the face descriptors of each face in the template set. In Section 5.2, the template vector is obtained by averaging the face descriptors of the images and SVM classifiers are used for identification. In Section 5.3 and Section 5.4 for IJB-A and IJB-B, where the template may contain both still images and video frames, we first compute the media vector (i.e. from images or video frames) by averaging the face descriptors in that media. A template vector is then generated by averaging the media vectors in that template, which is then L$2$ normalised. Cosine similarity is used to represent the similarity between two templates.

A face descriptor is obtained from the trained networks as follows: first the extended bounding box of the face is resized so that the shorter side is $256$ pixels; then the centre $224\times 224$ crop of the face image is used as input to the network. The face descriptor is extracted from from the layer adjacent to the classifier layer. This leads to a $2048$ dimensional descriptor, which is then L$2$ normalised.

Training implementation details. All the networks are trained for classification using the soft-max loss function. During training, the extended bounding box of the face is resized so that the shorter side is $256$ pixels, then a $224\times 224$ pixels region is randomly cropped from each sample. The mean value of each channel is subtracted for each pixel.

Monochrome augmentation is used with a probability of $20%$ to reduce the over-fitting on colour images. Stochastic gradient descent is used with mini-batches of size $256$, with a balancing-sampling strategy for each mini-batch due to the unbalanced training distributions. The initial learning rate is $0.1$ for the models trained from scratch, and this is decreased twice with a factor of $10$ when errors plateau. The weights of the models are initialised as described in [23]. The learning rate for model fine-tuning starts from $0.005$ and decreases to $0.001$.

5.2 Experiments on the new dataset

In this section, we evaluate ResNet-50 trained from scratch on the three datasets as described in the Section 5.1, and VGGFace2 test set. We test identification performance and also similarity over pose and age, and validate the capability of VGGFace2 to tackle pose and age variations.

Face identification. This scenario aims to predict, for a given test image, whose face it is. Specifically, for each of the $500$ subjects in the evaluation set, $50$ images are randomly chosen as the testing split and the remaining images are used as the training split. This training split is used to learn 1-vs-rest SVM classifiers for each subject. A top-1 classification error is then used to evaluate the performance of these classifiers on the test images. As shown in Table 3, there is a significant improvement for the model trained on VGGFace2 rather than on VGGFace. This demonstrates the benefit of increasing data variation (e.g, subject number, pose and age variations) in the VGGFace2 training dataset. More importantly, models trained on VGGFace2 also achieve better result than that on MS1M even though it has tenfold more subjects and threefold more images, demonstrating the good quality of VGGFace2. In particular, the very low top-1 error of VGGFace2 provides evidence that there is very little label noise in the dataset – which is one of our design goals.

: Table 3: Identification performance (top-1 classification error) on the VGGFace2 test set for ResNet models trained on different datasets. A lower value is better.

::: {caption="Table 4: Two example templates of front-to-profile matching. Similarity scores are evaluated across pose templates. A higher value is better."}

:::

:::: {.figure cols="2"}

Figure 4: Two example templates of front-to-profile matching. Left: the similarity scores produced by VGGFace, MS1M, VGGFace2 are $0.41$, $0.35$ and $0.59$, respectively; Right: the scores are $0.41$, $0.31$ and $0.57$, respectively. ::::

Probing across pose. This test aims to assess how well templates match across three pose views: front, three-quarter and profile views. As described in Section 3.2, $300$ subjects in the evaluation set are annotated with pose templates, and there are six templates for each subject: two each for front, three-quarter view and profile views.

These six templates are divided into two sets, one pose for each set, and a $3 \times 3$ similarity matrix is constructed between the two sets. Figure 4 visualises two example of these cosine similarity scores for front-to-profile templates.

Table 4 compares the similarity matrix averaged over the $300$ subjects. We can observe that (i) all the three models perform better when matching similar poses, i.e., front-to-front, three-quarter-to-three-quarter and profile-to-profile; and (ii) the performance drops when probing for different poses, e.g., front-to-three-quarter and front-to-profile, showing that recognition across poses is a much harder problem. Figure 3 shows histograms of similarity scores. It is evident that the mass of the VGGFace2 trained model is to the right of the MS1M and VGGFace trained models. This clearly demonstrates the benefit of training on a dataset with larger pose variation.

::: {caption="Table 5: Face probing across ages. Similarity scores are evaluated across age templates. A higher value is better."}

:::

:::: {.figure cols="2"}

Figure 6: Two example templates of young-to-mature matching. Left: the similarity scores produced by VGGFace, MS1M, VGGFace2 are $0.42$, $0.30$ and $0.58$, respectively; Right: the scores are $0.43$, $0.41$ and $0.73$, respectively. ::::

Probing across age. This test aims to assess how well templates match across age, for two ages ranges: young and mature ages. As described in Section 3.2, $100$ subjects in the evaluation set are annotated with age templates, and there are four templates for each subject: two each for young and mature faces.

For each subject a $2 \times 2$ similarity matrix is computed, where an element is the cosine similarity between two templates. Figure 6 shows two examples of the young-to-mature templates, and their similarity scores.

Table 5 compares the similarity matrix averaged over the $100$ subjects as the model changes. For all the three models, there is always a big drop in performance when matching across young and mature faces, which reveals that young-to-mature matching is substantially more challenging than young-to-young and mature-to-mature. Moreover, young-to-young matching is more difficult than mature-to-mature matching. Figure 5 illustrates the histograms of the young-to-mature template similarity scores.

Discussion. In the evaluation of pose and age protocols, models trained on VGGFace2 always achieve the highest similarity scores, and MS1M dataset the lowest. This can be explained by the fact that the MS1M dataset is designed to focus more on inter-class diversities, and this harms the matching performance across different pose and age, illustrating the value of VGGFace2 in having more intra-class diversities that cover large variations in pose and age.

Figure 7 shows the top $3$ and bottom $3$ front-to-profile template matches sorted using the similarity scores produced by the ResNet model trained on VGGFace2. We can observe that the model gives high scores to front-to-profile templates where there is little variation beyond pose; while it gives lower scores to templates where many other variations exist, such as expression and resolution.

Figure 8 shows the top $3$ and bottom $3$ young-to-mature template matches sorted using the similarity scores produced by the ResNet model trained on VGGFace2. It can be seen that the model gives low scores to young-to-mature templates where variations such as pose and occlusion exist.

5.3 Experiments on IJB-A

::: {caption="Table 6: Performance evaluation on the IJB-A dataset. A higher value is better. The values with ${†}$ are read from [13]."}

:::

In this section, we compare the performance of the models trained on the different datasets on the public IARPA Janus Benchmark A (IJB-A dataset) [1].

The IJB-A dataset contains $5712$ images and $2085$ videos from $500$ subjects, with an average of 11.4 images and 4.2 videos per subject. All images and videos are captured from unconstrained environment and show large variations in expression and image qualities. As a pre-processing, we detect the faces using MTCNN [18] to keep the cropping consistent between training and evaluation.

IJB-A provides ten-split evaluations with two standard protocols, namely, 1:1 face verification and 1:N face identification, where we directly extract the features from the models for the test sets and use cosine similarity score. For verification, the performance is reported using the true accept rates (TAR) vs. false positive rates (FAR) (i.e. receiver operating characteristics (ROC) curve). For identification, the performance is reported using the true positive identification rate (TPIR) vs. false positive identification rate (FPIR) (equivalent to a decision error trade-off (DET) curve) and the Rank-N (i.e. the cumulative match characteristic (CMC) curve). Table 6 and Figure 9 presents the comparison results.

The effect of training set. We first investigate the effect of different training sets based on the same architecture ResNet-50 (Table 6), and start with networks trained from scratch. we can observe that the model trained on VGGFace2 outperforms the one trained on VGGFace by a large margin, even though VGGFace has a similar scale (2.6M images) it has fewer identities and pose/age variations (and more label noise). Moreover, the model of VGGFace2 is significantly superior to the one of MS1M which has $10$ times subjects over our dataset. Specially, it achieve $\sim 4.4%$ improvement over MS1M on FAR=$0.001$ for verification, $\sim 3.7%$ on FPIR=$0.01$ and $\sim 1.5%$ on Rank-$1$ for identification.

When comparing with the results of existing works, the model trained on VGGFace2 surpasses previously reported results on all metrics (best to our knowledge, reported on IJB-A 1:1 verification and 1:N identification protocols), which further demonstrate the advantage of the VGGFace2 dataset. In addition, the generalisation power can be further improved by first training with MS1M and then fine-tuning with VGGFace2 (i.e. "VGGFace2_ft"), however, the difference is only $0.908$ vs. $0.895$.

Many existing datasets are constructed by following the assumption of the superiority of wider dataset (more identities) [9, 5, 11], where the huge number of subjects would increase the difficulty of model training. In contrast, VGGFace2 takes both aspects of breath (subject number) and depth (sample number per subject) into account, guaranteeing rich intra-variation and inter-diversity.

The effect of architectures. We next investigate the effect of architectures trained on VGGFace2 (Table 6). The comparison between ResNet-50 and SENet both learned from scratch reveals that SENet has a consistently superior performance on both verification and identification. More importantly, SENet trained from scratch achieves comparable results to the fine-turned ResNet-50 (i.e. first pre-trained on the MS1M dataset), demonstrating that the diversity of our dataset can be further exploited by an advanced network. In addition, the performance of SENet can be further improved by training on the two datasets VGGFace2 and MS1M, exploiting the different advantages that each offer.

5.4 Experiments on IJB-B

::: {caption="Table 7: Performance evaluation on the IJB-B dataset. A higher value is better. The results of [2] are read from the curves reported in the paper. Note, [2] has a different evaluation for the verification protocol where pairs generated from different galleries are evaluated separately and averaged to get the final results."}

:::

The IJB-B dataset is an extension of IJB-A, having $1,845$ subjects with $21.8$K still images (including $11, 754$ face and $10,044$ non-face) and $55$K frames from $7,011$ videos. We evaluate the models on the standard 1:1 verification protocol (matching between the Mixed Media probes and two galleries) and 1:N identification protocol (1:N Mixed Media probes across two galleries).

We observe a similar behaviour to that of the IJB-A evaluation. For the comparison between different training sets (Table 7 and Figure 10), the models trained on VGGFace2 significantly surpass the ones trained on MS1M, and the performance can be further improved by integrating the advantages of the two datasets. In addition, SENet's superiority over ResNet-50 is evident in both verification and identification with the two training settings (i.e. trained from scratch and fine-tuned). Moreover, we also compare to the results reported by others on the benchmark [2] (as shown in Table 7), and there is a considerable improvement over their performance for all measures.

5.5 Experiments on IJB-C

::: {caption="Table 8: Performance evaluation on the IJB-C dataset. A higher value is better. The results of [3] are read from the curves reported in the paper. Note, [3] has a different evaluation for the verification protocol where pairs generated from different galleries are evaluated separately and averaged to get the final results."}

:::

The IJBC dataset is a further extension of IJB-B, including $3531$ subjects with $31.3$K still images and $117.5$K frames from $11,779$ videos. We evaluate the models on the standard 1:1 verification protocol and 1:N identification protocol. Results are shown in Table 8 and Figure 11. Compared to the results reported in [3], there is a considerable improvement for all measures.

6. Conclusion

Section Summary: The researchers developed a method to gather a large and diverse collection of face images called VGGFace2, which includes many different poses and ages. They trained advanced neural network models on this data and showed that these models perform better than previous ones on standard face recognition tests. Both the dataset and the trained models are freely available online for others to use.

In this work, we have proposed a pipeline for collecting a high-quality dataset, VGGFace2, with a wide range of pose and age. Furthermore, we demonstrate that deep models (ResNet-50 and SENet) trained on VGGFace2, achieve state-of-the-art performance on the IJB-A, IJB-B and IJB-C benchmarks. The dataset and models are available at [https://www.robots.ox.ac.uk/ vgg/data/vgg_face2/](https://www.robots.ox.ac.uk/ vgg/data/vgg_face2/).

Acknowledgment

Section Summary: The authors thank the companies Elancer and Momenta for helping prepare the dataset used in the research. The project received funding from the Office of the Director of National Intelligence and the Intelligence Advanced Research Projects Activity under a government contract. The text also notes that the views expressed belong only to the authors, not to the funding agencies or the U.S. government, which is allowed to reproduce the work for its own purposes.

We would like to thank Elancer and Momenta for their part in preparing the dataset[^2]. This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via contract number 2014-14071600010. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purpose notwithstanding any copyright annotation thereon.

[^2]: http://elancerits.com/ https://momenta.ai/

References

Section Summary: This section compiles a list of academic papers, technical reports, and online resources primarily focused on face recognition research. It covers major benchmarks and datasets designed to test systems under real-world conditions, along with deep learning approaches such as neural networks for feature extraction and verification. Several entries also address supporting techniques like image descriptors, domain adaptation, and age estimation from facial images.

[1] B. F. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. Allen, P. Grother, A. Mah, and A. K. Jain. Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a. In CVPR, pages 1931–1939, 2015.

[2] C. Whitelam, E. Taborsky, A. Blanton, B. Maze, J. Adams, T. Miller, N. Kalka, A. K. Jain, J. A. Duncan, K. Allen, et al. Iarpa janus benchmark-b face dataset. In CVPR Workshop on Biometrics, 2017.

[3] B. Maze, J. Adams, J. A. Duncan, N. Kalka, T. Miller, C. Otto, A. K. Jain, W. T. Niggel, J. Anderson, J. Cheney, and P. Grother. \textIARPA janus benchmark-c: Face dataset and protocol. In 11th IAPR International Conference on Biometrics, 2018.

[4] J. Hu, L. Shen, and G. Sun. Squeeze-and-Excitation networks. arXiv preprint arXiv:1709.01507, 2017.

[5] Y. Guo, L. Zhang, Y. Hu, X. He, and J. Gao. Ms-celeb-1m: A dataset and benchmark for large-scale face recognition. arXiv preprint arXiv:1607.08221, 2016.

[6] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. Technical report, 2007.

[7] L. Wolf, T. Hassner, and I. Maoz. Face recognition in unconstrained videos with matched background similarity. In CVPR, pages 529–534. IEEE, 2011.

[8] Y. Sun, X. Wang, and X. Tang. Deep learning face representation from predicting 10,000 classes. In CVPR, pages 1891–1898, 2014.

[9] D. Yi, Z. Lei, S. Liao, and S. Z. Li. Learning face representation from scratch. arXiv preprint arXiv:1411.7923, 2014.

[10] O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. In Proc. BMVC., 2015.

[11] I. Kemelmacher-Shlizerman, S. M. Seitz, D. Miller, and E. Brossard. The megaface benchmark: 1 million faces for recognition at scale. In CVPR, pages 4873–4882, 2016.

[12] A. Bansal, A. Nanduri, C. Castillo, R. Ranjan, and R. Chellappa. Umdfaces: An annotated face dataset for training deep networks. arXiv preprint arXiv:1611.01484, 2016.

[13] A. Bansal, C. Castillo, R. Ranjan, and R. Chellappa. The do's and don'ts for cnn-based face verification. arXiv preprint arXiv:1705.07426, 2017.

[14] H.-W. Ng and S. Winkler. A data-driven approach to cleaning large face datasets. In ICIP, pages 343–347. IEEE, 2014.

[15] G. Panis and A. Lanitis. An overview of research activities in facial age estimation using the fg-net aging database. In ECCV, pages 737–750. Springer, 2014.

[16] Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Web-scale training for face identification. In CVPR, pages 2746–2754, 2015.

[17] F. Schroff, D. Kalenichenko, and J. Philbin. Facenet: A unified embedding for face recognition and clustering. In CVPR, pages 815–823, 2015.

[18] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 23(10):1499–1503, 2016.

[19] Freebase. http://www.freebase.com/.

[20] Dbpedia. http://wiki.dbpedia.org/.

[21] R. Arandjelović and A. Zisserman. All about VLAD. In Proc. CVPR, 2013.

[22] H. Jegou, F. Perronnin, M. Douze, J. Sánchez, P. Perez, and C. Schmid. Aggregating local image descriptors into compact codes. TPAMI, 34(9):1704–1716, 2012.

[23] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, pages 770–778, 2016.

[24] R. Rothe, R. Timofte, and L. Van Gool. Dex: Deep expectation of apparent age from a single image. In CVPR Workshops, pages 10–15, 2015.

[25] N. Crosswhite, J. Byrne, C. Stauffer, O. Parkhi, Q. Cao, and A. Zisserman. Template adaptation for face verification and identification. In Automatic Face & Gesture Recognition (FG), pages 1–8. IEEE, 2017.

[26] K. Sohn, S. Liu, G. Zhong, X. Yu, M.-H. Yang, and M. Chandraker. Unsupervised domain adaptation for face recognition in unlabeled videos. arXiv preprint arXiv:1708.02191, 2017.

[27] J. Yang, P. Ren, D. Zhang, D. Chen, F. Wen, H. Li, and G. Hua. Neural aggregation network for video face recognition. In CVPR, pages 4362–4371, 2017.