Back

Learning Face Representation from Scratch

Dong YiZhen LeiShengcai LiaoStan Z. Li

article2014arXiv2,220 citations

Introduces CASIA-WebFace, a publicly accessible dataset of 500,000 images across 10,000 identities, alongside a deep neural network architecture that allows researchers to train competitive face recognition models without relying on proprietary industrial data.

Listen

Recent advances in automated face recognition have increasingly relied on deep learning models trained on massive volumes of data. While open-source machine learning frameworks are widely accessible, high-performing algorithms have depended heavily on private, proprietary image collections held by large technology firms. This lack of public data has prevented the broader research community from training competitive models and conducting fair, standardized benchmark comparisons.

The article addresses this gap by constructing a large-scale, publicly available face database and developing a deep neural network baseline to evaluate its effectiveness on standard industry benchmarks.

To build the dataset, called CASIA-WebFace, the researchers implemented a semi-automated collection pipeline using celebrity web profiles. They extracted facial images, applied a clustering algorithm to match faces with identity tags, filtered out duplicate identities present in standard evaluation benchmarks, and performed manual quality checks. The resulting collection contains 494,414 face images across 10,575 distinct individuals. Using this dataset, the researchers trained an 11-layer convolutional neural network incorporating modern deep learning design principles, including small filter sizes, joint identification-verification loss functions, and low-dimensional feature representations.

Testing showed that a single network trained on CASIA-WebFace achieved state-of-the-art accuracy. On the standard Labeled Faces in the Wild benchmark, the model achieved 97.73% accuracy, outperforming leading proprietary models that relied on multi-network ensembles. On the YouTube Faces video benchmark, the model achieved 92.24% accuracy, exceeding prior proprietary baselines. Furthermore, under the more rigorous BLUFR benchmark—which evaluates open-set identification at low false accept rates—the deep network achieved an 80.26% verification rate, roughly doubling the 41.66% rate of traditional shallow models.

These findings indicate that access to well-structured public data removes a major barrier to developing competitive face recognition systems. The results show that deep architectures can deliver strong generalization on both static web images and compressed video frames without requiring massive ensemble models or private enterprise datasets.

The article outlines several actionable next steps for future research, including expanding dataset size through commercial search engines, refining automated annotation tools, and exploring network architectures that maximize single-model efficiency. Researchers working on real-world surveillance systems should focus on addressing low false accept rate scenarios, where detection and identification rates remain relatively low.

Key limitations include minor label noise and missed detections inherent to semi-automated web scraping, as well as computational constraints that prevented testing deeper networks or 3D alignment. Nonetheless, the consistent performance gains across multiple benchmarks provide high confidence in the utility and quality of the dataset as an open research foundation.

Institute of Automation, Chinese Academy of Sciencesbig dataconvolutional neural networksfacial imagesFaceLFWFace Recognition
arXiv: 1411.7923
  • Paper: ImageNet Classification with Deep Convolutional Neural Networks, Alex Krizhevsky et al. (2012). Reading Krizhevsky et al.'s foundational work on deep convolutional networks for ImageNet classification provides essential background on the CNN architectures and regularization techniques that the source paper adapts for face representation.
  • Paper: ImageNet Large Scale Visual Recognition Challenge, Olga Russakovsky et al. (2014). Familiarity with the ImageNet challenge benchmark helps clarify the broader computer vision shift toward large-scale datasets and deep learning that directly motivated the creation of the CASIA-WebFace dataset.
Cover for Learning Face Representation from Scratch

Abstract

Pushing by big data and deep convolutional neural network (CNN), the performance of face recognition is becoming comparable to human. Using private large scale training datasets, several groups achieve very high performance on LFW, i.e., 97% to 99%. While there are many open source implementations of CNN, none of large scale face dataset is publicly available. The current situation in the field of face recognition is that data is more important than algorithm. To solve this problem, this paper proposes a semi-automatical way to collect face images from Internet and builds a large scale dataset containing about 10,000 subjects and 500,000 images, called CASIAWebFace. Based on the database, we use a 11-layer CNN to learn discriminative representation and obtain state-of-theart accuracy on LFW and YTF. The publication of CASIAWebFace will attract more research groups entering this field and accelerate the development of face recognition in the wild.

Topics

big dataconvolutional neural networksfacial imagesface representationsface featuresfrontal faces

Keywords

FaceLFWFace RecognitionDatasetRepresentationLearningDeepImagesFace RepresentationLarge

Table of Contents

  • 1 Introduction
  • 2 Related Work
  • 2.1 Face Dataset
  • 2.2 Face Representation Learning
  • 3 CASIA-WebFace Dataset
  • 3.1 Name and Image Collection
  • 3.2 Face Annotation
  • 3.3 Dataset Statistics
  • 4 Learning Deep Representation
  • 4.1 Convolutional Network
  • 4.2 Training Methodology
  • 5 Experiments
  • 5.1 Results on LFW
  • 5.1.1 Standard Protocol
  • 5.1.2 BLUFR Protocol
  • 5.2 Results on YTF
  • 6 Conclusion
  • References

Knowls

  1. Knowl 1 — Tag-Similarity Clustering Pipeline for Web Face Annotation

    algorithm

    Annotating identities from crawled web pages is challenging because photo galleries frequently contain multiple individuals. To resolve identities semi-automatically without computing an infeasibly large N×NN \times N sample similarity matrix, a five-step clustering method leverages existing face recognition feature templates and IMDb name tags to constrain the search space.

    Input: Web-crawled celebrity pages with names, main photos, and photo galleries containing name tags
    Output: Clustered face image dataset with unique identity labels
    1. Run multi-view face detection across all crawled images
    2. Extract a face feature template for every detected face using a pre-trained face recognition engine
    3. Initialize the seed identity template for each celebrity using the single face from their "main photo"
    4. For all gallery photos containing exactly 1 detected face, match to the corresponding celebrity to augment the celebrity's seed set
    5. For each remaining photo containing multiple faces in the gallery:
         Match each detected face to candidate identities whose names appear in the photo's name tags based on maximum feature similarity
    6. Crop detected faces and partition them into separate identity directories
    7. Perform manual inspection to eliminate incorrectly assigned face clusters
    8. Filter out any subject possessing fewer than 15 images
    9. Compute name edit distance against LFW to identify and remove all overlapping subjects (1,043 subjects removed)

    This pipeline produces a cleaned, public large-scale dataset (CASIA-WebFace) comprising 10,575 subjects and 494,414 face images with no subject identity overlap with the LFW evaluation benchmark.

  2. Knowl 2 — CASIA-WebFace Dataset Statistics and Multi-Dataset Comparison

    data/table

    The CASIA-WebFace dataset was created to provide the research community with a large-scale, publicly accessible training dataset for face recognition in the wild. Prior to this work, top-performing deep face recognition models were trained on private datasets (such as Facebook's SFC and CUHK's CelebFaces).

    Dataset #Subjects #Images Availability
    LFW 5,749 13,233 Public
    WDRef 2,995 99,773 Public (feature only)
    CelebFaces 10,177 202,599 Private
    SFC 4,030 4,400,000 Private
    CACD 2,000 163,446 Public (partial annotated)
    CASIA-WebFace 10,575 494,414 Public

    CASIA-WebFace is the largest publicly accessible raw face image dataset in the literature, surpassed in scale only by Facebook's private SFC dataset. All 1,043 subjects appearing in both IMDb and LFW were identified by name edit distance and removed, making CASIA-WebFace an independent training set suitable for standard LFW evaluations.

  3. Knowl 3 — 11-Layer Deep Convolutional Neural Network Architecture for Face Representation

    model/method

    The baseline face recognition network adopts an 11-layer architecture comprising 10 convolutional layers, 5 pooling layers, and 1 fully connected classification layer.

    The key architectural characteristics include:

    • Input: 100×100×1100 \times 100 \times 1 single-channel grayscale normalized face image.
    • Small Filter Kernels: All 10 convolutional layers use 3×33 \times 3 filters with a stride of 1 to approximate larger receptive fields with fewer parameters and increased non-linearity.
    • Pooling Layers: The first four pooling stages (Pool1 to Pool4) employ 2×22 \times 2 max pooling with a stride of 2. The final pooling stage (Pool5) applies 7×77 \times 7 average pooling with stride 1 over the 7×7×3207 \times 7 \times 320 output of Conv52, producing a compact 320-dimensional feature vector (1×1×3201 \times 1 \times 320).
    • Dense Representation Constraint: Rectified Linear Unit (ReLU) non-linearities are placed after all convolutional layers except Conv52. Conv52 omits ReLU because ReLU induces sparsity, whereas a dense, compact representation is required for the low-dimensional face representation extracted via average pooling.
    • Regularization and Classification: A dropout layer with a 40% drop probability is applied to the 320-dimensional Pool5 representation before connecting to the fully connected layer Fc6, which outputs 10,575 identity class logits.

  4. Knowl 4 — Layer-by-Layer Architectural Specifications of the Baseline Face CNN

    data/table

    The baseline convolutional neural network contains 11 parameterized layers (10 convolutional layers and 1 fully connected layer) totaling approximately 5.015 million parameters.

    Name Type Filter Size / Stride Output Size Depth #Params
    Conv11 convolution 3×33\times3 / 1 100×100×32100\times100\times32 1 0.28K
    Conv12 convolution 3×33\times3 / 1 100×100×64100\times100\times64 1 18K
    Pool1 max pooling 2×22\times2 / 2 50×50×6450\times50\times64 0 -
    Conv21 convolution 3×33\times3 / 1 50×50×6450\times50\times64 1 36K
    Conv22 convolution 3×33\times3 / 1 50×50×12850\times50\times128 1 72K
    Pool2 max pooling 2×22\times2 / 2 25×25×12825\times25\times128 0 -
    Conv31 convolution 3×33\times3 / 1 25×25×9625\times25\times96 1 108K
    Conv32 convolution 3×33\times3 / 1 25×25×19225\times25\times192 1 162K
    Pool3 max pooling 2×22\times2 / 2 13×13×19213\times13\times192 0 -
    Conv41 convolution 3×33\times3 / 1 13×13×12813\times13\times128 1 216K
    Conv42 convolution 3×33\times3 / 1 13×13×25613\times13\times256 1 288K
    Pool4 max pooling 2×22\times2 / 2 7×7×2567\times7\times256 0 -
    Conv51 convolution 3×33\times3 / 1 7×7×1607\times7\times160 1 360K
    Conv52 convolution 3×33\times3 / 1 7×7×3207\times7\times320 1 450K
    Pool5 avg pooling 7×77\times7 / 1 1×1×3201\times1\times320 0 -
    Dropout dropout (40%) - 1×1×3201\times1\times320 0 -
    Fc6 fully connection - 10575 1 3305K
    Cost1 softmax - 10575 0 -
    Cost2 contrastive - 1 0 -
    Total 11 5015K

    The 320-dimensional Pool5 activation serves as the learned face representation for metric learning and verification tasks.

  5. Knowl 5 — Multi-Task Identification-Verification Optimization and Preprocessing Pipeline

    model/method

    Face images are converted to grayscale and normalized to 100×100100 \times 100 pixels via 2D similarity transformation based on two facial landmarks chosen such that their inter-point distance is 25 pixels and relatively invariant to yaw pose variations. Horizontal mirroring doubles the 494,414 images in CASIA-WebFace to 986,912 training samples.

    The network is trained using a multi-task objective combining identification loss (Softmax on the 10,575-dimensional Fc6 output) and verification loss (Contrastive loss on the 320-dimensional Pool5 feature vector):

    L=LSoftmax+αLContrastive\mathcal{L} = \mathcal{L}_{\text{Softmax}} + \alpha \mathcal{L}_{\text{Contrastive}}

    Training configurations:

    • Pair Sampling: Positive and negative face pairs for the Contrastive loss are sampled online within each mini-batch.
    • Loss Weighting: Because Softmax converges faster than Contrastive loss, α\alpha is initialized at 3.2×10−43.2 \times 10^{-4} and gradually increased during training to 6.4×10−36.4 \times 10^{-3}.
    • Optimization: The learning rate starts at 10−210^{-2} and anneals down to 10−510^{-5}. Weight decay is set to 00 for all convolutional layers and 5×10−45 \times 10^{-4} for the fully connected classification layer Fc6.

  6. Knowl 6 — Face Verification Accuracy on the LFW View 2 Benchmark

    data/table

    The deep representation (DR) extracted from the Pool5 layer of the baseline network was evaluated on the standard Labeled Faces in the Wild (LFW) View 2 benchmark across 6,000 face pairs in 10 splits under different protocols and post-processing methods.

    Method #Net Accuracy ±\pm SE Protocol
    DeepFace 1 95.92±0.29%95.92 \pm 0.29\% unsupervised
    DeepFace 1 97.00±0.28%97.00 \pm 0.28\% restricted
    DeepFace 3 97.15±0.27%97.15 \pm 0.27\% restricted
    DeepFace 7 97.35±0.25%97.35 \pm 0.25\% unrestricted
    DeepID2 1 95.43%95.43\% unrestricted
    DeepID2 2 97.28%97.28\% unrestricted
    DeepID2 4 97.75%97.75\% unrestricted
    DeepID2 25 98.97%98.97\% unrestricted
    Ours A 1 96.13±0.30%96.13 \pm 0.30\% unsupervised
    Ours B 1 96.30±0.35%96.30 \pm 0.35\% unsupervised
    Ours C 1 97.30±0.31%97.30 \pm 0.31\% unsupervised
    Ours D 1 96.33±0.42%96.33 \pm 0.42\% unsupervised
    Ours E 1 97.73±0.31%97.73 \pm 0.31\% unrestricted

    Experimental configurations for the proposed model:

    • Ours A: DR + Cosine similarity.
    • Ours B: DR + PCA (trained on CASIA-WebFace) + Cosine similarity.
    • Ours C: DR + Joint Bayesian (trained on CASIA-WebFace).
    • Ours D: DR + PCA (trained on LFW training splits) + Cosine similarity.
    • Ours E: DR + Joint Bayesian (trained on LFW training splits).

    A single network trained on CASIA-WebFace with Joint Bayesian metric learning (Ours E, 97.73%97.73\%) outperforms Facebook's 7-network DeepFace ensemble (97.35%97.35\%) and DeepID2's 2-network ensemble (97.28%97.28\%), while being comparable to DeepID2's 4-network ensemble (97.75%97.75\%).

  7. Knowl 7 — Face Verification and Open-Set Identification on the BLUFR Benchmark

    data/table

    The proposed deep representation with Joint Bayesian metric learning (Ours E) was evaluated against conventional high-dimensional shallow models on the Benchmark of Large-Scale Unconstrained Face Recognition (BLUFR) protocol over all 13,233 LFW images (10 trials with an average of 156,915 genuine pairs and 46,960,863 impostor pairs per trial).

    Method VR@FAR=0.1% DIR@FAR=1%, Rank=1
    HD-LBP + JB 41.66% 18.07%
    HD-LBP + LDA 36.12% 14.94%
    Ours E 80.26% 28.90%

    Reported numbers represent μ−σ\mu - \sigma over the 10 experimental trials, where μ\mu is mean accuracy and σ\sigma is standard deviation. Performance metrics include:

    • Verification Rate (VR) at a False Accept Rate (FAR) of 0.1%0.1\%.
    • Detection and Identification Rate (DIR) at FAR=1%\text{FAR} = 1\% and Rank=1\text{Rank} = 1.

    The 11-layer CNN representation (Ours E) improves VR@FAR=0.1% from 41.66%41.66\% to 80.26%80.26\%. However, the open-set identification DIR at Rank=1 (28.90%28.90\%) reveals a substantial remaining performance gap for real-world surveillance watch-list applications.

  8. Knowl 8 — Video Face Verification Performance on the YouTube Faces Benchmark

    data/table

    The generalization ability of the CASIA-WebFace representation was evaluated on the YouTube Faces (YTF) video face dataset. For each video, 15 frames are randomly sampled, and video pair similarity is computed as the mean similarity across all 15×15=22515 \times 15 = 225 pairwise frame combinations.

    Method #Net Accuracy Protocol
    DeepFace 1 91.4±1.1%91.4 \pm 1.1\% supervised
    Ours A 1 88.00±1.50%88.00 \pm 1.50\% unsupervised
    Ours D 1 90.60±1.24%90.60 \pm 1.24\% unsupervised
    Ours E 1 92.24±1.28%92.24 \pm 1.28\% supervised

    Configurations:

    • Ours A: Direct Cosine similarity between base representations without fine-tuning.
    • Ours D: Representation projected via PCA trained on YTF training splits followed by Cosine similarity.
    • Ours E: Representation mapped using Joint Bayesian metric learning trained on YTF training splits.

    When applying Joint Bayesian metric learning (Ours E), the single-network model achieves 92.24±1.28%92.24 \pm 1.28\%, exceeding DeepFace (91.4±1.1%91.4 \pm 1.1\%).

Coverage note — No substantial contributed material was omitted. All dataset collection algorithms, architectural specifications, loss functions, training hyperparameters, and experimental evaluations across LFW, BLUFR, and YTF benchmarks are fully captured.

References

  1. 1.P. N. Belhumeur, J. P. Hespanha, and D. J. Kriegman. "Eigenfaces vs. Fisherfaces: Recognition using class specific linear projection". In Proceedings of the European Conference on Computer Vision, pages 45–58, 1996.
  2. 2.Y. Bengio. "Learning deep architectures for AI". Foundations and trends in Machine Learning, 2(1):1–127, 2009.
  3. 3.B.-C. Chen, C.-S. Chen, and W. H. Hsu. "Cross-age reference coding for age-invariant face recognition and retrieval". In Computer Vision–ECCV 2014, pages 768–783. Springer, 2014.
  4. 4.D. Chen, X. Cao, L. Wang, F. Wen, and J. Sun. "Bayesian face revisited: A joint formulation". In Computer Vision–ECCV 2012, pages 566–579. Springer, 2012.
  5. 5.D. Chen, X. Cao, F. Wen, and J. Sun. "Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification". In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 3025–3032, 2013.
  6. 6.G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. "Improving neural networks by preventing co-adaptation of feature detectors". arXiv preprint arXiv:1207.0580, 2012.
  7. 7.G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller. "Labeled faces in the wild: A database for studying face recognition in unconstrained environments". 2007.
  8. 8.Y. Jia. Caffe. http://caffe.berkeleyvision.org/.
  9. 9.A. Krizhevsky. cuda-convnet. http://code.google.com/p/cuda-convnet/.
  10. 10.A. Krizhevsky, I. Sutskever, and G. E. Hinton. "Imagenet classification with deep convolutional neural networks". In Advances in neural information processing systems, pages 1097–1105, 2012.
  11. 11.N. Kumar, A. C. Berg, P. N. Belhumeur, and S. K. Nayar. Attribute and simile classifiers for face verification. In Computer Vision, 2009 IEEE 12th International Conference on, pages 365–372. IEEE, 2009.
  12. 12.Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. "Gradient-based learning applied to document recognition". Proceedings of the IEEE, 86(11):2278–2324, 1998.
  13. 13.Z. Lei, S. Liao, M. Pietikainen, and S. Z. Li. "Face recognition by exploring information jointly in space, scale and orientation". Image Processing, IEEE Transactions on, 20(1):247–256, 2011.
  14. 14.Z. Lei, M. Pietikainen, and S. Li. "Learning discriminant face descriptor". 2013.
  15. 15.S. Z. Li, R. Chu, S. Liao, and L. Zhang. "Illumination invariant face recognition using near-infrared images". IEEE Transactions on Pattern Analysis and Machine Intelligence, 26, April 2007.
  16. 16.S. Liao. "Benchmark of large-scale unconstrained face recognition". http://www.cbsr.ia.ac.cn/users/scliao/projects/blufr/.
  17. 17.S. Liao, Z. Lei, D. Yi, and S. Z. Li. "A benchmark study of large-scale unconstrained face recognition". In IAPR/IEEE International Joint Conference on Biometrics, Clearwater, Florida, USA, 2014.
  18. 18.C. Liu and H. Wechsler. "Gabor feature based classification using the enhanced fisher linear discriminant model for face recognition". IEEE Transactions on Image Processing, 11(4):467–476, 2002.
  19. 19.D. Maturana, D. Mery, and A. Soto. "Face recognition with decision tree-based local binary patterns". In Computer Vision–ACCV 2010, pages 618–629. Springer, 2011.
  20. 20.V. Nair and G. E. Hinton. "Rectified linear units improve restricted boltzmann machines". In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 807–814, 2010.
  21. 21.P. J. Phillips, H. Moon, S. A. Rizvi, and P. J. Rauss. "The FERET evaluation methodology for face-recognition algorithms". IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(10):1090–1104, 2000.
  22. 22.T. Sim, S. Baker, and M. Bsat. "The cmu pose, illumination, and expression (PIE) database". In Automatic Face and Gesture Recognition, 2002. Proceedings. Fifth IEEE International Conference on, pages 46–51. IEEE, 2002.
  23. 23.K. Simonyan and A. Zisserman. "Very deep convolutional networks for large-scale image recognition". arXiv preprint arXiv:1409.1556, 2014.
  24. 24.Y. Sun, X. Wang, and X. Tang. "Deep learning face representation by joint identification-verification". arXiv preprint arXiv:1406.4773, 2014.
  25. 25.C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. "Going deeper with convolutions". arXiv preprint arXiv:1409.4842, 2014.
  26. 26.Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. "Deepface: Closing the gap to human-level performance in face verification". In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on, pages 1701–1708. IEEE, 2014.
  27. 27.M. A. Turk and A. P. Pentland. "Eigenfaces for recognition". Journal of Cognitive Neuroscience, 3(1):71–86, March 1991.
  28. 28.L. Wolf, T. Hassner, and I. Maoz. "Face recognition in unconstrained videos with matched background similarity". In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 529–534. IEEE, 2011.
  29. 29.D. Yi, Z. Lei, and S. Z. Li. "Towards pose robust face recognition". In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on, pages 3539–3545, 2013.

Citation

MLA
Yi, D., et al. “Learning Face Representation from Scratch”. arXiv, 2014, https://doi.org/10.48550/arxiv.1411.7923.
APA
Yi, D., Lei, Z., Liao, S., & Li, S. Z. (2014). Learning Face Representation from Scratch. arXiv. https://doi.org/10.48550/arxiv.1411.7923
Chicago
Yi, D., Z. Lei, S. Liao, and S. Z. Li. 2014. “Learning Face Representation from Scratch”. Preprint, ArXiv. https://doi.org/10.48550/arxiv.1411.7923.
Harvard
Yi, D. et al. (2014) “Learning Face Representation from Scratch”. arXiv. Available at: https://doi.org/10.48550/arxiv.1411.7923.
Vancouver
1. Yi D, Lei Z, Liao S, Li SZ (2014) Learning Face Representation from Scratch. https://doi.org/10.48550/arxiv.1411.7923

BibTeX

@misc{https://doi.org/10.48550/arxiv.1411.7923,
  doi = {10.48550/ARXIV.1411.7923},
  url = {https://arxiv.org/abs/1411.7923},
  author = {Yi, Dong and Lei, Zhen and Liao, Shengcai and Li, Stan Z.},
  keywords = {Computer Vision and Pattern Recognition (cs.CV), FOS: Computer and information sciences, FOS: Computer and information sciences},
  title = {Learning Face Representation from Scratch},
  publisher = {arXiv},
  year = {2014},
  copyright = {arXiv.org perpetual, non-exclusive license}
}
Metadata:DOI registry

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: Authors