Skip to Content
LibraryPapersAlexNet (2012)

AlexNet: ImageNet Classification with Deep Convolutional Neural Networks

Authors: Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton Year: 2012 Venue: NeurIPS (NIPS) Impact: ⭐⭐⭐ - Sparked the modern deep learning revolution

Overview

AlexNet was the breakthrough CNN that won the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) 2012, achieving a top-5 error rate of 15.3% - a 10.8 percentage point improvement over the second-place entry. This decisive victory demonstrated that deep learning could dramatically outperform traditional computer vision methods and catalyzed the modern deep learning era.

Key Innovations

1. ReLU Activation

AlexNet used ReLU (Rectified Linear Unit) activations instead of tanh or sigmoid:

ReLU(x)=max⁡(0,x)\text{ReLU}(x) = \max(0, x)

Why it mattered:

  • Faster training: No vanishing gradient problem for positive activations
  • Biological plausibility: More similar to biological neurons
  • Computational efficiency: Simple max operation

This choice enabled training much deeper networks than previously possible.

2. Dropout Regularization

Applied dropout (p=0.5) in the fully-connected layers to combat overfitting:

# During training: randomly drop 50% of activations
dropout = nn.Dropout(p=0.5)

This prevented co-adaptation of neurons and significantly reduced overfitting on ImageNet.

3. Data Augmentation

Aggressive data augmentation to artificially increase training data:

  • Random crops and horizontal flips
  • Color jittering (PCA-based intensity adjustments)

Generated 2048× more training data from the original images.

4. GPU Training

Trained on two NVIDIA GTX 580 GPUs with custom parallelization:

  • Split model across GPUs
  • Communication only at certain layers
  • Made training deep networks on large datasets feasible

5. Local Response Normalization (LRN)

Introduced local response normalization to encourage lateral inhibition (now largely replaced by batch normalization).

Architecture

AlexNet consists of 8 learned layers: 5 convolutional and 3 fully-connected.

Layer Structure

Input: 224×224×3 RGB image

Conv1: 96 filters, 11×11, stride 4 → ReLU → MaxPool (3×3, stride 2)
  Output: 55×55×96 → 27×27×96

Conv2: 256 filters, 5×5, pad 2 → ReLU → MaxPool (3×3, stride 2)
  Output: 27×27×256 → 13×13×256

Conv3: 384 filters, 3×3, pad 1 → ReLU
  Output: 13×13×384

Conv4: 384 filters, 3×3, pad 1 → ReLU
  Output: 13×13×384

Conv5: 256 filters, 3×3, pad 1 → ReLU → MaxPool (3×3, stride 2)
  Output: 13×13×256 → 6×6×256

FC6: 4096 units → ReLU → Dropout (0.5)
FC7: 4096 units → ReLU → Dropout (0.5)
FC8: 1000 units (ImageNet classes) → Softmax

Parameters

  • Total parameters: ~60 million
  • Majority in FC layers: FC6 and FC7 contain ~90% of parameters
  • Modern insight: FC layers are now often replaced with global average pooling

Training Details

  • Dataset: ImageNet ILSVRC-2012 (1.2M training images, 1000 classes)
  • Optimization: SGD with momentum (0.9)
  • Learning rate: 0.01, reduced by 10× when validation error plateaus
  • Weight decay: 0.0005
  • Batch size: 128
  • Training time: ~6 days on two GPUs

Results

ILSVRC-2012 Performance:

  • Top-5 error: 15.3% (AlexNet) vs 26.2% (second place)
  • Top-1 error: 37.5%

This massive improvement over traditional methods (SIFT features + SVM) convinced the computer vision community that deep learning was the future.

Historical Impact

AlexNet’s success had far-reaching consequences:

  1. Sparked deep learning boom: Demonstrated clear superiority on a challenging real-world task
  2. GPU acceleration became standard: Showed that GPUs could make deep learning practical
  3. Transfer learning era began: Pre-trained AlexNet features proved useful across many vision tasks
  4. Industrial adoption: Tech companies rapidly invested in deep learning research
  5. Academic shift: Computer vision conferences became dominated by deep learning papers

Limitations and Modern Perspective

Outdated Design Choices

  • Large filters: 11×11 and 5×5 filters are inefficient; modern networks use mostly 3×3
  • Large FC layers: Modern networks use global average pooling instead
  • LRN: Replaced by batch normalization in modern architectures
  • GPU parallelization: Modern frameworks handle this automatically

What Still Matters

  • ReLU activation: Still the default choice
  • Dropout: Still widely used for regularization
  • Data augmentation: Essential technique for all vision models
  • Deep architecture: Validated the principle of learning hierarchical features

Legacy

While AlexNet itself is rarely used today (replaced by ResNet, EfficientNet, Vision Transformers), its historical importance cannot be overstated. It:

  • Proved deep learning could solve real-world vision problems
  • Established design patterns (ReLU, dropout, data augmentation) still used today
  • Made GPU acceleration standard in deep learning
  • Catalyzed the rapid advancement of computer vision that followed

AlexNet represents the beginning of the modern deep learning era.

  • VGG Networks - Demonstrated the importance of depth
  • ResNet - Solved the vanishing gradient problem with skip connections
  • Transfer Learning - AlexNet features proved highly transferable

Key Papers

Learning Resources

Videos

Articles

OverviewVGG (2014)