Emotion Detection using Swin Transformer (Image + Video)

A two-stage deep learning pipeline for facial emotion recognition, combining spatial learning from images and temporal understanding from videos.

---

Overview

This project explores emotion recognition by first training a model on static facial images and then extending it to video-based emotion detection using temporal aggregation.

---

⚙️ Pipeline

Stage 1 – Image Training

• Dataset: FER-2013
• Model: Swin Transformer
• Goal: Learn facial feature representations

Stage 2 – Video Training

• Datasets: IEMOCAP, RAVDESS
• Input: 8 frames per video
• Method: Temporal pooling over frame features
• Goal: Learn emotion dynamics over time

---

🧩 Model Architecture

• Backbone: Swin Transformer
• Transfer Learning: Stage 1 → Stage 2
• Temporal Layer: Fully connected head over pooled frame features

---

📊 Results

Stage 1 (FER-2013)

• Accuracy: ~74%
• Balanced performance across classes

Stage 2 (Video Model)

• Observed high validation accuracy (~90%)
• Identified data leakage due to improper dataset split
• Proposed fix: video-level splitting to ensure independence

---

📈 Visualizations

Confusion Matrix

Accuracy Curve

Loss Curve

ROC Curve

Precision-Recall Curve

---

🛠️ Tech Stack

• PyTorch
• Hugging Face Transformers
• NumPy
• Matplotlib, Seaborn
• OpenCV

---

📁 Project Structure

emotion_det/ ├── data/ ├── models/ ├── scripts/ ├── results/ ├── checkpoints/ ├── requirements.txt

---

⚠️ Notes

• Video model performance is affected by dataset leakage during validation
• Proper evaluation requires strict separation at video level
• Future work includes improving temporal modeling (LSTM / 3D CNN)

---

📌 Conclusion

This project demonstrates a structured approach to emotion recognition using transfer learning and highlights the importance of proper validation strategies in deep learning workflows.