🖼️ Image Captioning with Transformer + EfficientNet

A step-by-step guide for building image captioning models using deep learning

---

🎯 Objective

This project walks through the implementation of a Transformer-based image captioning model using the Flickr30k dataset. The architecture leverages:

• 🧠 A pre-trained EfficientNetB0 CNN for image feature extraction
• 🧠 A Transformer decoder to generate human-readable captions

This project serves as a hands-on learning tool for understanding Transformer internals and applying them to a vision-language task.

---

🧠 What is a Transformer?

The Transformer architecture, introduced in the 2017 paper "Attention is All You Need", revolutionized deep learning by enabling efficient parallel training and powerful attention mechanisms.

In this project, we break down:

• 🔤 Input Embeddings + Positional Encoding
• 🎯 Multi-head Attention (Self & Cross)
• 🚫 Masks to ignore padding/future tokens
• 🧩 Decoder block structure with Layer Norm and FFNs

---

🛠 Methodology

📁 1. Data & Preprocessing

• Dataset: Flickr30k
• Caption cleaning, tokenization, filtering based on length
• Image preprocessing (resize, normalize)
• TensorFlow Datasets used with shuffling, mapping, batching, and prefetching

🔍 2. Model Architecture

• Encoder:

  • Pre-trained EfficientNetB0 (feature extractor)
  • Dense layers to project into decoder dimension

• Decoder:

  • Positional Embedding
  • Masked Multi-head Self-Attention
  • Multi-head Cross Attention
  • Feed-forward + LayerNorm
  • Output: Dense layer with Softmax

🏋️ 3. Training

• Input: image + caption sequence (excluding <end>)
• Target: shifted caption (excluding <start>)
• Loss: Sparse Categorical Crossentropy
• Optimizer: Adam with learning rate scheduling

🧪 4. Inference & Evaluation

• Uses Greedy decoding during inference
• Evaluated using BLEU score

---

📂 Project Structure

├── image_captioning_transformer.ipynb # Main notebook

├── images/ # Model architecture + sample predictions

└── README.md

---

🚀 How to Run

1. Clone the Repo

git clone https://github.com/SaharZargarzadeh/ImageCaptioning-Transformer-EfficientNet.git
cd ImageCaptioning-Transformer-EfficientNet

2. Install Dependencies

pip install -r requirements.txt

3. Run the Notebook

jupyter notebook image_captioning_transformer.ipynb