Human-in-the-Loop Post-Training with Prolific

Post-training is what separates a capable base model from one that's actually useful. It's how you make a model safer, more helpful, domain-specific, or aligned to a particular style.

This repository shows how to use Prolific as the infrastructure for collecting that human data across two post-training stages: Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO).

Overview

Both SFT and DPO require human data: demonstrations for SFT, preference judgments for DPO. This repo shows how to run those collection steps through Prolific's API and feed the results directly into training.

The same structure applies to most post-training tasks:

• Style or tone adaptation (brand voice, formal/informal register)
• Domain-specific assistants (medical, legal, customer support)
• Safety fine-tuning via human judgments on harmful outputs
• Instruction following in specific formats (JSON, citations, structured reports)
• Multilingual alignment to native speaker preferences

---

The example in this repo

To keep things concrete and fun, we use helpful pirate-style responses as the post-training target. We take Qwen/Qwen3-8B-Base and teach it to respond like a helpful pirate through two human-powered training stages:

1. SFT — Prolific participants rewrite normal AI responses into pirate style. We train on those rewrites.
2. DPO — Prolific participants pick the most helpful pirate response from pairs generated by the SFT model. We use those preferences to refine the model further.

The pirate task is deliberately lightweight so the pipeline is easy to follow. Swap in your own prompts, instructions, and reward criteria to adapt it to any use case.

---

Notebooks

Each step in the pipeline is a self-contained notebook, written to be readable and easy to follow. The goal is to make every decision visible.

#	Notebook	Purpose	Prolific?	Tinker?
1	1_tinker_generate_base.ipynb	Generate base model responses for every prompt	—	✓ Sample
2	2_prolific_sft_collection.ipynb	Show responses to participants, collect human rewrites	✓ free_text	—
3	3_tinker_sft_training.ipynb	Train SFT model on human-written responses	—	✓ Train
4	4_tinker_compare_base_sft.ipynb	Sanity check: does the SFT model behave differently?	—	✓ Sample
5	5_tinker_generate_sft_pairs.ipynb	Generate response pairs from SFT model for preference annotation	—	✓ Sample
6	6_prolific_dpo_collection.ipynb	Show response pairs to participants, collect preferences	✓ pairwise	—
7	7_tinker_dpo_training.ipynb	DPO on the SFT model using human preferences	—	✓ Train
8	8_tinker_compare_all.ipynb	3-way comparison: base / SFT / DPO	—	✓ Sample

The Prolific API handles all human data collection (notebooks 2 and 6). Tinker (an LLM training API) is used here for model inference and fine-tuning.

---

Config files

File	Used by	What to change
config/prompts_sft.jsonl	nb 1	Add/remove SFT prompts (100 prompts)
config/prompts_dpo.jsonl	nb 5	Add/remove DPO prompts (1000 prompts)
config/1_tinker_base_gen.yaml	nb 1	Base model, sampling params
config/2_prolific_sft_config.yaml	nb 2	Task instructions, reward, participant count
config/3_tinker_sft_training.yaml	nb 3	Learning rate, epochs, LoRA rank
config/5_tinker_sft_gen.yaml	nb 5	Temperature, number of samples per prompt
config/6_prolific_dpo_config.yaml	nb 6	Task instructions, reward, participant count
config/7_tinker_dpo_training.yaml	nb 7	Learning rate, beta, epochs

---

Output files

output/
├── 1_base_responses.jsonl              # {prompt_id, prompt, response}
├── 2_prolific_upload.csv               # upload file used to create the Prolific study
├── 2_prolific_responses_raw.csv        # raw Prolific free-text submissions (audit trail)
├── 2_prolific_demographic_data.csv     # participant demographics from SFT study
├── 2_sft_training_data.jsonl           # {"messages": [{user}, {assistant: human rewrite}]}
├── 3_sft_model/                        # Tinker SFT artifacts (checkpoints.jsonl, metrics.jsonl, …)
├── 5_dpo_response_pairs.jsonl          # {prompt_id, prompt, response_a, response_b}
├── 6_prolific_upload.csv               # upload file used to create the Prolific study
├── 6_prolific_responses_raw.csv        # raw Prolific pairwise preferences (audit trail)
├── 6_prolific_responses_raw_clean.csv  # cleaned version of pairwise preferences
├── 6_prolific_demographic_data.csv     # participant demographics from DPO study
├── 6_dpo_training_data.jsonl           # Tinker comparison format
└── 7_dpo_model/                        # Tinker DPO artifacts

---

Setup

1. Create the conda environment

conda env create -f environment.yml
conda activate dpo_prolific

2. Set up environment variables

Copy the example env file and fill in your credentials:

cp .env.example .env

HF_TOKEN=your_huggingface_token
TINKER_TOKEN=your_tinker_api_token
PROLIFIC_API_TOKEN=your_prolific_api_token
PROLIFIC_WORKSPACE_ID=your_prolific_workspace_id
PROLIFIC_PROJECT_ID=your_prolific_project_id

3. Launch Jupyter

jupyter notebook

---

About Prolific 💙

Prolific connects AI researchers and developers with high-quality human data. Source large samples of domain experts, experienced AI trainers, and diverse demographics in hours, not weeks.

• Documentation: https://docs.prolific.com/
• Get started today: https://app.prolific.com/

---

Important Notice

This project is provided as-is for educational and research purposes only.

• Beta Status: This is experimental code and may contain bugs or incomplete features
• Not Maintained: No active development or support is provided
• Educational Use: Intended as a learning resource
• Use at Your Own Risk: Test thoroughly before using in production environments

Contributions are welcome! Please feel free to submit issues or pull requests.