Biobank Data Release Manager

Secure genomic data delivery pipeline — SQL query formatting, bcftools VCF extraction, and post-extraction sample concordance validation.

Portfolio disclaimer: This repository contains sanitised, generalised versions of tooling developed at NIHR BioResource. No real participant data or internal paths are included.

---

Overview

When a research team requests genotype data for a specific cohort, the delivery process must ensure they receive exactly the approved samples. This toolkit automates:

• Genotype extraction — bcftools-based VCF/BCF subset extraction with pre-flight and post-extraction validation.
• Sample concordance — verifying that output files contain precisely the requested participants.
• SQL query helpers — converting flat barcode lists into SQL IN clauses and cleaning database exports.

Repository Structure

.
├── src/release_manager/            Python package
│   ├── __init__.py
│   ├── extractor.py                bcftools VCF extraction engine
│   ├── validator.py                Sample concordance validation
│   └── sql_helper.py               SQL clause generation, TSV cleaning
├── tests/
│   ├── test_validator.py
│   └── test_sql_helper.py
├── legacy/                         Original shell scripts
├── .github/workflows/ci.yml
├── pyproject.toml
└── Makefile

Quick Start

pip install -e ".[dev]"

Python API

from release_manager import GenotypeExtractor, SampleValidator, SQLQueryBuilder

# Generate SQL IN clause
clause = SQLQueryBuilder.build_in_clause(["BC001", "BC002", "BC003"])
# -> "barcode IN ('BC001', 'BC002', 'BC003')"

# Validate sample concordance
validator = SampleValidator()
report = validator.validate_fam("request.txt", "output.fam")
print(f"Concordance: {report.concordance_rate:.0%}")
print(f"Missing: {report.missing}")

# Extract genotypes (requires bcftools)
extractor = GenotypeExtractor()
result = extractor.extract("source.vcf.gz", "samples.txt", "output.vcf.gz")
print(f"Extracted {result.extracted_samples}/{result.requested_samples}")

Testing

make test   # or: pytest tests/ -v

Jira Provenance

• WGS/WES data provisioning — extracting participant subsets from master VCF files for approved data releases.
• Sample concordance — post-extraction auditing to verify delivery completeness.
• SQL query automation — formatting barcode lists for clinical database queries.

Licence

MIT