Skip to content

VED-000 README documentation updates and removed obsolete dev tools #1181

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dlzhry2nhs merged 2 commits into from
Feb 3, 2026
Merged

VED-000 README documentation updates and removed obsolete dev tools #1181

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
620e4c9
Updated READMEs and removed obsolete devtools
dlzhry2nhs Feb 3, 2026
5e26017
PR comments
dlzhry2nhs Feb 3, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,6 @@ openapi.json
**/.vscode/**/*
!**/.vscode/settings.json.default

devtools/volume/
**/.coverage
**/test-results.xml
allure-results/
Expand Down
77 changes: 39 additions & 38 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,39 +14,49 @@ See https://nhsd-confluence.digital.nhs.uk/display/APM/Glossary.

**Note:** Each Lambda has its own `README.md` file for detailed documentation. For non-Lambda-specific folders, refer to `README.specification.md`.

### Lambdas

| Folder | Description |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `backend` | **Imms API** – Handles CRUD operations for the Immunisation API. |
| `delta_backend` | **Imms Sync** – Lambda function that reacts to events in the Immunisation database. |
| `ack_backend` | **Imms Batch** – Generates the final Business Acknowledgment (BUSACK) file from processed messages and writes it to the designated S3 location. |
| `filenameprocessor` | **Imms Batch** – Processes batch file names. |
| `mesh_processor` | **Imms Batch** – MESH-specific batch processing functionality. |
| `recordprocessor` | **Imms Batch** – Handles batch record processing. |
| `redis_sync` | **Imms Redis** – Handles sync s3 to REDIS. |
| `id_sync` | **Imms Redis** – Handles sync SQS to IEDS. |
| `shared` | **Imms Redis** – Not a lambda but Shared Code for lambdas |
### Lambdas (compute microservices)

| Folder | Description |
| ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `ack_backend` | **Imms Batch** – Generates the final Business Acknowledgment (BUSACK) file from processed messages and writes it to the designated destination. |
| `backend` | **Imms API** – Handles CRUD operations for the Immunisation API. |
| `batch_processor_filter` | **Imms Batch** – Controller function that consumes from a queue and forwards file event for processing if filter conditions are met. |
| `delta_backend` | **Imms Sync** – Lambda function that reacts to events in the Immunisation Event Data Store (IEDS). |
| `filenameprocessor` | **Imms Batch** – Validates and processes new batch file events. |
| `id_sync` | **Imms Cross-cutting** – Handles [MNS](https://digital.nhs.uk/developer/api-catalogue/multicast-notification-service) NHS Number Change events and applies updates to affected records. |
Comment thread
edhall-nhs marked this conversation as resolved.
| `mesh_processor` | **Imms Batch** – Triggered when new files are received via MESH. Moves them into the Imms Batch processing system. |
| `mns_subscription` | **Imms Cross-cutting** – Simple helper Lambda which sets up our required MNS subscription. Used in pipelines in DEV. |
| `recordforwarder` | **Imms Batch** – Consumes from the stream and applies the processed batch file row operations (CUD) to IEDS. |
| `recordprocessor` | **Imms Batch** – ECS Task - **not** a Lambda function - responsible for processing batch file rows and forwarding to the stream. |
| `redis_sync` | **Imms Cross-cutting** – Handles config file updates. E.g. disease mapping or permission files. |
| `shared` | **Imms Cross-cutting** – Shared `common` code that can be shared by other Lambda functions. This is not a standalone Lambda. |

---

### Pipelines

| Folder | Description |
| ------- | ------------------------------------------- |
| `azure` | Pipeline definition and orchestration code. |
Due to the timing of the project's inception, Azure pipelines have been inherited from the API Management team for deploying
the Apigee proxy and sandbox. The new way to manage and deploy said resources is [Proxygen](https://digital.nhs.uk/developer/api-catalogue/proxy-generator).

In future a migration plan will be provided by the API Management team so we can move to the new process and use purely
GitHub Actions for our entire pipeline.

| Folder | Description |
| --------- | ------------------------------------------------------------------------------------------------------------------------------- |
| `azure` | Pipeline definition and orchestration code concerned purely with the management and deployment of the Apigee proxy and sandbox. |
| `.github` | Pipeline definition and orchestration code for deploying and testing our backend project. |

---

### Infrastructure

| Folder | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------ |
| `account` | Base infrastructure components. |
| `grafana` | Terraform configuration for Grafana, built on top of core infra. |
| `instance` | Core Terraform infrastructure code. This is run in each PR and sets up lambdas associated with the PR. |
| `terraform_aws_backup` | Streamlined backup processing with AWS. |
| `proxies` | Apigee API proxy definitions. |
| Folder | Description |
| ---------------------- | ---------------------------------------------------------------- |
| `account` | Base infrastructure components deployed on a per account basis. |
| `grafana` | Terraform configuration for Grafana, built on top of core infra. |
| `instance` | Core Terraform app infrastructure. |
| `terraform_aws_backup` | Streamlined backup processing with AWS. |
| `proxies` | Apigee API proxy definitions. |

---

Expand All @@ -60,13 +70,12 @@ See https://nhsd-confluence.digital.nhs.uk/display/APM/Glossary.

### Utilities

| Folder | Description |
| ---------------- | ------------------------------------------------------------- |
| `devtools` | Helper tools and utilities for local development |
| `quality_checks` | Dependencies for linting and formatting Python code |
| `scripts` | Standalone or reusable scripts for development and automation |
| `specification` | Specification files to document API and related definitions |
| `sandbox` | Simple sandbox API |
| Folder | Description |
| ------------------- | ------------------------------------------------------------- |
| `quality_checks` | Dependencies for linting and formatting Python code |
| `utilities/scripts` | Standalone or reusable scripts for development and automation |
| `specification` | Specification files to document API and related definitions |
| `sandbox` | Simple sandbox API |

---

Expand Down Expand Up @@ -268,11 +277,3 @@ run a different set of tests. To do this:
Please note that this project requires that all commits are verified using a GPG key.
To set up a GPG key please follow the instructions specified here:
https://docs.github.com/en/authentication/managing-commit-signature-verification

## AWS configuration: Getting credentials for AWS federated user account

In the 'Access keys' popup menu under AWS Access Portal:

**NOTE** that AWS's 'Recommended' method of getting credentials **(AWS IAM Identity Center credentials)** will break mocking in unit tests; specifically any tests calling `dynamodb_client.create_table()` will fail with `botocore.errorfactory.ResourceInUseException: Table already exists`.

Instead, use **Option 2 (Add a profile to your AWS credentials file)**.
4 changes: 2 additions & 2 deletions lambdas/delta_backend/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,9 @@ This project is designed to convert FHIR-compliant JSON data (e.g., Immunization

| File Name | What It Does |
| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **`converter.py`** | 🧠 The main brain — applies the schema, runs conversions, handles errors. |
| **`converter.py`** | The main brain — applies the schema, runs conversions, handles errors. |
| **`conversion_layout.py`** | A plain Python list that defines which fields you want, and how they should be formatted (e.g. date format, renaming rules). |
| **`delta.py`** | Holds the function called by AWS Lambda |
| **`delta.py`** | Holds the function called by AWS Lambda. |
| **`extractor.py`** | Tailored functionality to extract target fields from immunization record received by the delta handler. |
| **`exception_messages.py`** | Holds reusable error messages and codes for clean debugging and validation feedback. |
| **`log_firehose.py`** | Firehose logging functionality. |
Expand Down
4 changes: 2 additions & 2 deletions lambdas/id_sync/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,12 @@

## Overview

**Id Sync** is an AWS Lambda function designed to trigger from SQS. It receives a list of NHS Numbers and checks for changes in PDS. If change found, it updates the Events Table..
**Id Sync** is an AWS Lambda function designed to trigger from SQS. It receives a list of NHS Numbers and checks for changes in PDS. If change found, it updates the Events Table.

## Features

- **SQS Event Driven:** Automatically triggered by SQS event.
- **DynamoDb Integration:** Reviews contents of DynbamoDb Events table and updates where required..
- **DynamoDb Integration:** Reviews contents of DynamoDb Events table and updates where required.
- **Logging:** Provides detailed logging for monitoring and troubleshooting.

## How It Works
Expand Down
10 changes: 9 additions & 1 deletion lambdas/recordforwarder/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,14 @@
# About

This document describes the environment setup for the recordforwarder Lambda.
This document describes the purpose and environment setup for the recordforwarder Lambda.

## Overview

The Record Forwarder Lambda consumes from an AWS Kinesis Stream, and is responsible for applying updates to the Immunisation
Event Data Store. It will receive up to 100 records per batch. Each record will have been processed and formatted as a
FHIR Immunization by the ECS Record Processor in the prior batch step.

The Record Forwarder Lambda will then execute the requested operation (Create, Update or Delete) on the given record.

## Setting up the recordforwarder lambda

Expand Down
2 changes: 1 addition & 1 deletion sonar-project.properties
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ sonar.projectKey=NHSDigital_immunisation-fhir-api
sonar.organization=nhsdigital
sonar.host.url=https://sonarcloud.io
sonar.python.version=3.11
sonar.exclusions=**/devtools/**,**/proxies/**,**/utilities/scripts/**,**/infrastructure/account/**,**/infrastructure/instance/**,**/infrastructure/grafana/**,**/terraform_aws_backup/**,**/tests/**
sonar.exclusions=**/proxies/**,**/utilities/scripts/**,**/infrastructure/account/**,**/infrastructure/instance/**,**/infrastructure/grafana/**,**/terraform_aws_backup/**,**/tests/**
sonar.coverage.exclusions=lambdas/shared/src/common/models/batch_constants.py
sonar.python.coverage.reportPaths=backend-coverage.xml,delta-coverage.xml,ack-lambda-coverage.xml,filenameprocessor-coverage.xml,recordforwarder-coverage.xml,recordprocessor-coverage.xml,mesh_processor-coverage.xml,redis_sync-coverage.xml,mns_subscription-coverage.xml,id_sync-coverage.xml,shared-coverage.xml,batchprocessorfilter-coverage.xml
sonar.cpd.exclusions=**/Dockerfile
Expand Down
4 changes: 0 additions & 4 deletions utilities/devtools/.gitignore
View file
Open in desktop

This file was deleted.

10 changes: 0 additions & 10 deletions utilities/devtools/.terraform.lock.hcl
View file
Open in desktop

This file was deleted.

20 changes: 0 additions & 20 deletions utilities/devtools/Makefile
View file
Open in desktop

This file was deleted.

Loading