Skip to content

Commit 17f6396

Browse files
authored
CCM-12869: add config -> DynamoDB publisher tooling + release-bundled GitHub Action (#50)
* CCM-12869: add config -> DynamoDB publisher tooling + release-bundled GitHub Action - Add @supplier-config/file-store loader/validator for local config store layout store = subdir per domain entity, JSON file per record zod-based parsing/validation; fail fast on parse errors - Add @supplier-config/ddb-publisher CLI (supplier-config-publish) to publish config records directly into a single DynamoDB table (no event envelope) -supports --dry-run (no AWS calls/creds; schema validation only) pre-load audit scans DDB; blocks upload if non-DISABLED records exist in DDB but not in local config unless --force - strict pk/sk handling: invalid/missing keys throw with pk/sk included for debug - add a lightweight composite action that downloads ddb-publish-bundle.tgz from the matching GitHub Release and runs index.cjs (avoids committing multi‑MB dist bundles; action contains no reimplemented logic) - Add release workflow to bundle and upload the CLI artifact to Releases, and run the bundling + smoke-test in stage-3-build to keep the release build healthy * Refactor Node.js setup in CI workflows and enhance dependency management * Ensure workspace packages are generated before bundling in release script * Enhance bundling process in release script to resolve workspace package imports * Use absolute path for alias in bundling to ensure correct resolution of imports * Update dependencies in package.json and package-lock.json for improved compatibility * Improve debug logging and update bundling configuration for workspace packages * Add ddb-publish bundling action and update release workflows * Add workflow artifact upload and improve ddb-publish bundle resolution * Refactor ddb-publish action to use action repository for artifact downloads and improve error handling * Enhance ddb-publish action to search multiple workflows for successful runs and improve error handling * Enhance runPublisher to log detailed steps and summarize loaded records * Add integration tests for DynamoDB Local and support custom endpoint configuration * Remove redundant console log eslint disables in bundle-release.mjs * Use testcontainers for ddb-local container management * Refactor pre-commit hooks to remove redundant local repo entries * feat(ddb-publisher): document local DynamoDB usage and add sample config store - add a short README for the ddb-publisher CLI - document running the CLI against a local DynamoDB container - add AWS CLI instructions to create the local supplier config table - add a minimal example config store under tests/example-config-store - rename the package script from start to cli for clearer CLI usage - allow localhost DynamoDB endpoints to use default local fake credentials - rename audit result field from blocking to blockingRecords for clarity - update audit and run tests to match the new audit shape - add coverage for local vs non-local endpoint credential handling * feat(tests): enhance integration tests for DynamoDB Local and improve config store loading * refactor(tests): replace 'kind' fields with 'id' in config store validation tests * feat(tests): add test for explicit AWS env values in local DynamoDB configuration * refactor(publish): remove 'entity' and 'env' fields from records and update validation logic * test(run): manage AWS environment variables during DynamoDB local tests * chore(template): add note for code generated by a coding agent in PR template * feat(lint): add linting and type checking scripts; update package.json and config files * fix(bundle): use fileURLToPath for packageRoot resolution; update version to 1.0.2 * feat(package): add pretypecheck script and update dependencies * Fix linting and integration test setup
1 parent 026a404 commit 17f6396

61 files changed

Lines changed: 13703 additions & 6827 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/PULL_REQUEST_TEMPLATE.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,7 @@
2626
- [ ] I have updated the documentation accordingly
2727
- [ ] This PR is a result of pair or mob programming
2828
<!-- - [ ] If I have used the 'skip-trivy-package' label I have done so responsibly and in the knowledge that this is being fixed as part of a separate ticket/PR. TODO - Re-visit Trivy usage https://nhsd-jira.digital.nhs.uk/browse/CCM-15549 -->
29+
- [ ] This PR includes code generated by a coding agent
2930

3031
---
3132

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
name: "Bundle ddb-publish"
2+
description: "Build and package the ddb-publish release bundle (ddb-publish-bundle.tgz) from the workspace sources."
3+
4+
inputs:
5+
node-version:
6+
description: "Node.js version to use"
7+
required: true
8+
run-typecheck:
9+
description: "Run workspace typecheck before bundling"
10+
required: false
11+
default: "true"
12+
13+
outputs:
14+
tarball-path:
15+
description: "Path to the generated tarball"
16+
value: ${{ steps.bundle.outputs.tarball_path }}
17+
18+
runs:
19+
using: "composite"
20+
steps:
21+
- name: Set up Node.js
22+
uses: actions/setup-node@v4
23+
with:
24+
node-version: "${{ inputs.node-version }}"
25+
cache: npm
26+
27+
- name: Install dependencies
28+
shell: bash
29+
run: |
30+
set -euo pipefail
31+
npm ci
32+
33+
- name: Generate dependencies
34+
shell: bash
35+
run: |
36+
set -euo pipefail
37+
npm run generate-dependencies --workspaces --if-present
38+
39+
- name: Typecheck
40+
if: inputs.run-typecheck == 'true'
41+
shell: bash
42+
run: |
43+
set -euo pipefail
44+
npm run typecheck --workspaces --if-present
45+
46+
- name: Build ddb-publish bundle
47+
shell: bash
48+
run: |
49+
set -euo pipefail
50+
npm run bundle:release --workspace @supplier-config/ddb-publisher
51+
52+
- name: Smoke-test bundle
53+
shell: bash
54+
run: |
55+
set -euo pipefail
56+
node packages/ddb-publisher/artifacts/ddb-publish/index.cjs --help > /dev/null
57+
58+
- name: Package tarball
59+
id: bundle
60+
shell: bash
61+
run: |
62+
set -euo pipefail
63+
tarball="ddb-publish-bundle.tgz"
64+
tar -czf "$tarball" -C packages/ddb-publisher/artifacts/ddb-publish .
65+
echo "tarball_path=$tarball" >> "$GITHUB_OUTPUT"
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
name: Release ddb-publish bundle
2+
3+
on:
4+
release:
5+
types: [published]
6+
workflow_dispatch:
7+
8+
permissions:
9+
contents: write
10+
11+
jobs:
12+
bundle-and-upload:
13+
runs-on: ubuntu-latest
14+
15+
steps:
16+
- uses: actions/checkout@v4
17+
18+
- name: Read Node version
19+
id: versions
20+
shell: bash
21+
run: |
22+
set -euo pipefail
23+
echo "node=$(grep '^nodejs\s' .tool-versions | awk '{print $2}')" >> "$GITHUB_OUTPUT"
24+
25+
- name: Bundle ddb-publish
26+
id: bundle
27+
uses: ./.github/actions/bundle-ddb-publish
28+
with:
29+
node-version: "${{ steps.versions.outputs.node }}"
30+
run-typecheck: "true"
31+
32+
- name: Upload release asset
33+
if: github.event_name == 'release'
34+
env:
35+
GH_TOKEN: ${{ github.token }}
36+
shell: bash
37+
run: |
38+
set -euo pipefail
39+
gh release upload "${{ github.event.release.tag_name }}" "${{ steps.bundle.outputs.tarball-path }}" --clobber
40+
41+
- name: Upload bundle as workflow artifact
42+
if: github.event_name != 'release'
43+
uses: actions/upload-artifact@v4
44+
with:
45+
name: ddb-publish-bundle
46+
path: "${{ steps.bundle.outputs.tarball-path }}"

.github/workflows/stage-2-test.yaml

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,3 +152,19 @@ jobs:
152152
sonar_organisation_key: "${{ vars.SONAR_ORGANISATION_KEY }}"
153153
sonar_project_key: "${{ vars.SONAR_PROJECT_KEY }}"
154154
sonar_token: "${{ secrets.SONAR_TOKEN }}"
155+
test-integration:
156+
name: "Integration tests"
157+
runs-on: ubuntu-latest
158+
timeout-minutes: 10
159+
steps:
160+
- name: "Checkout code"
161+
uses: actions/checkout@v4
162+
- name: "Repo setup"
163+
run: |
164+
npm ci
165+
- name: "Generate dependencies"
166+
run: |
167+
npm run generate-dependencies --workspaces --if-present
168+
- name: "Run integration tests"
169+
run: |
170+
npm run test:integration

.github/workflows/stage-3-build.yaml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,28 @@ jobs:
4444
uses: ./.github/actions/build-docs
4545
with:
4646
version: "${{ inputs.version }}"
47+
48+
ddb-publish-bundle:
49+
name: "Bundle ddb-publish CLI"
50+
runs-on: ubuntu-latest
51+
timeout-minutes: 3
52+
steps:
53+
- name: "Checkout code"
54+
uses: actions/checkout@v4
55+
56+
- name: "Bundle ddb-publish"
57+
id: bundle
58+
uses: ./.github/actions/bundle-ddb-publish
59+
with:
60+
node-version: "${{ inputs.nodejs_version }}"
61+
run-typecheck: "false"
62+
63+
- name: "Upload bundle as workflow artifact"
64+
uses: actions/upload-artifact@v4
65+
with:
66+
name: ddb-publish-bundle
67+
path: "${{ steps.bundle.outputs.tarball-path }}"
68+
4769
artefact-1:
4870
name: "Artefact 1"
4971
runs-on: ubuntu-latest

.tool-versions

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
act 0.2.64
22
gitleaks 8.24.0
33
jq 1.6
4-
nodejs 22.11.0
4+
nodejs 24.14.0
55
pre-commit 3.6.0
66
terraform 1.9.2
77
terraform-docs 0.19.0

AGENTS.md

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,7 @@ When proposing a change, agents should:
8181

8282
to catch formatting and basic lint issues. Domain specific checks will be defined in appropriate nested AGENTS.md files.
8383

84+
- When editing Markdown files, also run `./scripts/githooks/check-markdown-format.sh <file>` (or `check=all ./scripts/githooks/check-markdown-format.sh <file>` when needed) to catch markdownlint issues before review.
8485
- Suggest at least one extra validation step (for example `npm test` in a lambda, or triggering a specific workflow).
8586
- Any required follow up activites which fall outside of the current task's scope should be clearly marked with a 'TODO: CCM-12345' comment. The human user should be prompted to create and provide a JIRA ticket ID to be added to the comment.
8687

@@ -96,12 +97,16 @@ If you are blocked by an unavailable secret, unclear architectural constraint, m
9697

9798
## `nhs-notify-supplier-config` repo-specific notes
9899

99-
- `nhs-notify-repository-template/` is a checked-in reference copy of the template repo. Do not make normal feature changes there; only edit it when intentionally comparing with or syncing the template.
100+
- Template sync workflows and helper scripts may use `nhs-notify-repository-template/` as a temporary upstream checkout path. Treat it as sync/reference material rather than normal product code if it appears during maintenance work.
100101
- The main application and domain work in this repo is concentrated under `packages/`. Use local `README` files, manifests, and tests there to understand package-specific behaviour before making non-trivial changes.
102+
- Schema and event payload work is concentrated in `packages/events/`, with CloudEvent construction in `packages/event-builder/`. When changing schemas under `packages/events/src/domain/` or `packages/events/src/events/`, update any affected builders and Jest tests in both workspaces.
103+
- Supplier-config schemas use Zod 4 `.meta()` metadata for field titles/descriptions and generated artefacts. Preserve or update that metadata when changing schema fields, and rerun the relevant `packages/events/` generators when outputs change.
101104
- Important repo data inputs live at the top level and under `packages/ui/data/`, especially `specifications.json`, `specifications.xlsx`, and `letterVariants.csv`. Treat these as source material for supplier configuration work and preserve their formats unless the task explicitly changes them.
102105
- `make build` currently builds the Jekyll docs site (`docs/`); it is not a full monorepo build for the supplier-config packages.
103106
- CI currently still includes some template-era scaffolding. In particular, root workspace scripts and some `make test-*` paths do not yet cover every directory under `packages/`, so validate the specific package or area you changed directly as well as running the standard repo hooks.
104107
- Suggested extra validation by area:
108+
- `packages/events/`: run focused scripts such as `npm run test:unit --workspace=@nhsdigital/nhs-notify-event-schemas-supplier-config`, plus generators when schema outputs or derived artefacts are affected.
109+
- `packages/event-builder/`: run `npm run test:unit --workspace=@supplier-config/event-builder` and `npm run typecheck --workspace=@supplier-config/event-builder` when changing event builder logic.
105110
- `packages/`: run the changed package's local scripts (for example tests, typecheck, or generators) in addition to repo-level hooks.
106111
- `docs/`: run `(cd docs && make build)` when changing Jekyll content or templates.
107112
- root config/docs guidance: run `pre-commit run --config scripts/config/pre-commit.yaml` from the repo root.

actions/ddb-publish/action.yml

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
name: "Publish supplier config to DynamoDB"
2+
description: "Reads config from a file store and publishes records to a DynamoDB table."
3+
author: "NHS Notify"
4+
5+
inputs:
6+
source-path:
7+
description: "Path to the config store root directory"
8+
required: true
9+
target-env:
10+
description: "Target environment label (e.g. draft/int/prod)"
11+
required: true
12+
table-name:
13+
description: "DynamoDB table name"
14+
required: true
15+
force:
16+
description: "Bypass audit safety checks"
17+
required: false
18+
default: "false"
19+
dry-run:
20+
description: "Validate only (no AWS calls). Does not require credentials."
21+
required: false
22+
default: "false"
23+
24+
runs:
25+
using: "composite"
26+
steps:
27+
- name: Resolve and download ddb-publish bundle
28+
shell: bash
29+
env:
30+
GH_TOKEN: ${{ github.token }}
31+
CALLER_REPO: ${{ github.repository }}
32+
ACTION_REPO: ${{ github.action_repository }}
33+
ACTION_REF: ${{ github.action_ref }}
34+
RELEASE_ASSET_NAME: ddb-publish-bundle.tgz
35+
WORKFLOW_ARTIFACT_NAME: ddb-publish-bundle
36+
run: |
37+
set -euo pipefail
38+
39+
download_dir="${RUNNER_TEMP}/ddb-publish"
40+
unpack_dir="${download_dir}/unpacked"
41+
42+
mkdir -p "${download_dir}" "${unpack_dir}"
43+
44+
echo "[ddb-publish] caller_repo=${CALLER_REPO}"
45+
echo "[ddb-publish] action_repo=${ACTION_REPO}"
46+
echo "[ddb-publish] action_ref=${ACTION_REF:-<empty>}"
47+
48+
if [[ -z "${ACTION_REF}" ]]; then
49+
echo "ERROR: github.action_ref is empty; unable to locate release asset or branch artifact." >&2
50+
exit 1
51+
fi
52+
53+
if [[ -z "${ACTION_REPO}" ]]; then
54+
echo "ERROR: github.action_repository is empty; unable to determine where to fetch bundle artifacts." >&2
55+
exit 1
56+
fi
57+
58+
# Prefer release assets when the action ref is a tag with a matching release in the action repo.
59+
if gh release view "${ACTION_REF}" --repo "${ACTION_REPO}" >/dev/null 2>&1; then
60+
echo "[ddb-publish] Found release for ref '${ACTION_REF}' in '${ACTION_REPO}'. Downloading '${RELEASE_ASSET_NAME}'."
61+
gh release download "${ACTION_REF}" --repo "${ACTION_REPO}" --pattern "${RELEASE_ASSET_NAME}" --dir "${download_dir}"
62+
tar -xzf "${download_dir}/${RELEASE_ASSET_NAME}" -C "${unpack_dir}"
63+
echo "[ddb-publish] Bundle extracted from release asset."
64+
exit 0
65+
fi
66+
67+
# Otherwise treat the ref as a branch-like ref and fetch the latest successful CI artifact from the action repo.
68+
branch="${ACTION_REF#refs/heads/}"
69+
echo "[ddb-publish] No release found for ref '${ACTION_REF}'. Falling back to latest workflow artifact on branch '${branch}' from '${ACTION_REPO}'."
70+
71+
run_id=""
72+
workflow_used=""
73+
74+
for workflow_file in "stage-3-build.yaml" "cicd-1-pull-request.yaml"; do
75+
echo "[ddb-publish] Looking for successful runs in workflow '${workflow_file}'."
76+
77+
set +e
78+
run_id_candidate="$(gh run list \
79+
--repo "${ACTION_REPO}" \
80+
--workflow "${workflow_file}" \
81+
--branch "${branch}" \
82+
--status success \
83+
--json databaseId \
84+
--jq '.[0].databaseId' 2>/tmp/ddb_publish_run_list_err.log)"
85+
rc=$?
86+
set -e
87+
88+
if [[ $rc -ne 0 ]]; then
89+
if grep -q "HTTP 404" /tmp/ddb_publish_run_list_err.log; then
90+
echo "[ddb-publish] Workflow '${workflow_file}' not found on default branch; trying next workflow candidate."
91+
continue
92+
fi
93+
94+
echo "ERROR: Failed to query workflow runs from '${ACTION_REPO}' for workflow '${workflow_file}'." >&2
95+
cat /tmp/ddb_publish_run_list_err.log >&2 || true
96+
echo "HINT: ensure the token has read access to '${ACTION_REPO}' and workflow permissions allow actions read." >&2
97+
exit 1
98+
fi
99+
100+
if [[ -n "${run_id_candidate}" && "${run_id_candidate}" != "null" ]]; then
101+
run_id="${run_id_candidate}"
102+
workflow_used="${workflow_file}"
103+
break
104+
fi
105+
done
106+
107+
if [[ -z "${run_id}" || "${run_id}" == "null" ]]; then
108+
echo "ERROR: Could not find a successful run on branch '${branch}' in '${ACTION_REPO}' containing artifact '${WORKFLOW_ARTIFACT_NAME}'. Checked workflows: stage-3-build.yaml, cicd-1-pull-request.yaml." >&2
109+
exit 1
110+
fi
111+
112+
echo "[ddb-publish] Downloading artifact '${WORKFLOW_ARTIFACT_NAME}' from workflow run ${run_id} (${workflow_used}) in '${ACTION_REPO}'."
113+
gh run download "${run_id}" --repo "${ACTION_REPO}" --name "${WORKFLOW_ARTIFACT_NAME}" --dir "${download_dir}"
114+
115+
tarball_path="${download_dir}/${RELEASE_ASSET_NAME}"
116+
if [[ ! -f "${tarball_path}" ]]; then
117+
echo "ERROR: Downloaded artifact did not contain expected tarball '${RELEASE_ASSET_NAME}'." >&2
118+
ls -la "${download_dir}" >&2
119+
exit 1
120+
fi
121+
122+
tar -xzf "${tarball_path}" -C "${unpack_dir}"
123+
echo "[ddb-publish] Bundle extracted from workflow artifact."
124+
125+
- name: Run publisher
126+
shell: bash
127+
run: |
128+
set -euo pipefail
129+
130+
echo "[ddb-publish] Starting publish run"
131+
echo "[ddb-publish] source='${{ inputs.source-path }}' env='${{ inputs.target-env }}' table='${{ inputs.table-name }}' force='${{ inputs.force }}' dry-run='${{ inputs.dry-run }}'"
132+
133+
node "${RUNNER_TEMP}/ddb-publish/unpacked/index.cjs" \
134+
--source "${{ inputs.source-path }}" \
135+
--env "${{ inputs.target-env }}" \
136+
--table "${{ inputs.table-name }}" \
137+
$([[ "${{ inputs.force }}" == "true" ]] && echo "--force") \
138+
$([[ "${{ inputs.dry-run }}" == "true" ]] && echo "--dry-run")
139+
140+
echo "[ddb-publish] Publish run completed"

0 commit comments

Comments
 (0)