Skip to content

Commit 0331745

Browse files
SamAinsworth-NHSSamAinsworth-NHS
authored
feat: DTOSS-10479 add nems poison container and logic (#1440)
* feat: added infrastructure for nems-poison container in blob storage * feat: if Nems PDS Update processing fails, copy file to nems-poison container * fix: update ServiceBusConnectionString to ServiceBusConnectionString_client_internal in PdsProcessorTests * feat: add support for handling failed NEMS updates by moving files to a poison container * fix: update ProcessNemsUpdateTests to ensure proper logging for poison file operations * test: enhance ProcessNemsUpdateTests to verify poison file handling for 404 responses * fix: removed blobstream logic from ProcessNemsUpdate as no longer needed * fix: streamline CopyToPoisonContainer method by using config for storage connection string * feat: changing config name from fileExceptions to NemsPoisonContainer - to separate from caas function fully * refactor: remove NemsMeshPoisonContainer from config and related tests in NemsMeshRetrieval function * fix: remove unnecessary whitespace in NemsMeshRetrievalTests for cleaner code * fix: remove unnecessary whitespace in NemsMeshRetrievalTests for cleaner code * feat: add timestamp option to CopyFileToPoisonAsync overload for improved file management * feat: add unit tests for BlobStorageHelper including timestamp generation and file copying functionality * feat: add unit tests for UploadFileToBlobStorage and GetFileFromBlobStorage methods in BlobStorageHelper to improve code coverage * feat: add BlobStorageHelperTests project to solution for code coverage reporting * fix: update BlobStorageHelperTests project references and package versions * fix: avoid using throws in logic - moving files to poison container is now explicit * refactor: simplify CopyFileToPoisonAsync by delegating to overloaded method * fix: correcting unit test assert post changes * fix: corrected assert for error log in Run_FailsToRetrievePdsRecord_LogsError unit test
1 parent bc92f49 commit 0331745

16 files changed

Lines changed: 885 additions & 29 deletions

File tree

‎application/CohortManager/compose.core.yaml‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -53,8 +53,9 @@ services:
5353
profiles: [non-essential]
5454
environment:
5555
- ASPNETCORE_URLS=http://*:9083
56-
- caasfolder_STORAGE=${AZURITE_CONNECTION_STRING}
57-
- NemsMessages="nems-messages"
56+
- nemsmeshfolder_STORAGE=${AZURITE_CONNECTION_STRING}
57+
- NemsMessages=nems-updates
58+
- NemsPoisonContainer=nems-poison
5859
- ExceptionFunctionURL=http://create-exception:7070/api/CreateException
5960
- RetrievePdsDemographicURL=http://retrieve-pds-demographic:8082/api/RetrievePDSDemographic
6061
- UnsubscribeNemsSubscriptionUrl=http://manage-nems-subscription:9081/api/Unsubscribe

‎application/CohortManager/src/Functions/Functions.sln‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -233,6 +233,8 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "PdsProcesserTests", "PdsPro
233233
EndProject
234234
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "PdsProcessorTests", "..\..\..\..\tests\UnitTests\PdsProcessorTests\PdsProcessorTests.csproj", "{392B3D99-C5C5-DB9F-4DCA-F389E679C7C0}"
235235
EndProject
236+
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "BlobStorageHelperTests", "..\..\..\..\tests\UnitTests\SharedTests\BlobStorageHelperTests\BlobStorageHelperTests.csproj", "{BFA68329-98DD-4039-B009-C6DF23146765}"
237+
EndProject
236238
Global
237239
GlobalSection(SolutionConfigurationPlatforms) = preSolution
238240
Debug|Any CPU = Debug|Any CPU
@@ -1371,6 +1373,30 @@ Global
13711373
{59CBDBE5-29BE-F38C-80E6-40843F2F8AF6}.Release|x64.Build.0 = Release|Any CPU
13721374
{59CBDBE5-29BE-F38C-80E6-40843F2F8AF6}.Release|x86.ActiveCfg = Release|Any CPU
13731375
{59CBDBE5-29BE-F38C-80E6-40843F2F8AF6}.Release|x86.Build.0 = Release|Any CPU
1376+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
1377+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Debug|Any CPU.Build.0 = Debug|Any CPU
1378+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Debug|x64.ActiveCfg = Debug|Any CPU
1379+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Debug|x64.Build.0 = Debug|Any CPU
1380+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Debug|x86.ActiveCfg = Debug|Any CPU
1381+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Debug|x86.Build.0 = Debug|Any CPU
1382+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Release|Any CPU.ActiveCfg = Release|Any CPU
1383+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Release|Any CPU.Build.0 = Release|Any CPU
1384+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Release|x64.ActiveCfg = Release|Any CPU
1385+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Release|x64.Build.0 = Release|Any CPU
1386+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Release|x86.ActiveCfg = Release|Any CPU
1387+
{FC22C311-57DD-B069-4041-AD2AC8F80B5D}.Release|x86.Build.0 = Release|Any CPU
1388+
{BFA68329-98DD-4039-B009-C6DF23146765}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
1389+
{BFA68329-98DD-4039-B009-C6DF23146765}.Debug|Any CPU.Build.0 = Debug|Any CPU
1390+
{BFA68329-98DD-4039-B009-C6DF23146765}.Debug|x64.ActiveCfg = Debug|Any CPU
1391+
{BFA68329-98DD-4039-B009-C6DF23146765}.Debug|x64.Build.0 = Debug|Any CPU
1392+
{BFA68329-98DD-4039-B009-C6DF23146765}.Debug|x86.ActiveCfg = Debug|Any CPU
1393+
{BFA68329-98DD-4039-B009-C6DF23146765}.Debug|x86.Build.0 = Debug|Any CPU
1394+
{BFA68329-98DD-4039-B009-C6DF23146765}.Release|Any CPU.ActiveCfg = Release|Any CPU
1395+
{BFA68329-98DD-4039-B009-C6DF23146765}.Release|Any CPU.Build.0 = Release|Any CPU
1396+
{BFA68329-98DD-4039-B009-C6DF23146765}.Release|x64.ActiveCfg = Release|Any CPU
1397+
{BFA68329-98DD-4039-B009-C6DF23146765}.Release|x64.Build.0 = Release|Any CPU
1398+
{BFA68329-98DD-4039-B009-C6DF23146765}.Release|x86.ActiveCfg = Release|Any CPU
1399+
{BFA68329-98DD-4039-B009-C6DF23146765}.Release|x86.Build.0 = Release|Any CPU
13741400
{392B3D99-C5C5-DB9F-4DCA-F389E679C7C0}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
13751401
{392B3D99-C5C5-DB9F-4DCA-F389E679C7C0}.Debug|Any CPU.Build.0 = Debug|Any CPU
13761402
{392B3D99-C5C5-DB9F-4DCA-F389E679C7C0}.Debug|x64.ActiveCfg = Debug|Any CPU

‎application/CohortManager/src/Functions/NemsSubscriptionService/ProcessNemsUpdate/ProcessNemsUpdate.cs‎

Lines changed: 33 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,6 @@
1313
using Model;
1414
using DataServices.Client;
1515
using System.Net;
16-
using FluentValidation.Validators;
1716

1817
public class ProcessNemsUpdate
1918
{
@@ -24,6 +23,7 @@ public class ProcessNemsUpdate
2423
private readonly IHttpClientFunction _httpClientFunction;
2524
private readonly IExceptionHandler _exceptionHandler;
2625
private readonly IDataServiceClient<ParticipantDemographic> _participantDemographic;
26+
private readonly IBlobStorageHelper _blobStorageHelper;
2727
private readonly ProcessNemsUpdateConfig _config;
2828
private long nhsNumberLong;
2929

@@ -35,7 +35,8 @@ public ProcessNemsUpdate(
3535
IHttpClientFunction httpClientFunction,
3636
IExceptionHandler exceptionHandler,
3737
IDataServiceClient<ParticipantDemographic> participantDemographic,
38-
IOptions<ProcessNemsUpdateConfig> processNemsUpdateConfig)
38+
IOptions<ProcessNemsUpdateConfig> processNemsUpdateConfig,
39+
IBlobStorageHelper blobStorageHelper)
3940
{
4041
_logger = logger;
4142
_fhirPatientDemographicMapper = fhirPatientDemographicMapper;
@@ -45,6 +46,7 @@ public ProcessNemsUpdate(
4546
_exceptionHandler = exceptionHandler;
4647
_participantDemographic = participantDemographic;
4748
_config = processNemsUpdateConfig.Value;
49+
_blobStorageHelper = blobStorageHelper;
4850
}
4951

5052
/// <summary>
@@ -66,18 +68,26 @@ public async Task Run([BlobTrigger("nems-updates/{name}", Connection = "nemsmesh
6668
{
6769
var nhsNumber = await GetNhsNumberFromFile(blobStream, name);
6870

69-
if (!ValidationHelper.ValidateNHSNumber(nhsNumber!))
71+
if (nhsNumber == null)
7072
{
71-
_logger.LogError("There was a problem parsing the NHS number from blob store in the ProcessNemsUpdate function");
72-
throw new InvalidDataException("Invalid NHS Number");
73+
_logger.LogError("No NHS number found in file {FileName}. Moving to poison container.", name);
74+
await CopyToPoisonContainer(name);
75+
return;
76+
}
77+
78+
if (!ValidationHelper.ValidateNHSNumber(nhsNumber))
79+
{
80+
_logger.LogError("There was a problem validating the NHS number from blob store in the ProcessNemsUpdate function for file {FileName}. Moving to poison container.", name);
81+
await CopyToPoisonContainer(name);
82+
return;
7383
}
7484
nhsNumberLong = long.Parse(nhsNumber!);
7585

76-
var pdsResponse = await RetrievePdsRecord(nhsNumber!);
86+
var pdsResponse = await RetrievePdsRecord(nhsNumber);
7787
if (pdsResponse!.StatusCode == HttpStatusCode.NotFound)
7888
{
79-
_logger.LogError("the PDS function has returned a 404 error. function now stopping processing");
80-
// we can stop processing here as we know that not found means the participant ether needed an update or they were actually not found
89+
_logger.LogError("the PDS function has returned a 404 error for file {FileName}. Moving file to poison container.", name);
90+
await CopyToPoisonContainer(name);
8191
return;
8292
}
8393

@@ -92,15 +102,27 @@ public async Task Run([BlobTrigger("nems-updates/{name}", Connection = "nemsmesh
92102
}
93103
else
94104
{
95-
await UnsubscribeFromNems(nhsNumber!, retrievedPdsRecord!);
105+
await UnsubscribeFromNems(nhsNumber, retrievedPdsRecord!);
96106
}
97-
98107
}
99108
catch (Exception ex)
100109
{
101-
_logger.LogError(ex, "There was an error processing NEMS update.");
110+
_logger.LogError(ex, "There was an error processing NEMS update for file {FileName}. Moving to poison container.", name);
111+
try
112+
{
113+
await CopyToPoisonContainer(name);
114+
}
115+
catch (Exception poisonEx)
116+
{
117+
_logger.LogError(poisonEx, "Failed to copy NEMS file {FileName} to poison container. Manual intervention required.", name);
118+
}
102119
}
120+
}
103121

122+
private async Task CopyToPoisonContainer(string fileName)
123+
{
124+
await _blobStorageHelper.CopyFileToPoisonAsync(_config.nemsmeshfolder_STORAGE, fileName, _config.NemsMessages, _config.NemsPoisonContainer, addTimestamp: true);
125+
_logger.LogInformation("Copied failed NEMS file {FileName} to poison container with timestamp.", fileName);
104126
}
105127

106128
private async Task UnsubscribeFromNems(string nhsNumber, PdsDemographic retrievedPdsRecord)

‎application/CohortManager/src/Functions/NemsSubscriptionService/ProcessNemsUpdate/ProcessNemsUpdateConfig.cs‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,4 +17,7 @@ public class ProcessNemsUpdateConfig
1717
[Required]
1818
public required string DemographicDataServiceURL { get; set; }
1919

20+
[Required]
21+
public required string nemsmeshfolder_STORAGE { get; set; }
22+
public string NemsPoisonContainer { get; set; } = "nems-poison";
2023
}

‎application/CohortManager/src/Functions/NemsSubscriptionService/ProcessNemsUpdate/Program.cs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@
1818
services.AddSingleton<IFhirPatientDemographicMapper, FhirPatientDemographicMapper>();
1919
services.AddScoped<ICreateBasicParticipantData, CreateBasicParticipantData>();
2020
services.AddScoped<IAddBatchToQueue, AddBatchToQueue>();
21+
services.AddScoped<IBlobStorageHelper, BlobStorageHelper>();
2122
services.AddBlobStorageHealthCheck("ProcessNemsUpdate");
2223
})
2324
.AddTelemetry()

‎application/CohortManager/src/Functions/Shared/Common/BlobstorageHelper.cs‎

Lines changed: 21 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,13 @@ public BlobStorageHelper(ILogger<BlobStorageHelper> logger)
1515
_logger = logger;
1616
}
1717
public async Task CopyFileToPoisonAsync(string connectionString, string fileName, string containerName)
18+
{
19+
// Delegate to the extended overload to avoid duplication; preserve env var behaviour
20+
var poisonContainerName = Environment.GetEnvironmentVariable("fileExceptions");
21+
await CopyFileToPoisonAsync(connectionString, fileName, containerName, poisonContainerName, addTimestamp: false);
22+
}
23+
24+
public async Task CopyFileToPoisonAsync(string connectionString, string fileName, string containerName, string poisonContainerName, bool addTimestamp = false)
1825
{
1926
var sourceBlobServiceClient = new BlobServiceClient(connectionString);
2027
var sourceContainerClient = sourceBlobServiceClient.GetBlobContainerClient(containerName);
@@ -23,8 +30,19 @@ public async Task CopyFileToPoisonAsync(string connectionString, string fileName
2330
BlobLeaseClient sourceBlobLease = new(sourceBlobClient);
2431

2532
var destinationBlobServiceClient = new BlobServiceClient(connectionString);
26-
var destinationContainerClient = destinationBlobServiceClient.GetBlobContainerClient(Environment.GetEnvironmentVariable("fileExceptions"));
27-
var destinationBlobClient = destinationContainerClient.GetBlobClient(fileName);
33+
var destinationContainerClient = destinationBlobServiceClient.GetBlobContainerClient(poisonContainerName);
34+
35+
// Conditionally add timestamp to prevent collisions and maintain audit trail
36+
var destinationFileName = fileName;
37+
if (addTimestamp)
38+
{
39+
var timestamp = DateTime.UtcNow.ToString("yyyyMMdd_HHmmss");
40+
var fileExtension = Path.GetExtension(fileName);
41+
var fileNameWithoutExtension = Path.GetFileNameWithoutExtension(fileName);
42+
destinationFileName = $"{fileNameWithoutExtension}_{timestamp}{fileExtension}";
43+
}
44+
45+
var destinationBlobClient = destinationContainerClient.GetBlobClient(destinationFileName);
2846

2947
await destinationContainerClient.CreateIfNotExistsAsync(PublicAccessType.None);
3048

@@ -36,7 +54,7 @@ public async Task CopyFileToPoisonAsync(string connectionString, string fileName
3654
catch (RequestFailedException ex)
3755
{
3856
_logger.LogError(ex, "There has been a problem while copying the file: {Message}", ex.Message);
39-
throw;
57+
throw new InvalidOperationException($"Failed to copy file '{fileName}' from container '{containerName}' to poison container as '{destinationFileName}'.", ex);
4058
}
4159
finally
4260
{

‎application/CohortManager/src/Functions/Shared/Common/Interfaces/IBlobstorageHelper.cs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@ namespace Common;
55
public interface IBlobStorageHelper
66
{
77
Task CopyFileToPoisonAsync(string connectionString, string fileName, string containerName);
8+
Task CopyFileToPoisonAsync(string connectionString, string fileName, string containerName, string poisonContainerName, bool addTimestamp = false);
89

910
Task<bool> UploadFileToBlobStorage(string connectionString, string containerName, BlobFile blobFile, bool overwrite = false);
1011

‎infrastructure/tf-core/environments/development.tfvars‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -438,6 +438,10 @@ function_apps = {
438438
{
439439
env_var_name = "NemsMessages"
440440
container_name = "nems-updates"
441+
},
442+
{
443+
env_var_name = "NemsPoisonContainer"
444+
container_name = "nems-poison"
441445
}
442446
]
443447
env_vars_static = {
@@ -1348,6 +1352,9 @@ storage_accounts = {
13481352
nems-config = {
13491353
container_name = "nems-config"
13501354
}
1355+
nems-poison = {
1356+
container_name = "nems-poison"
1357+
}
13511358
}
13521359
}
13531360
}

‎infrastructure/tf-core/environments/integration.tfvars‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -344,6 +344,10 @@ function_apps = {
344344
{
345345
env_var_name = "NemsMessages"
346346
container_name = "nems-updates"
347+
},
348+
{
349+
env_var_name = "NemsPoisonContainer"
350+
container_name = "nems-poison"
347351
}
348352
]
349353
env_vars_static = {
@@ -1253,6 +1257,9 @@ storage_accounts = {
12531257
nems-config = {
12541258
container_name = "nems-config"
12551259
}
1260+
nems-poison = {
1261+
container_name = "nems-poison"
1262+
}
12561263
}
12571264
}
12581265
}

‎infrastructure/tf-core/environments/nft.tfvars‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -437,6 +437,10 @@ function_apps = {
437437
{
438438
env_var_name = "NemsMessages"
439439
container_name = "nems-updates"
440+
},
441+
{
442+
env_var_name = "NemsPoisonContainer"
443+
container_name = "nems-poison"
440444
}
441445
]
442446
env_vars_static = {
@@ -1346,6 +1350,9 @@ storage_accounts = {
13461350
nems-config = {
13471351
container_name = "nems-config"
13481352
}
1353+
nems-poison = {
1354+
container_name = "nems-poison"
1355+
}
13491356
}
13501357
}
13511358
}

0 commit comments

Comments
 (0)