Skip to content

Commit c2cbdc6

Browse files
Warren-PittersonCopilot
Warren-Pitterson
and
Copilot
authored
feat: Participant Audit Log Table (#1881)
* feat: Audit Source Enum * feat: add queues package * feat: docker * feat: sln * feat: audit writer function * feat: migration * feat: EF model * feat: queue * feat: ParticipantAuditMessage model * feat: receive function queue implementation * test: tests * chore: removed unnecessary usings * chore: docker sonarqube security fix * Update application/CohortManager/src/Functions/Shared/Common/Extensions/AzureQueueExtension.cs Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * chore: PR comment reviews * chore: removed PII * chore: removed PII * chore: PII removed * feat: return changed to throw * feat: lazy initiation * test: updated tests * test: test covering the audit failure path. * feat: removed redundant Function suffix and changed raw_data_ref to be dd-mm-yyyy * test: added using, date change * feat: write to blob container, rename container, fallback * test: test added * chore: changed from CREATED_DATETIME to DATE_CREATED to match other schemas * chore: removed source from blobPath, * fix: sonarqube error handling * fix: sonarqube - dockerfile change * test: update test to new implementation * refactor: changed from queue to service bus trigger * feat: createdDatetime model changes * feat: participantAuditLogs made public * feat: index changes * test: unit tests * test: unit test * chore: remove unnecessary using * feat: replaced auditQueueSender for IQueueClient * chore: queue renamed * feat: removed lazy loading and added BlobStorageHelper Method * chore: typo * test: updated tests * chore: whitespace and hardcoded value changed to enum * chore: whitespace * feat: add environment variables to dockerfile * feat: Sequential audit sends in batches * feat: Queues added to service-bus config, graceful fallback on durablefunction * feat: revert null forgiving * feat: added AuditLogClient, refactored logic, renamed Config file * feat: removed queue from config, added sub to compose * refactor: ServiceBusClient * feat: Configuration injection * test: unit tests * feat: renamed participant-audit-queue to participant-audit-topic * feat: return not throw * feat: terraform * test: updated tests * feat: removed repeat call, moved logic to common, refactor AddBatchAsync * test: unit test to new implementation * test: removed unnecessary parameters * fix: trailing comma * test: removed old parameter from constructor * chore: SonarQube removed redundant jump * refactor: AddBatchAsync refactored with using to remove dispose and for clarity * feat: removed unneeded registration * feat: Ioptions for config --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
1 parent 8eccf78 commit c2cbdc6

43 files changed

Lines changed: 2427 additions & 268 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎Directory.Packages.props‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,7 @@
2020
<PackageVersion Include="Azure.Security.KeyVault.Secrets" Version="4.6.0" />
2121
<PackageVersion Include="Azure.Storage.Blobs" Version="12.20.0" />
2222
<PackageVersion Include="Azure.Storage.Queues" Version="12.20.1" />
23+
<PackageVersion Include="Microsoft.Azure.Functions.Worker.Extensions.Storage.Queues" Version="5.5.0" />
2324
<PackageVersion Include="Azure.Messaging.EventGrid" Version="4.28.0" />
2425
<PackageVersion Include="Microsoft.ApplicationInsights.DependencyCollector" Version="2.23.0" />
2526
<PackageVersion Include="Microsoft.ApplicationInsights.WorkerService" Version="2.23.0" />

‎application/CohortManager/Set-up/service-bus/config.json‎

Lines changed: 35 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -3,18 +3,41 @@
33
"Namespaces": [
44
{
55
"Name": "sbemulatorns",
6-
"Topics": [
7-
{
8-
"Name": "create-exception-topic",
9-
"Properties": {
10-
"DefaultMessageTimeToLive": "PT1H",
11-
"DuplicateDetectionHistoryTimeWindow": "PT20S",
12-
"RequiresDuplicateDetection": false
13-
},
14-
"Subscriptions": [
15-
{
16-
"Name": "create-exception-sub",
17-
"Properties": {
6+
"Queues": [],
7+
"Topics": [
8+
{
9+
"Name": "participant-audit-topic",
10+
"Properties": {
11+
"DefaultMessageTimeToLive": "PT1H",
12+
"DuplicateDetectionHistoryTimeWindow": "PT20S",
13+
"RequiresDuplicateDetection": false
14+
},
15+
"Subscriptions": [
16+
{
17+
"Name": "audit-writer-sub",
18+
"Properties": {
19+
"DeadLetteringOnMessageExpiration": false,
20+
"DefaultMessageTimeToLive": "PT1H",
21+
"LockDuration": "PT1M",
22+
"MaxDeliveryCount": 10,
23+
"ForwardDeadLetteredMessagesTo": "",
24+
"ForwardTo": "",
25+
"RequiresSession": false
26+
}
27+
}
28+
]
29+
},
30+
{
31+
"Name": "create-exception-topic",
32+
"Properties": {
33+
"DefaultMessageTimeToLive": "PT1H",
34+
"DuplicateDetectionHistoryTimeWindow": "PT20S",
35+
"RequiresDuplicateDetection": false
36+
},
37+
"Subscriptions": [
38+
{
39+
"Name": "create-exception-sub",
40+
"Properties": {
1841
"DeadLetteringOnMessageExpiration": false,
1942
"DefaultMessageTimeToLive": "PT1H",
2043
"LockDuration": "PT1M",

‎application/CohortManager/compose.core.yaml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -197,6 +197,23 @@ services:
197197
- ServiceBusConnectionString_client_internal=Endpoint=sb://service-bus;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;
198198
- ServiceNowParticipantManagementTopic=servicenow-participant-management-topic
199199
- RetrievePdsDemographicURL=http://retrieve-pds-demographic:8082/api/RetrievePDSDemographic
200+
# Audit Services
201+
audit-writer:
202+
container_name: audit-writer
203+
image: cohort-manager-audit-writer
204+
networks: [cohman-network]
205+
build:
206+
context: ./src/Functions/
207+
dockerfile: AuditServices/AuditWriter/Dockerfile
208+
args:
209+
BASE_IMAGE: ${FUNCTION_BASE_IMAGE}
210+
environment:
211+
- AzureWebJobsStorage=${AZURITE_CONNECTION_STRING}
212+
- DtOsDatabaseConnectionString=Server=db,1433;Database=${DB_NAME};User Id=SA;Password=${PASSWORD};TrustServerCertificate=True
213+
- ServiceBusConnectionString=Endpoint=sb://service-bus;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;
214+
- AuditTopicName=participant-audit-topic
215+
- AuditSubscription=audit-writer-sub
216+
200217

201218
# Screening Data Service
202219
create-exception:

‎application/CohortManager/src/Functions/AuditServices/AuditWriter/AuditConfig.cs‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
namespace NHS.CohortManager.AuditServices;
2+
3+
using System.ComponentModel.DataAnnotations;
4+
5+
public class AuditConfig
6+
{
7+
[Required]
8+
public required string ServiceBusConnectionString { get; set; }
9+
[Required]
10+
public required string AzureWebJobsStorage { get; set; }
11+
[Required]
12+
public required string AuditTopicName { get; set; }
13+
}

‎application/CohortManager/src/Functions/AuditServices/AuditWriter/AuditWriter.cs‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
namespace NHS.CohortManager.AuditServices;
2+
3+
using System.Text.Json;
4+
using DataServices.Database;
5+
using Microsoft.Azure.Functions.Worker;
6+
using Microsoft.Extensions.Logging;
7+
using Model;
8+
9+
public class AuditWriter
10+
{
11+
private static readonly JsonSerializerOptions JsonOptions = new()
12+
{
13+
PropertyNameCaseInsensitive = true
14+
};
15+
16+
private readonly DataServicesContext _dbContext;
17+
private readonly ILogger<AuditWriter> _logger;
18+
19+
public AuditWriter(DataServicesContext dbContext, ILogger<AuditWriter> logger)
20+
{
21+
_dbContext = dbContext;
22+
_logger = logger;
23+
}
24+
25+
[Function(nameof(AuditWriter))]
26+
public async Task Run(
27+
[ServiceBusTrigger(topicName: "%AuditTopicName%", subscriptionName: "%AuditSubscription%", Connection = "ServiceBusConnectionString")] string messageText, FunctionContext context)
28+
{
29+
ParticipantAuditMessage? audit;
30+
try
31+
{
32+
audit = JsonSerializer.Deserialize<ParticipantAuditMessage>(messageText, JsonOptions);
33+
}
34+
catch (JsonException ex)
35+
{
36+
_logger.LogError(ex, "Failed to deserialise audit message.");
37+
return;
38+
}
39+
40+
if (audit is null)
41+
{
42+
_logger.LogError("Audit message deserialised to null.");
43+
return;
44+
}
45+
46+
var auditLog = new ParticipantAuditLog
47+
{
48+
CorrelationId = audit.CorrelationId,
49+
NhsNumber = audit.NhsNumber,
50+
BatchId = audit.BatchId,
51+
CreatedDatetime = audit.CreatedDatetime,
52+
RecordSource = (int)audit.Source,
53+
RecordSourceDesc = audit.RecordSourceDesc,
54+
CreatedBy = audit.CreatedBy,
55+
ScreeningId = audit.ScreeningId,
56+
RawDataRef = audit.RawDataRef
57+
};
58+
59+
try
60+
{
61+
_dbContext.participantAuditLogs.Add(auditLog);
62+
var rowsAffected = await _dbContext.SaveChangesAsync();
63+
64+
if (rowsAffected <= 0)
65+
{
66+
_logger.LogError(
67+
"SaveChangesAsync reported 0 rows affected for CorrelationId {CorrelationId}.",
68+
audit.CorrelationId);
69+
return;
70+
}
71+
72+
_logger.LogInformation(
73+
"Audit written | Source: {Source} | Correlation: {CorrelationId} | Rows: {Rows}",
74+
audit.Source, audit.CorrelationId, rowsAffected);
75+
}
76+
catch (Exception ex)
77+
{
78+
_logger.LogError(ex,
79+
"Failed to persist audit log for CorrelationId {CorrelationId}.",
80+
audit.CorrelationId);
81+
}
82+
}
83+
}

‎application/CohortManager/src/Functions/AuditServices/AuditWriter/AuditWriter.csproj‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
<Project Sdk="Microsoft.NET.Sdk">
2+
<PropertyGroup>
3+
<ProjectGuid>{A1B2C3D4-E5F6-7890-ABCD-EF1234567890}</ProjectGuid>
4+
<TargetFramework>net8.0</TargetFramework>
5+
<AzureFunctionsVersion>v4</AzureFunctionsVersion>
6+
<OutputType>Exe</OutputType>
7+
<ImplicitUsings>enable</ImplicitUsings>
8+
<Nullable>enable</Nullable>
9+
</PropertyGroup>
10+
<ItemGroup>
11+
<PackageReference Include="Microsoft.Azure.Functions.Worker" />
12+
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" />
13+
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" />
14+
<PackageReference Include="Microsoft.Extensions.Diagnostics.HealthChecks" />
15+
<PackageReference Include="Azure.Storage.Blobs" />
16+
<PackageReference Include="Azure.Messaging.ServiceBus" />
17+
</ItemGroup>
18+
<ItemGroup>
19+
<None Update="host.json">
20+
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
21+
</None>
22+
<None Update="local.settings.json">
23+
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
24+
<CopyToPublishDirectory>Never</CopyToPublishDirectory>
25+
</None>
26+
</ItemGroup>
27+
<ItemGroup>
28+
<Using Include="System.Threading.ExecutionContext" Alias="ExecutionContext" />
29+
</ItemGroup>
30+
<ItemGroup>
31+
<ProjectReference Include="..\..\Shared\DataServices.Database\DataServices.Database.csproj" />
32+
<ProjectReference Include="..\..\Shared\DataServices.Core\DataServices.Core.csproj" />
33+
<ProjectReference Include="..\..\Shared\Common\Common.csproj" />
34+
<ProjectReference Include="..\..\Shared\HealthChecks\HealthChecks.csproj" />
35+
</ItemGroup>
36+
</Project>

‎application/CohortManager/src/Functions/AuditServices/AuditWriter/Dockerfile‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
ARG BASE_IMAGE
2+
FROM $BASE_IMAGE AS function
3+
4+
COPY ./AuditServices/AuditWriter /app/src/dotnet-function-app
5+
WORKDIR /app/src/dotnet-function-app
6+
7+
RUN --mount=type=cache,target=/root/.nuget/packages \
8+
dotnet publish ./*.csproj --output /home/site/wwwroot
9+
10+
# To enable ssh & remote debugging on app service change the base image to the one below
11+
# FROM mcr.microsoft.com/azure-functions/dotnet-isolated:4-dotnet-isolated8.0-appservice
12+
FROM mcr.microsoft.com/azure-functions/dotnet-isolated:4-dotnet-isolated8.0
13+
ENV AzureWebJobsScriptRoot=/home/site/wwwroot \
14+
AzureFunctionsJobHost__Logging__Console__IsEnabled=true
15+
16+
COPY --from=function ["/home/site/wwwroot", "/home/site/wwwroot"]

‎application/CohortManager/src/Functions/AuditServices/AuditWriter/Program.cs‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
using Microsoft.Extensions.Hosting;
2+
using Microsoft.Extensions.DependencyInjection;
3+
using DataServices.Core;
4+
using DataServices.Database;
5+
using HealthChecks.Extensions;
6+
using Common;
7+
using NHS.CohortManager.AuditServices;
8+
9+
var host = new HostBuilder()
10+
.ConfigureFunctionsWorkerDefaults()
11+
.AddConfiguration<AuditConfig>(out AuditConfig auditConfig)
12+
.AddDataServicesHandler<DataServicesContext>()
13+
.AddServiceBusClient(auditConfig.ServiceBusConnectionString)
14+
.ConfigureServices(services =>
15+
{
16+
services.AddDatabaseHealthCheck("AuditWriter");
17+
})
18+
.AddTelemetry()
19+
.Build();
20+
21+
await host.RunAsync();

‎application/CohortManager/src/Functions/CaasIntegration/receiveCaasFile/ProcessFileClasses/CallDurableDemographicFunc.cs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,7 @@ public async Task<bool> PostDemographicDataAsync(List<ParticipantDemographic> pa
6565
var response = await _httpClientFunction.SendPost(DemographicFunctionURI, content);
6666

6767
responseContent = response.Headers.Location!.ToString();
68+
6869
// This is not retrying the function if it fails but checking if it has done yet.
6970
var retryPolicy = Policy
7071
.HandleResult<WorkFlowStatus>(status => status != WorkFlowStatus.Completed && status != WorkFlowStatus.Failed)

‎application/CohortManager/src/Functions/CaasIntegration/receiveCaasFile/Program.cs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
using Microsoft.Azure.Functions.Worker;
21
using Microsoft.Extensions.Hosting;
32
using Microsoft.Extensions.DependencyInjection;
43
using Common;
@@ -9,7 +8,6 @@
98
using Model;
109
using DataServices.Client;
1110
using HealthChecks.Extensions;
12-
using Microsoft.Extensions.Options;
1311

1412

1513
var loggerFactory = LoggerFactory.Create(builder => builder.AddConsole());
@@ -19,6 +17,7 @@
1917
{
2018
var host = new HostBuilder()
2119
.AddConfiguration<ReceiveCaasFileConfig>(out ReceiveCaasFileConfig config)
20+
.AddConfiguration<AuditClientConfig>()
2221
.AddDataServicesHandler()
2322
.AddDataService<ParticipantDemographic>(config.DemographicDataServiceURL)
2423
.AddCachedDataService<ScreeningLkp>(config.ScreeningLkpDataServiceURL)
@@ -38,6 +37,7 @@
3837
services.AddTransient<IBlobStorageHelper, BlobStorageHelper>();
3938
services.AddTransient<ICopyFailedBatchToBlob, CopyFailedBatchToBlob>();
4039
services.AddScoped<IValidateDates, ValidateDates>();
40+
services.AddTransient<IAuditLogClient, AuditLogClient>();
4141
// Register health checks
4242
services.AddBlobStorageHealthCheck("receiveCaasFile");
4343
})

0 commit comments

Comments
 (0)