Skip to content

Commit 29a0414

Browse files
Ian-Hodgesgareth-allan
Ian-Hodges
and
gareth-allan
authored
CCM-13295: add status reporter component (#190)
* CCM-13295: add basic status reporter component * CCM-13295: Update workspace config to include new Node applications in Jest config --------- Co-authored-by: Gareth Allan <157592212+gareth-allan@users.noreply.github.com>
1 parent 80cec66 commit 29a0414

26 files changed

Lines changed: 1385 additions & 4 deletions

‎infrastructure/terraform/components/dl/README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ No requirements.
3131
| <a name="input_parent_acct_environment"></a> [parent\_acct\_environment](#input\_parent\_acct\_environment) | Name of the environment responsible for the acct resources used, affects things like DNS zone. Useful for named dev environments | `string` | `"main"` | no |
3232
| <a name="input_pdm_mock_access_token"></a> [pdm\_mock\_access\_token](#input\_pdm\_mock\_access\_token) | Mock access token for PDM API authentication (used in local/dev environments) | `string` | `"mock-pdm-token"` | no |
3333
| <a name="input_pdm_use_non_mock_token"></a> [pdm\_use\_non\_mock\_token](#input\_pdm\_use\_non\_mock\_token) | Whether to use the shared APIM access token from SSM (/component/environment/apim/access\_token) instead of the mock token | `bool` | `false` | no |
34+
| <a name="input_pii_data_retention_policy_days"></a> [pii\_data\_retention\_policy\_days](#input\_pii\_data\_retention\_policy\_days) | The number of days for data retention policy for PII | `number` | `534` | no |
3435
| <a name="input_project"></a> [project](#input\_project) | The name of the tfscaffold project | `string` | n/a | yes |
3536
| <a name="input_queue_batch_size"></a> [queue\_batch\_size](#input\_queue\_batch\_size) | maximum number of queue items to process | `number` | `10` | no |
3637
| <a name="input_queue_batch_window_seconds"></a> [queue\_batch\_window\_seconds](#input\_queue\_batch\_window\_seconds) | maximum time in seconds between processing events | `number` | `1` | no |
@@ -53,11 +54,13 @@ No requirements.
5354
| <a name="module_pdm_uploader"></a> [pdm\_uploader](#module\_pdm\_uploader) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.29/terraform-lambda.zip | n/a |
5455
| <a name="module_print_analyser"></a> [print\_analyser](#module\_print\_analyser) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.29/terraform-lambda.zip | n/a |
5556
| <a name="module_print_status_handler"></a> [print\_status\_handler](#module\_print\_status\_handler) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.29/terraform-lambda.zip | n/a |
57+
| <a name="module_report_event_transformer"></a> [report\_event\_transformer](#module\_report\_event\_transformer) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.29/terraform-lambda.zip | n/a |
5658
| <a name="module_s3bucket_cf_logs"></a> [s3bucket\_cf\_logs](#module\_s3bucket\_cf\_logs) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.24/terraform-s3bucket.zip | n/a |
5759
| <a name="module_s3bucket_file_safe"></a> [s3bucket\_file\_safe](#module\_s3bucket\_file\_safe) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.24/terraform-s3bucket.zip | n/a |
5860
| <a name="module_s3bucket_letters"></a> [s3bucket\_letters](#module\_s3bucket\_letters) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.24/terraform-s3bucket.zip | n/a |
5961
| <a name="module_s3bucket_non_pii_data"></a> [s3bucket\_non\_pii\_data](#module\_s3bucket\_non\_pii\_data) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.24/terraform-s3bucket.zip | n/a |
6062
| <a name="module_s3bucket_pii_data"></a> [s3bucket\_pii\_data](#module\_s3bucket\_pii\_data) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.24/terraform-s3bucket.zip | n/a |
63+
| <a name="module_s3bucket_reporting"></a> [s3bucket\_reporting](#module\_s3bucket\_reporting) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.24/terraform-s3bucket.zip | n/a |
6164
| <a name="module_s3bucket_static_assets"></a> [s3bucket\_static\_assets](#module\_s3bucket\_static\_assets) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.24/terraform-s3bucket.zip | n/a |
6265
| <a name="module_sqs_core_notifier"></a> [sqs\_core\_notifier](#module\_sqs\_core\_notifier) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.24/terraform-sqs.zip | n/a |
6366
| <a name="module_sqs_event_publisher_errors"></a> [sqs\_event\_publisher\_errors](#module\_sqs\_event\_publisher\_errors) | https://github.com/NHSDigital/nhs-notify-shared-modules/releases/download/v2.0.24/terraform-sqs.zip | n/a |

‎infrastructure/terraform/components/dl/athena_workgroup_reporting.tf‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
resource "aws_athena_workgroup" "reporting" {
2+
name = local.csi
3+
description = "Athena Workgroup for ${var.environment}"
4+
force_destroy = true
5+
6+
configuration {
7+
enforce_workgroup_configuration = true
8+
9+
result_configuration {
10+
expected_bucket_owner = var.aws_account_id
11+
output_location = "s3://${module.s3bucket_reporting.bucket}/athena-output/"
12+
13+
encryption_configuration {
14+
encryption_option = "SSE_KMS"
15+
kms_key_arn = module.kms.key_arn
16+
}
17+
}
18+
}
19+
}

‎infrastructure/terraform/components/dl/aws_iam_policy_document_eventbridge_firehose_assume_role.tf‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
# IAM role for EventBridge to write to Kinesis Firehose
2+
data "aws_iam_policy_document" "eventbridge_firehose_assume_role" {
3+
statement {
4+
effect = "Allow"
5+
6+
principals {
7+
type = "Service"
8+
identifiers = ["events.amazonaws.com"]
9+
}
10+
11+
actions = ["sts:AssumeRole"]
12+
}
13+
}
14+
15+
resource "aws_iam_role" "eventbridge_firehose" {
16+
name = "${local.csi}-eventbridge-firehose"
17+
description = "Role for EventBridge to write to Kinesis Firehose"
18+
assume_role_policy = data.aws_iam_policy_document.eventbridge_firehose_assume_role.json
19+
}
20+
21+
data "aws_iam_policy_document" "eventbridge_firehose_policy" {
22+
statement {
23+
effect = "Allow"
24+
25+
actions = [
26+
"firehose:PutRecord",
27+
"firehose:PutRecordBatch"
28+
]
29+
30+
resources = [
31+
aws_kinesis_firehose_delivery_stream.to_s3_reporting.arn
32+
]
33+
}
34+
}
35+
36+
resource "aws_iam_role_policy" "eventbridge_firehose" {
37+
name = "${local.csi}-eventbridge-firehose"
38+
role = aws_iam_role.eventbridge_firehose.id
39+
policy = data.aws_iam_policy_document.eventbridge_firehose_policy.json
40+
}

‎infrastructure/terraform/components/dl/cloudwatch_event_rule_all_events.tf‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
resource "aws_cloudwatch_event_rule" "all_events" {
2+
name = "${local.csi}-all-events"
3+
description = "Event rule to match all Digital Letters events"
4+
event_bus_name = aws_cloudwatch_event_bus.main.name
5+
6+
event_pattern = jsonencode({
7+
"detail" : {
8+
"type" : [{
9+
"prefix" : "uk.nhs.notify.digital.letters."
10+
}]
11+
}
12+
})
13+
}
14+
15+
resource "aws_cloudwatch_event_target" "reporting_firehose" {
16+
rule = aws_cloudwatch_event_rule.all_events.name
17+
arn = aws_kinesis_firehose_delivery_stream.to_s3_reporting.arn
18+
role_arn = aws_iam_role.eventbridge_firehose.arn
19+
event_bus_name = aws_cloudwatch_event_bus.main.name
20+
}

‎infrastructure/terraform/components/dl/cloudwatch_log_group_kinesis_logs.tf‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
resource "aws_cloudwatch_log_group" "kinesis_logs" {
2+
name = "/aws/kinesisfirehose/${local.csi}-to-s3-reporting"
3+
retention_in_days = var.log_retention_in_days
4+
}
5+
6+
resource "aws_cloudwatch_log_stream" "reporting_kinesis_logs" {
7+
name = "${local.csi}reportingKinesisLogs"
8+
log_group_name = aws_cloudwatch_log_group.kinesis_logs.name
9+
}

‎infrastructure/terraform/components/dl/glue_catalog_database_reporting.tf‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
resource "aws_glue_catalog_database" "reporting" {
2+
name = "${local.csi}-reporting"
3+
description = "Reporting database for ${var.environment}"
4+
}

‎infrastructure/terraform/components/dl/glue_catalog_table_event_record.tf‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
resource "aws_glue_catalog_table" "event_record" {
2+
name = "event_record"
3+
description = "Event records for ${var.environment}"
4+
database_name = aws_glue_catalog_database.reporting.name
5+
6+
table_type = "EXTERNAL_TABLE"
7+
8+
storage_descriptor {
9+
location = "s3://${module.s3bucket_reporting.bucket}/${local.firehose_output_path_prefix}/reporting/parquet/event_record"
10+
11+
input_format = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"
12+
output_format = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat"
13+
14+
ser_de_info {
15+
serialization_library = "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"
16+
}
17+
18+
# additional columns must be added at the end of the list
19+
columns {
20+
name = "messagereference"
21+
type = "string"
22+
}
23+
columns {
24+
name = "pagecount"
25+
type = "int"
26+
}
27+
columns {
28+
name = "supplierid"
29+
type = "string"
30+
}
31+
columns {
32+
name = "time"
33+
type = "string"
34+
}
35+
columns {
36+
name = "type"
37+
type = "string"
38+
}
39+
}
40+
41+
partition_keys {
42+
name = "senderid"
43+
type = "string"
44+
}
45+
46+
partition_keys {
47+
name = "__year"
48+
type = "int"
49+
}
50+
partition_keys {
51+
name = "__month"
52+
type = "int"
53+
}
54+
partition_keys {
55+
name = "__day"
56+
type = "int"
57+
}
58+
59+
parameters = {
60+
EXTERNAL = "TRUE"
61+
"parquet.compression" = "SNAPPY"
62+
compressionType = "none"
63+
classification = "parquet"
64+
}
65+
}
66+
67+
resource "aws_glue_partition_index" "event_record" {
68+
database_name = aws_glue_catalog_database.reporting.name
69+
table_name = aws_glue_catalog_table.event_record.name
70+
71+
partition_index {
72+
index_name = "data"
73+
keys = ["senderid", "__year", "__month", "__day"]
74+
}
75+
76+
timeouts {
77+
create = "60m"
78+
delete = "60m"
79+
}
80+
}

0 commit comments

Comments
 (0)