Skip to content

Commit 8631401

Browse files
lapenna-bjsslapenna-bjss
committed
CCM-12616: mock mesh only when enabled
1 parent 94a2f08 commit 8631401

6 files changed

Lines changed: 302 additions & 19 deletions

File tree

‎infrastructure/terraform/components/dl/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,14 @@ No requirements.
1313
| <a name="input_component"></a> [component](#input\_component) | The variable encapsulating the name of this component | `string` | `"dl"` | no |
1414
| <a name="input_default_tags"></a> [default\_tags](#input\_default\_tags) | A map of default tags to apply to all taggable resources within the component | `map(string)` | `{}` | no |
1515
| <a name="input_enable_dynamodb_delete_protection"></a> [enable\_dynamodb\_delete\_protection](#input\_enable\_dynamodb\_delete\_protection) | Enable DynamoDB Delete Protection on all Tables | `bool` | `true` | no |
16+
| <a name="input_enable_mock_mesh"></a> [enable\_mock\_mesh](#input\_enable\_mock\_mesh) | Enable mock mesh access (dev only). Grants lambda permission to read mock-mesh prefix in non-pii bucket. | `bool` | `false` | no |
1617
| <a name="input_environment"></a> [environment](#input\_environment) | The name of the tfscaffold environment | `string` | n/a | yes |
1718
| <a name="input_force_lambda_code_deploy"></a> [force\_lambda\_code\_deploy](#input\_force\_lambda\_code\_deploy) | If the lambda package in s3 has the same commit id tag as the terraform build branch, the lambda will not update automatically. Set to True if making changes to Lambda code from on the same commit for example during development | `bool` | `false` | no |
1819
| <a name="input_group"></a> [group](#input\_group) | The group variables are being inherited from (often synonmous with account short-name) | `string` | n/a | yes |
1920
| <a name="input_kms_deletion_window"></a> [kms\_deletion\_window](#input\_kms\_deletion\_window) | When a kms key is deleted, how long should it wait in the pending deletion state? | `string` | `"30"` | no |
2021
| <a name="input_log_level"></a> [log\_level](#input\_log\_level) | The log level to be used in lambda functions within the component. Any log with a lower severity than the configured value will not be logged: https://docs.python.org/3/library/logging.html#levels | `string` | `"INFO"` | no |
2122
| <a name="input_log_retention_in_days"></a> [log\_retention\_in\_days](#input\_log\_retention\_in\_days) | The retention period in days for the Cloudwatch Logs events to be retained, default of 0 is indefinite | `number` | `0` | no |
22-
| <a name="input_mesh_poll_schedule"></a> [mesh\_poll\_schedule](#input\_mesh\_poll\_schedule) | Schedule to poll MESH for messages | `string` | `"cron(0,30 8-16 ? * MON-FRI *)"` | no |
23+
| <a name="input_mesh_poll_schedule"></a> [mesh\_poll\_schedule](#input\_mesh\_poll\_schedule) | Schedule to poll MESH for messages | `string` | `"rate(5 minutes)"` | no |
2324
| <a name="input_parent_acct_environment"></a> [parent\_acct\_environment](#input\_parent\_acct\_environment) | Name of the environment responsible for the acct resources used, affects things like DNS zone. Useful for named dev environments | `string` | `"main"` | no |
2425
| <a name="input_project"></a> [project](#input\_project) | The name of the tfscaffold project | `string` | n/a | yes |
2526
| <a name="input_queue_batch_size"></a> [queue\_batch\_size](#input\_queue\_batch\_size) | maximum number of queue items to process | `number` | `10` | no |

‎infrastructure/terraform/components/dl/module_lambda_mesh_download.tf‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,14 +36,60 @@ module "mesh_download" {
3636
log_subscription_role_arn = local.acct.log_subscription_role_arn
3737

3838
lambda_env_vars = {
39+
# Required by Config
3940
EVENT_PUBLISHER_EVENT_BUS_ARN = aws_cloudwatch_event_bus.main.arn
4041
EVENT_PUBLISHER_DLQ_URL = module.sqs_event_publisher_dlq.queue_url
4142
ENVIRONMENT = var.environment
4243
DEPLOYMENT = var.deployment
44+
45+
# Optional
46+
USE_MESH_MOCK = var.enable_mock_mesh ? "true" : "false"
47+
MOCK_MESH_BUCKET = module.s3bucket_non_pii_data.bucket
4348
}
49+
4450
}
4551

4652
data "aws_iam_policy_document" "mesh_download_lambda" {
53+
# Mock S3 ListBucket only when enabled
54+
dynamic "statement" {
55+
for_each = var.enable_mock_mesh ? [1] : []
56+
content {
57+
sid = "MockMeshListBucket"
58+
effect = "Allow"
59+
60+
actions = [
61+
"s3:ListBucket"
62+
]
63+
64+
resources = [
65+
module.s3bucket_non_pii_data.arn
66+
]
67+
68+
condition {
69+
test = "StringLike"
70+
variable = "s3:prefix"
71+
values = ["mock-mesh/*"]
72+
}
73+
}
74+
}
75+
76+
# Mock S3 GetObject only when enabled
77+
dynamic "statement" {
78+
for_each = var.enable_mock_mesh ? [1] : []
79+
content {
80+
sid = "MockMeshGetObject"
81+
effect = "Allow"
82+
83+
actions = [
84+
"s3:GetObject"
85+
]
86+
87+
resources = [
88+
"${module.s3bucket_non_pii_data.arn}/mock-mesh/*"
89+
]
90+
}
91+
}
92+
4793
statement {
4894
sid = "KMSPermissions"
4995
effect = "Allow"

‎infrastructure/terraform/components/dl/module_lambda_mesh_poll.tf‎

Lines changed: 57 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -37,35 +37,78 @@ module "mesh_poll" {
3737
log_subscription_role_arn = local.acct.log_subscription_role_arn
3838

3939
lambda_env_vars = {
40+
# Required by Config
41+
SSM_PREFIX = var.ssm_prefix
42+
SSM_CLIENTS_PARAMETER_PATH = var.ssm_clients_parameter_path
43+
INBOX_WORKFLOW_ID = var.inbox_workflow_id
44+
OUTBOX_WORKFLOW_ID = var.outbox_workflow_id
45+
MAXIMUM_RUNTIME_MILLISECONDS = var.maximum_runtime_milliseconds
46+
ENVIRONMENT = var.environment
47+
EVENT_PUBLISHER_EVENT_BUS_ARN = aws_cloudwatch_event_bus.main.arn
48+
CERTIFICATE_EXPIRY_METRIC_NAME = var.certificate_expiry_metric_name
49+
CERTIFICATE_EXPIRY_METRIC_NAMESPACE = var.certificate_expiry_metric_namespace
50+
POLLING_METRIC_NAME = var.polling_metric_name
51+
POLLING_METRIC_NAMESPACE = var.polling_metric_namespace
52+
53+
# Optional
54+
USE_MESH_MOCK = var.enable_mock_mesh ? "true" : "false"
55+
MOCK_MESH_BUCKET = module.s3bucket_non_pii_data.bucket
4056
}
57+
4158
}
4259

4360
data "aws_iam_policy_document" "mesh_poll_lambda" {
44-
statement {
45-
sid = "KMSPermissions"
46-
effect = "Allow"
61+
# Mock S3 ListBucket only when enabled
62+
dynamic "statement" {
63+
for_each = var.enable_mock_mesh ? [1] : []
64+
content {
65+
sid = "MockMeshListBucket"
66+
effect = "Allow"
4767

48-
actions = [
49-
"kms:Decrypt",
50-
"kms:GenerateDataKey",
51-
]
68+
actions = [
69+
"s3:ListBucket"
70+
]
5271

53-
resources = [
54-
module.kms.key_arn,
55-
]
72+
resources = [
73+
module.s3bucket_non_pii_data.arn
74+
]
75+
76+
condition {
77+
test = "StringLike"
78+
variable = "s3:prefix"
79+
values = ["mock-mesh/*"]
80+
}
81+
}
82+
}
83+
84+
# Mock S3 GetObject only when enabled
85+
dynamic "statement" {
86+
for_each = var.enable_mock_mesh ? [1] : []
87+
content {
88+
sid = "MockMeshGetObject"
89+
effect = "Allow"
90+
91+
actions = [
92+
"s3:GetObject"
93+
]
94+
95+
resources = [
96+
"${module.s3bucket_non_pii_data.arn}/mock-mesh/*"
97+
]
98+
}
5699
}
57100

58101
statement {
59-
sid = "LettersBucketPermissions"
102+
sid = "KMSPermissions"
60103
effect = "Allow"
61104

62105
actions = [
63-
"s3:ListBucket",
64-
"s3:GetObject",
106+
"kms:Decrypt",
107+
"kms:GenerateDataKey",
65108
]
66109

67110
resources = [
68-
"${module.s3bucket_letters.arn}/mock-mesh/*",
111+
module.kms.key_arn,
69112
]
70113
}
71114
}

‎infrastructure/terraform/components/dl/variables.tf‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -89,7 +89,13 @@ variable "parent_acct_environment" {
8989
variable "mesh_poll_schedule" {
9090
type = string
9191
description = "Schedule to poll MESH for messages"
92-
default = "cron(0,30 8-16 ? * MON-FRI *)" # Every 30 minutes between 8am and 4:30pm Mon-Fri
92+
default = "rate(5 minutes)" # Every 5 minutes
93+
}
94+
95+
variable "enable_mock_mesh" {
96+
description = "Enable mock mesh access (dev only). Grants lambda permission to read mock-mesh prefix in non-pii bucket."
97+
type = bool
98+
default = false
9399
}
94100

95101
variable "queue_batch_size" {

‎lambdas/mesh-download/src/config.py‎

Lines changed: 188 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,188 @@
1+
"""
2+
Module for configuring MESH Download application
3+
"""
4+
import json
5+
import os
6+
import tempfile
7+
import boto3
8+
import structlog
9+
import mesh_client
10+
from py_mock_mesh.mesh_client import MockMeshClient
11+
from metric_publishers.certificate_monitor import report_expiry_time
12+
from metric_publishers.metric_client import Metric
13+
14+
structlog.configure(processors=[structlog.processors.JSONRenderer()])
15+
log = structlog.get_logger()
16+
17+
18+
class InvalidMeshEndpointError(Exception):
19+
"""
20+
Indicates an invalid MESH endpoint in configuration
21+
"""
22+
23+
24+
class InvalidEnvironmentVariableError(Exception):
25+
"""
26+
Indicates an invalid environment variable
27+
"""
28+
29+
30+
def store_file(content):
31+
"""
32+
Writes a temp file and returns the name
33+
"""
34+
35+
with tempfile.NamedTemporaryFile(delete=False) as file:
36+
file.write(content)
37+
file.close()
38+
return file.name
39+
40+
41+
_REQUIRED_ENV_VAR_MAP = {
42+
"ssm_prefix": "SSM_PREFIX",
43+
"inbox_workflow_id": "INBOX_WORKFLOW_ID",
44+
"outbox_workflow_id": "OUTBOX_WORKFLOW_ID",
45+
"environment": "ENVIRONMENT",
46+
"certificate_expiry_metric_name": "CERTIFICATE_EXPIRY_METRIC_NAME",
47+
"certificate_expiry_metric_namespace": "CERTIFICATE_EXPIRY_METRIC_NAMESPACE",
48+
"polling_metric_name": "POLLING_METRIC_NAME",
49+
"polling_metric_namespace": "POLLING_METRIC_NAMESPACE"
50+
}
51+
52+
_OPTIONAL_ENV_VAR_MAP = {
53+
"use_mesh_mock": "USE_MESH_MOCK"
54+
}
55+
56+
57+
class Config: # pylint: disable=too-many-instance-attributes
58+
59+
"""
60+
Represents the configuration of the MESH Download application
61+
"""
62+
63+
def __init__(self,
64+
ssm=None,
65+
s3_client=None):
66+
67+
self.ssm = ssm if ssm is not None else boto3.client('ssm')
68+
self.s3_client = s3_client if s3_client is not None else boto3.client('s3')
69+
self.mesh_endpoint = None
70+
self.mesh_mailbox = None
71+
self.mesh_mailbox_password = None
72+
self.mesh_shared_key = None
73+
self.client_cert = None
74+
self.client_key = None
75+
self.mesh_client = None
76+
self.ssm_prefix = None
77+
self.inbox_workflow_id = None
78+
self.outbox_workflow_id = None
79+
self.environment = None
80+
self.certificate_expiry_metric_name = None
81+
self.certificate_expiry_metric_namespace = None
82+
self.use_mesh_mock = False
83+
84+
missing_env_vars = []
85+
for attr, key in _REQUIRED_ENV_VAR_MAP.items():
86+
if key not in os.environ:
87+
missing_env_vars.append(f'"{key}"')
88+
else:
89+
setattr(self, attr, os.environ[key])
90+
91+
# Handle optional environment variables
92+
for attr, key in _OPTIONAL_ENV_VAR_MAP.items():
93+
if key in os.environ:
94+
value = os.environ[key]
95+
if attr == "use_mesh_mock":
96+
# Convert string to boolean
97+
setattr(self, attr, value.lower() in ('true', '1', 'yes', 'on'))
98+
else:
99+
setattr(self, attr, value)
100+
101+
if len(missing_env_vars) > 0:
102+
raise InvalidEnvironmentVariableError(
103+
f"Required environment variables {', '.join(missing_env_vars)} not set.")
104+
105+
def __enter__(self):
106+
ssm_response = self.ssm.get_parameter(
107+
Name=self.ssm_prefix + '/config',
108+
WithDecryption=True
109+
)
110+
mesh_config = json.loads(
111+
ssm_response['Parameter']['Value']
112+
)
113+
114+
self.mesh_endpoint = mesh_config['mesh_endpoint']
115+
self.mesh_mailbox = mesh_config['mesh_mailbox']
116+
self.mesh_mailbox_password = mesh_config['mesh_mailbox_password']
117+
self.mesh_shared_key = mesh_config['mesh_shared_key'].encode('ascii')
118+
119+
# Build Mesh Client
120+
121+
client_cert_parameter = self.ssm.get_parameter(
122+
Name=self.ssm_prefix + '/client-cert',
123+
WithDecryption=True
124+
)
125+
client_key_parameter = self.ssm.get_parameter(
126+
Name=self.ssm_prefix + '/client-key',
127+
WithDecryption=True
128+
)
129+
130+
self.client_cert = store_file(
131+
client_cert_parameter['Parameter']['Value'].encode('utf-8')
132+
)
133+
self.client_key = store_file(
134+
client_key_parameter['Parameter']['Value'].encode('utf-8')
135+
)
136+
137+
self.mesh_client = self.build_mesh_client()
138+
139+
return self
140+
141+
def __exit__(self, exc_type, exc_value, traceback):
142+
log.info('Cleaning up temporary files')
143+
os.unlink(self.client_cert)
144+
os.unlink(self.client_key)
145+
self.mesh_client.close()
146+
147+
def lookup_endpoint(self, endpoint_identifier):
148+
"""
149+
Looks up a MESH endpoint URL
150+
"""
151+
152+
variable_name = f"{endpoint_identifier}_ENDPOINT"
153+
154+
if hasattr(mesh_client, variable_name):
155+
return getattr(mesh_client, variable_name)
156+
157+
raise InvalidMeshEndpointError(
158+
f"mesh_client module has no such endpoint {variable_name}")
159+
160+
def build_mesh_client(self):
161+
"""
162+
Returns a MESH client based on the USE_MESH_MOCK environment variable
163+
"""
164+
if self.use_mesh_mock:
165+
mock_bucket = os.getenv("MOCK_MESH_BUCKET")
166+
mock_endpoint = f"s3://{mock_bucket}/mock-mesh/"
167+
return MockMeshClient(
168+
boto3.client('s3'),
169+
mock_endpoint,
170+
self.mesh_mailbox,
171+
log
172+
)
173+
174+
# Use real MESH client
175+
report_expiry_time(
176+
self.client_cert,
177+
self.certificate_expiry_metric_name,
178+
self.certificate_expiry_metric_namespace,
179+
self.environment)
180+
181+
return mesh_client.MeshClient(
182+
self.lookup_endpoint(self.mesh_endpoint),
183+
self.mesh_mailbox,
184+
self.mesh_mailbox_password,
185+
self.mesh_shared_key,
186+
transparent_compress=True,
187+
cert=(self.client_cert, self.client_key)
188+
)

‎lambdas/mesh-poll/src/config.py‎

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -75,7 +75,7 @@ def __init__(self,
7575
self.polling_metric_name = None
7676
self.polling_metric_namespace = None
7777
self.polling_metric = None
78-
self.use_mesh_mock = None
78+
self.use_mesh_mock = False
7979

8080
missing_env_vars = []
8181
for attr, key in _REQUIRED_ENV_VAR_MAP.items():
@@ -162,8 +162,7 @@ def build_mesh_client(self):
162162
Returns a MESH client based on the USE_MESH_MOCK environment variable
163163
"""
164164
if self.use_mesh_mock:
165-
# Use mock MESH client with S3 bucket for dev/testing
166-
mock_bucket = f"{self.environment}-mock-mesh"
165+
mock_bucket = os.getenv("MOCK_MESH_BUCKET")
167166
mock_endpoint = f"s3://{mock_bucket}/mock-mesh/"
168167
return MockMeshClient(
169168
boto3.client('s3'),

0 commit comments

Comments
 (0)