Skip to content

VED-814 Resolve sonarcloud warnings on S3 bucket ownership #882

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
dlzhry2nhs merged 1 commit into from
Oct 7, 2025
Merged

VED-814 Resolve sonarcloud warnings on S3 bucket ownership #882

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 17 additions & 4 deletions mesh_processor/src/converter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@
import boto3
from smart_open import open

EXPECTED_BUCKET_OWNER_ACCOUNT = os.getenv("ACCOUNT_ID")
DESTINATION_BUCKET_NAME = os.getenv("DESTINATION_BUCKET_NAME")

logging.basicConfig(level=logging.INFO)
Expand Down Expand Up @@ -80,9 +81,15 @@ def move_file(source_bucket: str, source_key: str, destination_bucket: str, dest
s3_client.copy_object(
CopySource={"Bucket": source_bucket, "Key": source_key},
Bucket=destination_bucket,
Key=destination_key
Key=destination_key,
ExpectedBucketOwner=EXPECTED_BUCKET_OWNER_ACCOUNT,
ExpectedSourceBucketOwner=EXPECTED_BUCKET_OWNER_ACCOUNT
Copy link
Copy Markdown
Contributor

@Akol125 Akol125 Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious as to why two arguments references the same value

Copy link
Copy Markdown
Contributor Author

@dlzhry2nhs dlzhry2nhs Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's a good question. So for any copy request (you can check boto3 - S3) you can set both the bucket owner and source bucket i.e. the bucket you are copying from.

In both cases, these are buckets that belong in our account. Hope that makes sense.

Copy link
Copy Markdown
Contributor

@Akol125 Akol125 Oct 7, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hence the same account ID, makes total sense

)
s3_client.delete_object(
Bucket=source_bucket,
Key=source_key,
ExpectedBucketOwner=EXPECTED_BUCKET_OWNER_ACCOUNT
)
s3_client.delete_object(Bucket=source_bucket, Key=source_key)


def transfer_multipart_content(bucket_name: str, file_key: str, boundary: bytes, filename: str) -> None:
Expand Down Expand Up @@ -122,7 +129,11 @@ def process_record(record: dict) -> None:
file_key = record["s3"]["object"]["key"]
logger.info(f"Processing {file_key}")

response = s3_client.head_object(Bucket=bucket_name, Key=file_key)
response = s3_client.head_object(
Bucket=bucket_name,
Key=file_key,
ExpectedBucketOwner=EXPECTED_BUCKET_OWNER_ACCOUNT
)
content_type = response['ContentType']
media_type, content_type_params = parse_header_value(content_type)
filename = response["Metadata"].get("mex-filename") or file_key
Expand All @@ -136,7 +147,9 @@ def process_record(record: dict) -> None:
s3_client.copy_object(
Bucket=DESTINATION_BUCKET_NAME,
CopySource={"Bucket": bucket_name, "Key": file_key},
Key=filename
Key=filename,
ExpectedBucketOwner=EXPECTED_BUCKET_OWNER_ACCOUNT,
ExpectedSourceBucketOwner=EXPECTED_BUCKET_OWNER_ACCOUNT
)

logger.info(f"Transfer complete for {file_key}")
Expand Down
4 changes: 3 additions & 1 deletion mesh_processor/tests/test_converter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,8 @@
from botocore.exceptions import ClientError
from moto import mock_aws

MOCK_MOTO_ACCOUNT_ID = "123456789012"


def invoke_lambda(file_key: str):
# Local import so that globals can be mocked
Expand All @@ -26,7 +28,7 @@ def invoke_lambda(file_key: str):


@mock_aws
@patch.dict(os.environ, {"DESTINATION_BUCKET_NAME": "destination-bucket"})
@patch.dict(os.environ, {"DESTINATION_BUCKET_NAME": "destination-bucket", "ACCOUNT_ID": MOCK_MOTO_ACCOUNT_ID})
class TestLambdaHandler(TestCase):
def setUp(self):
s3 = boto3.client("s3", region_name="eu-west-2")
Expand Down
1 change: 1 addition & 0 deletions terraform/mesh_processor.tf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -217,6 +217,7 @@ resource "aws_lambda_function" "mesh_file_converter_lambda" {

environment {
variables = {
ACCOUNT_ID = var.immunisation_account_id
DESTINATION_BUCKET_NAME = aws_s3_bucket.batch_data_source_bucket.bucket
}
}
Expand Down
Loading