lakeformation

AWS Lake Formation CDK Python project to grant permissions on Data Catalog Resources

This is an AWS Lake Formation project for CDK development with Python.

This project shows how to grant AWS Lake Formation permissions on Data Catalog Resources.

The cdk.json file tells the CDK Toolkit how to execute your app.

This project is set up like a standard Python project. The initialization process also creates a virtualenv within this project, stored under the .venv directory. To create the virtualenv it assumes that there is a python3 (or python for Windows) executable in your path with access to the venv package. If for any reason the automatic creation of the virtualenv fails, you can create the virtualenv manually.

To manually create a virtualenv on MacOS and Linux:

$ python3 -m venv .venv

After the init process completes and the virtualenv is created, you can use the following step to activate your virtualenv.

$ source .venv/bin/activate

If you are a Windows platform, you would activate the virtualenv like this:

% .venv\Scripts\activate.bat

Once the virtualenv is activated, you can install the required dependencies.

(.venv) $ pip install -r requirements.txt

At this point you can now synthesize the CloudFormation template for this code.

(.venv) $ cdk synth -c database_name=lf_testdb

Use cdk deploy command to create the stack shown above.

(.venv) $ cdk deploy -c database_name=lf_testdb --require-approval never

ℹ️ Replace lf_testdb to your own database name in Glue Data Catalog.

To add additional dependencies, for example other CDK libraries, just add them to your setup.py file and rerun the pip install -r requirements.txt command.

After all CDK stacks are successfully deployed, you can see a list of permissions on lf_testdb to GlueJobRoleExample IAM Role.

aws lakeformation list-permissions | jq -r '.PrincipalResourcePermissions[] | select(.Principal.DataLakePrincipalIdentifier | endswith("GlueJobRoleExample"))'

ℹ️ GlueJobRoleExample is an IAM Role created by this project.

Clean Up

Delete the CloudFormation stack by running the below command.

(.venv) $ cdk destroy -c database_name=lf_testdb --force --all

Useful commands

• cdk ls list all stacks in the app
• cdk synth emits the synthesized CloudFormation template
• cdk deploy deploy this stack to your default AWS account/region
• cdk diff compare deployed stack with current state
• cdk docs open CDK documentation

Enjoy!

Useful AWS Lake Formation commands

• aws lakeformation list-permissions Returns a list of the principal permissions on the resource, filtered by the permissions of the caller. See more
• aws lakeformation grant-permissions Grants permissions to the principal to access metadata in the Data Catalog and data organized in underlying data storage such as Amazon S3. See more
• aws lakeformation revoke-permissions Revokes permissions to the principal to access metadata in the Data Catalog and data organized in underlying data storage such as Amazon S3. See more
• aws lakeformation list-resources Lists the resources registered to be managed by the Data Catalog. See more

References

• AWS Lake Formation Workshop
• AWS Lake Formation - Create a data lake administrator
• AWS Lake Formation - Granting Data Catalog permissions using the named resource method
• AWS Lake Formation Permissions Reference
• Troubleshooting Lake Formation
• Known issues for AWS Lake Formation
• AWS Lake Formation - Working with other AWS services