Troubleshooting AWS IAM permissions in CDP Public Cloud

[#1]

Introduction

If the AWS IAM roles are not configured correctly, and if you are running into issues using these IAM roles, the best way to troubleshoot it is by using the AWS CLI. This post contains the steps and the required setup for this.

Steps

If the data lake admin role is not configured correctly, you will see the following error during the data lake deployment.

Error: Failed to create HDFS directory., Command [CreateRangerAuditDir], with id [1546333865] failed: Failed to create HDFS directory s3a://<bucketname>/data/ranger/audit/hdfs., Command [RangerPluginCreateAuditDir].

To get to the root cause of the issue, check the S3 permissions on the IAM role. If you want to further troubleshoot, then you can assume the role from the ID Broker's instance profile and use AWS CLI to reproduce and troubleshoot the issue. To do this, use the following troubleshooting steps:

1. Log in to the ID broker host as a Cloudbreak user using the pem file provided during the environment build
2. Download this script

   wget https://raw.githubusercontent.com/karthikeyanvijay/cdp-publiccloud/main/aws/scripts/get-dladmin-access.sh 
   chmod u+x get-dladmin-access.sh​

3. Modify the role_arn in the script to point to the data lake admin role
4. Execute the script

   ./get-dladmin-access.sh

   The temporary AWS credentials for the role are now exported to the AWS profile dladmin
5. Use the AWS CLI to troubleshoot S3 IAM issues

   aws s3 -cp sample.txt s3://<bucketname-path>/<path> --profile dladmin​

   You should now be able to reproduce the same error that occurred during the CDP deployment using AWS CLI.
6. You can now tweak the permissions and come up with the right IAM permissions for the role. Resume the CDP deployment after fixing the permission issues.

Conclusion

Using the above process, you can accelerate troubleshooting and help narrow down permission issues. The example above shows troubleshooting issues with the data lake admin role, but the same process can be used to troubleshoot any AWS IAM roles.

-------------

Vijay Anand Karthikeyan