Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Cloudera Employee

Recently I came around an interesting problem: how to use boto to get data from a secure bucket in a Jupyter notebook in Cloudera Machine Learning.

 

The missing piece was: I needed to get my code integrated with my AWS permissions given by IDBroker.

Since CML already authenticated me to Kerberos, all I need was getting the goods from IDBroker.

 

In this article, I will show you pseudo code on how to get these access keys both in bash and python.

Note: Special thanks to @Kevin Risden to whom I owe this article and many more things. 

Find your IDBroker URL

Regardless of the method, you will need to get the URL for your IDBroker host. This is done simply in the management console of your datalake. The following is an example:

Screen Shot 2020-05-05 at 9.17.52 PM.png

Getting Access Keys in bash

After you are connected to one of your cluster's node and ensure you kinit, run the following:

IDBROKER_DT="$(curl -s --negotiate -u: "https:/[IDBROKER_HOST]:8444/gateway/dt/knoxtoken/api/v1/token")"
IDBROKER_ACCESS_TOKEN="$(echo "$IDBROKER_DT" | python -c "import json,sys; print(json.load(sys.stdin)['access_token'])")"
IDBROKER_CREDENTIAL_OUTPUT="$(curl -s -H "Authorization: Bearer $IDBROKER_ACCESS_TOKEN" "https://[IDBROKER_HOST]:8444/gateway/aws-cab/cab/api/v1/credentials")"

The credentials can be found in the $IDBROKER_CREDENTIAL_OUTPUT variable.

Getting Access Keys in Python 

Before getting started, the following libraries are installed:

pip3 install requests requests-kerberos boto3

Then, run the following code:

import requests

from requests_kerberos import HTTPKerberosAuth
r = requests.get("https://[IDBROKER_URL]:8444/gateway/dt/knoxtoken/api/v1/token", auth=HTTPKerberosAuth())

url = "https://[IDBROKER_URL]:8444/gateway/aws-cab/cab/api/v1/credentials"
headers = {
    'Authorization': "Bearer "+ r.json()['access_token'],
    'cache-control': "no-cache"
    }

response = requests.request("GET", url, headers=headers)

ACCESS_KEY=response.json()['Credentials']['AccessKeyId']
SECRET_KEY=response.json()['Credentials']['SecretAccessKey']
SESSION_TOKEN=response.json()['Credentials']['SessionToken']

import boto3
client = boto3.client(
    's3',
    aws_access_key_id=ACCESS_KEY,
    aws_secret_access_key=SECRET_KEY,
    aws_session_token=SESSION_TOKEN,
)

 You can then access your buckets via the following:

data = client.get_object(Bucket='[YOUR_BUCKET]', Key='[FILE_PATH]')
contents = data['Body'].read()
403 Views
Tags (2)
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
3 of 3
Last update:
‎06-10-2020 10:54 PM
Updated by:
 
Top Kudoed Authors