Cloudera Data Platform provides two complementing services for data processing. Cloudera Data Engineering (CDE) for preparing and organising data to be consumed by Data Scientists in Cloudera Machine Learning (CML). This article provides a high level example of how to call CDE from CML in Python.
## Part 0: Imports # Requests will be used to make REST requests to the CDE service # JSON is used to manage payloads and responses # REST calls are made over HTTP and use BasicAuth # cde_endpoint is set to the high level URI to the CDE cluster endpoint
import requests import json from requests.auth import HTTPBasicAuth
The first step is establishing the credentials required to call the API. This is done by calling the Knox proxy and requesting a token. The token is then passed using the "Authorization" header.
## Part 1: Connect to Knox service. # Retrieve the JWT token and parse it for the 'access_token' part. # Need to pass workload username and password which is set as a project ENV variable.