Programmatically tracking MR Job status using the Cloudera Manager API (and Python libs)?

[#1]

Hi Cloudera community:

tl;dr: Is there a more straighforward way to query the Cloudera Manager API to get information (mapper/reducer completion, bytes processed, etc) about jobs, perhaps by simply providing a jobId or a jobname?

I am using a Python script to check on the status of various MapReduce jobs, using the Cloudera Manager API, roughly something like this:

from cm_api.api_client import ApiResource
 
api = ApiResource('zzzz',
                        version=1,
                        username='zzz',
                        password='zzz')
 
for s in api.get_cluster('my cluster').get_all_services():
        if s.name == 'MR':
        # my activities are in s.get_running_activities()

I then retrieve the job ids from the MR activities, and use the Hadoop command 'mapred job -status' to asertain information about them.

I am using CDH 4, and I am not currently using Yarn.

Thanks!

[#2 · Accepted solution]

The CM API does support fetching metrics of a given activity, as described at http://cloudera.github.io/cm_api/apidocs/v6/path__clusters_-clusterName-_services_-serviceName-_acti...

[#3]

I initially found this confusing, because the Python library for the Cloudera Manager API lacks helper functions for this API endpoint. Nonetheless, it is easy to implement the API call in Python. I will look into adding a helper class to the open-source Python library.

HOST = 'myhost'
CLUSTER_NAME = 'mycluster'
SERVICE = 'mapreduce1'
ACTIVITY_ID = 'your_activity_job_id'

parameters = 'clusters/%s/services/%s/activities/%s/metrics' % (
    CLUSTER_NAME, SERVICE, ACTIVITY_ID)

url = '%s:7180/api/v1/%s' % (HOST, urllib.quote(parameters))
r = requests.get(url,auth=(USERNAME, PASSWORD))
print r.json()