Reply
Explorer
Posts: 19
Registered: ‎06-23-2014
Accepted Solution

Programmatically tracking MR Job status using the Cloudera Manager API (and Python libs)?

Hi Cloudera community:

 

tl;dr: Is there a more straighforward way to query the Cloudera Manager API to get information (mapper/reducer completion, bytes processed, etc) about jobs, perhaps by simply providing a jobId or a jobname?

 

I am using a Python script to check on the status of various MapReduce jobs, using the Cloudera Manager API, roughly something like this:

 

from cm_api.api_client import ApiResource
 
api = ApiResource('zzzz',
                        version=1,
                        username='zzz',
                        password='zzz')
 
for s in api.get_cluster('my cluster').get_all_services():
        if s.name == 'MR':
        # my activities are in s.get_running_activities()

I then retrieve the job ids from the MR activities, and use the Hadoop command 'mapred job -status' to asertain information about them.

 

I am using CDH 4, and I am not currently using Yarn.

 

Thanks!

 

 

Posts: 1,760
Kudos: 380
Solutions: 282
Registered: ‎07-31-2013

Re: Programmatically tracking MR Job status using the Cloudera Manager API (and Python libs)?

The CM API does support fetching metrics of a given activity, as described at http://cloudera.github.io/cm_api/apidocs/v6/path__clusters_-clusterName-_services_-serviceName-_acti...
Explorer
Posts: 19
Registered: ‎06-23-2014

Re: Programmatically tracking MR Job status using the Cloudera Manager API (and Python libs)?

I initially found this confusing, because the Python library for the Cloudera Manager API lacks helper functions for this API endpoint. Nonetheless, it is easy to implement the API call in Python. I will look into adding a helper class to the open-source Python library.

 

HOST = 'myhost'
CLUSTER_NAME = 'mycluster'
SERVICE = 'mapreduce1'
ACTIVITY_ID = 'your_activity_job_id'

parameters = 'clusters/%s/services/%s/activities/%s/metrics' % (
    CLUSTER_NAME, SERVICE, ACTIVITY_ID)

url = '%s:7180/api/v1/%s' % (HOST, urllib.quote(parameters))
r = requests.get(url,auth=(USERNAME, PASSWORD))
print r.json()
Announcements