Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Programmatically tracking MR Job status using the Cloudera Manager API (and Python libs)?

Solved Go to solution

Programmatically tracking MR Job status using the Cloudera Manager API (and Python libs)?

Hi Cloudera community:

 

tl;dr: Is there a more straighforward way to query the Cloudera Manager API to get information (mapper/reducer completion, bytes processed, etc) about jobs, perhaps by simply providing a jobId or a jobname?

 

I am using a Python script to check on the status of various MapReduce jobs, using the Cloudera Manager API, roughly something like this:

 

from cm_api.api_client import ApiResource
 
api = ApiResource('zzzz',
                        version=1,
                        username='zzz',
                        password='zzz')
 
for s in api.get_cluster('my cluster').get_all_services():
        if s.name == 'MR':
        # my activities are in s.get_running_activities()

I then retrieve the job ids from the MR activities, and use the Hadoop command 'mapred job -status' to asertain information about them.

 

I am using CDH 4, and I am not currently using Yarn.

 

Thanks!

 

 

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Programmatically tracking MR Job status using the Cloudera Manager API (and Python libs)?

Master Guru
The CM API does support fetching metrics of a given activity, as described at http://cloudera.github.io/cm_api/apidocs/v6/path__clusters_-clusterName-_services_-serviceName-_acti...
2 REPLIES 2

Re: Programmatically tracking MR Job status using the Cloudera Manager API (and Python libs)?

Master Guru
The CM API does support fetching metrics of a given activity, as described at http://cloudera.github.io/cm_api/apidocs/v6/path__clusters_-clusterName-_services_-serviceName-_acti...

Re: Programmatically tracking MR Job status using the Cloudera Manager API (and Python libs)?

I initially found this confusing, because the Python library for the Cloudera Manager API lacks helper functions for this API endpoint. Nonetheless, it is easy to implement the API call in Python. I will look into adding a helper class to the open-source Python library.

 

HOST = 'myhost'
CLUSTER_NAME = 'mycluster'
SERVICE = 'mapreduce1'
ACTIVITY_ID = 'your_activity_job_id'

parameters = 'clusters/%s/services/%s/activities/%s/metrics' % (
    CLUSTER_NAME, SERVICE, ACTIVITY_ID)

url = '%s:7180/api/v1/%s' % (HOST, urllib.quote(parameters))
r = requests.get(url,auth=(USERNAME, PASSWORD))
print r.json()