Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How can I use resource manager api calls in python script

Highlighted

How can I use resource manager api calls in python script

I am able to get the required result by running the below command on the shell curl --negotiate -u : "http://rm_host:8088/ws/v1/cluster"

I need to generate a report of jobs running/failing on a every day basis. Also need to add other information like CPU utilization, memory usage, categorizing different jobs etc. I am thinking of using resource manager api calls for this purpose.

The response received by the api call is in json format.

I am thinking of parsing the json object to get the desired result

example:

info=json.loads(curl_command_response)

print(info['clusterInfo']['state'])

When I am executing the below python code, I am getting authorization required error

import urllib2, base64

request = urllib2.Request("http://rm_host:8088/ws/v1/cluster")

base64string = base64.b64encode('%s:%s' % ('username','password'))

request.add_header("Authorization", "Basic %s" % base64string)

result = urllib2.urlopen(request)

===================================================

Error:

Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/lib64/python2.6/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib64/python2.6/urllib2.py", line 397, in open response = meth(req, response) File "/usr/lib64/python2.6/urllib2.py", line 510, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib64/python2.6/urllib2.py", line 435, in error return self._call_chain(*args) File "/usr/lib64/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib64/python2.6/urllib2.py", line 518, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 401: Authentication required

=====================================================

3 REPLIES 3

Re: How can I use resource manager api calls in python script

Cloudera Employee

You do not have access to RM from terminal where you are running the script, which is why you see Authentication error.

Highlighted

Re: How can I use resource manager api calls in python script

Super Guru

@Vishal Prakash Shah

You have kerberos enabled in your cluster hence it works fine with --negotiate parameter with curl.

In your python script, you are not passing --negotiate parameter. Can you please check and see how you can pass below parameter to the curl to make it working?

--negotiate -u :

Also, please see this - http://stackoverflow.com/questions/29875920/python-curl-execute-command-with-parameter

Hope this helps! :)

Highlighted

Re: How can I use resource manager api calls in python script

@Vishal Prakash Shah

Additionally can you please try changing the following line:

request.add_header("Authorization", "Basic %s" % base64string)

with the following

request.add_unredirected_header("Authorization", "Basic %s" % base64string)

.

Don't have an account?
Coming from Hortonworks? Activate your account here