question Re: BDR using CM API in Archives of Support Questions (Read Only)

BDR usinf CM API

DanielWhite — Thu, 15 Dec 2016 14:01:50 GMT

I'm new to cloudera and am trying to automatically start and stop replication jobs. We're replicating from prod to our DR site.

Can someone share examples of python or java code that shows how to call the cm api to start and stop replication jobs?

Any help is appreciated.

Re: BDR using CM API

DanielWhite — Tue, 10 Jan 2017 13:12:09 GMT

Re: BDR using CM API

cjervis — Tue, 10 Jan 2017 13:22:51 GMT

Thank you for marking your issue as solved @DanielWhite. Can you advise what the solution was?

Re: BDR using CM API

DanielWhite — Tue, 10 Jan 2017 13:40:52 GMT

I've solved half of the question. I can start bdr jobs but haven't figured out how to stop them yet.

I'm using the cloudera api and curl to start the jobs via a bash script. I have a python script I'm developing for use in the future.

Like -

curl -X POST -u user:password \
'http://server:7180/api/v9/clusters/Cluster%201/services/hive/replications/jobnumber/run'

I tried using the hadoop and mapred commands to kill a job but this made the job lose it's mind and didn't kill it gracefully. This caused an error in the cloudera manager and I had to set null in the jobs log to get it to die.

Like -

hadoop job -kill "$mapred_jobid"

Re: BDR using CM API

DanielWhite — Wed, 01 Feb 2017 13:51:59 GMT

I solved the stop bdr jobs. This is done with the command api. Once a bdr job is started it gets a unique command id. Its runs using this id rather then the job number that we see in CM. The following is how I'm stopping the jobs

Firstly, get the command id by using the status api -

command_id=`curl -u admin:password "http://servername:7180/api/v9/clusters/C
luster%201/services/"$i"/replications" | grep -A 1 '"active" : true' | awk '{print $3}' | aw
k -F '/' '{print $4}' | sed '/^\s*$/d'`

Second, stop the job using the command abort api -

curl -X POST -u admin:password \
"http://servername:7180/api/v9/commands/"$command_id"/abort"

Thanks,

Daniel

Re: BDR using CM API

sorabhj412 — Wed, 20 Jun 2018 10:09:13 GMT

Hi ,

i am trying in the same way but its not showing any replication schedules .

hostname:/$ curl -su <user> https://hostname:7183/api/v1/clusters/cluster%20/services/hdfs/replications/
Enter host password for user 'user':
hostname:/$

Please advise .

Re: BDR using CM API

bgooley — Wed, 20 Jun 2018 16:42:39 GMT

@sorabhj412,

As described here, the "replications" api call is only available starting with API v11:

https://cloudera.github.io/cm_api/apidocs/v19/path__clusters_-clusterName-_services_-serviceName-_replications.html

You should be able to get this to work if you change your URL to v11 or as high as your CM supports:

https://hostname:7183/api/v11/clusters/cluster/services/hdfs/replications/

To see what the max version of the API your Cloudera Manager supports, run:

curl -su <user> https://hostname:7183/api/version

NOTE: you also seem to have a space character after your "cluster" cluster name: "clusters/cluster%20/" I believe that may also cause you problems. To see what cluster name you have, check with:

curl -su <user> https://hostname:7183/api/v11/clusters

Re: BDR using CM API

DanielWhite — Wed, 20 Jun 2018 17:26:47 GMT

Try the following -

$i = hive or hdfs

curl -u username:password "http://hostname:7180/api/v13/clusters/Cluster%201/services/"$i"/replications"

Re: BDR using CM API

sorabhj412 — Thu, 21 Jun 2018 05:26:58 GMT

Hi Daniel,

Thanks , I tried your suggestion and the commands looks working . but it’s not showing any output even after 60mins .

Below command just keep running .

I am looking for a way to get BDR Job history from command line . please help.

cloudera-scm@gbrpsr000004835 PROD $ curl -su <user> -X GET 'https://<localhost>:7183/api/v15/clusters/Cluster%DR'/services/hdfs/replications

Re: BDR using CM API

scottwong — Thu, 05 Mar 2020 14:55:42 GMT

I've same the issue, but i got the BDR job history back after waited for 4 hours. The BDR job history file is very large. Is anyway to filter out only extract the job from yesterday?

Thanks,

Scott

Re: BDR using CM API

ask_bill_brooks — Thu, 05 Mar 2020 20:20:29 GMT

@scottwong,

The thread you've posted your question to was marked 'Solved' quite a while ago and hasn't had any activity recently. You would have a better chance of receiving a satisfactory resolution by posting a new question. This will also provide the opportunity to provide details specific to what you are attempting to do that could aid other community members in providing a more tailored answer to your issue.