Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Setting up hosts through CM API results in Postgres Deadlock

Highlighted

Setting up hosts through CM API results in Postgres Deadlock

Explorer

Hello,

 

We are provisioning nodes through CM API and are constantly hitting deadlocks while running the "host install" call. Once the deadlock is hit, Host Install stage hangs indefinitely in CM and needs to be manually aborted.

 

CDH 5.7.2

 

cm-server-log:

http://pastebin.com/3MSLPsMv

 

psq-log:

http://pastebin.com/QtDBFrga

 

 

 

4 REPLIES 4

Re: Setting up hosts through CM API results in Postgres Deadlock

Super Guru

Hello,

 

So we can understand this better, could you please supply the API calls you are making or, even better, the code you are using?

 

Thanks,

 

Ben

Re: Setting up hosts through CM API results in Postgres Deadlock

Explorer

Hey Ben,

 

Thanks for the reply. Roughly here is how the nodes are provisioned: We have a script that decides when our cluster needs more vCPU's. It then spins up as many nodes as it feels we need simultaneously and each node fires a script that setups it's self. The script uses CMApiHelper wrapper class  and executes the following:

 

cm_api.host_install(args.host_user, args.host, args.key)
cm_api.add_host(args.host)
# All prior steps need to be fully completed before apply_host_template is called
sleep(30)
cm_api.apply_host_template(args.role_template, args.host)
cm_api.host_recommission(args.host)
cm_api.start_roles(args.host)
# Give the service a chance to properly start up before checking whether it's up and healthy
sleep(180)
cm_api.check_host(args.host)

 

Let me know if you need any more info, I am more than happy to provide it.

 

Re: Setting up hosts through CM API results in Postgres Deadlock

Super Guru

This could be a bit more complex.  I'm leaning toward this being a db locking issue rather than something that CM is doing wrong, but it is hard to say without digging in deep here.

What version of postgres are you using?  Perhaps it is older?

Re: Setting up hosts through CM API results in Postgres Deadlock

Explorer

We are using the built in db via the yum package:

 

cloudera-manager-server-db-2.x86_64 5.7.2-1.cm572.p0.17.el6

postgres (PostgreSQL) 8.4.20

 

We would not mind a remote session to gather more info if it would help to resolve this issue.

Don't have an account?
Coming from Hortonworks? Activate your account here