Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here. Want to know more about what has changed? Check out the Community News blog.

Getting cannot open port 7180 when starting multi node cluster with clusterdock

Getting cannot open port 7180 when starting multi node cluster with clusterdock

Explorer

Hi, 

 

I'm trying to start multi node cluster using clusterdock. I have 32 GB RAM and 2 cores. It fails with following error. 

 

[root@156-56-32-113 clusterdock]# systemctl restart docker
[root@156-56-32-113 clusterdock]# clusterdock_run bin/start_cluster cdh
INFO:clusterdock.cluster:Successfully started node-2.cluster (IP address: 192.168.123.3).
INFO:clusterdock.cluster:Successfully started node-1.cluster (IP address: 192.168.123.2).
INFO:clusterdock.cluster:Started cluster in 22.59 seconds.
INFO:clusterdock.topologies.cdh.actions:Changing server_host to node-1.cluster in /etc/cloudera-scm-agent/config.ini...
INFO:clusterdock.topologies.cdh.actions:Restarting CM agents...
cloudera-scm-agent is already stopped
Stopping cloudera-scm-agent: [ OK ]
Starting cloudera-scm-agent: [ OK ]
Starting cloudera-scm-agent: [ OK ]
INFO:clusterdock.topologies.cdh.actions:Waiting for Cloudera Manager server to come online...
Traceback (most recent call last):
File "bin/start_cluster", line 70, in <module>
main()
File "bin/start_cluster", line 63, in main
actions.start(args)
File "/root/clusterdock/clusterdock/topologies/cdh/actions.py", line 108, in start
CM_SERVER_PORT, timeout_sec=180)
File "/root/clusterdock/clusterdock/utils.py", line 52, in wait_for_port_open
timeout_sec, address, port
Exception: Timed out after 180 seconds waiting for 192.168.123.2:7180 to be open.

 

I'm using RHEL 7. 

 

Thank you..

 

Chathuri

5 REPLIES 5

Re: Getting cannot open port 7180 when starting multi node cluster with clusterdock

Master Guru
I've not attempted a clusterdock run, but wanted to note that Cloudera Manager startup, especially in low-heap-memory or limited CPU ability situations, may take a while to fully load all its required modules and begin serving on the configured port (7180). You may want to try and increase the timeout period clusterdock uses.

You can verify the CM log under /var/log/cloudera-scm-server/cloudera-scm-server.log to see if its actually mid-process of coming up when the clusterdock action gives up.

Re: Getting cannot open port 7180 when starting multi node cluster with clusterdock

Expert Contributor

Following up on Harsh's comments, if you find Cloudera Manager is not fully started up yet after 3 minutes due to system being slow, then you can also manually adjust the "timeout_sec" parameter from 180 s to a larger value like 360 s in the actions.py source code to avoid getting the reported error message:

 

https://github.com/cloudera/clusterdock/blob/120970d73c314815effa273e98017555379e22da/clusterdock/to...

Re: Getting cannot open port 7180 when starting multi node cluster with clusterdock

Explorer

I was trying exactly the same thing. But it seems, it still keep the timeout as 180 s for some reason. 

 

logger.info('Waiting for Cloudera Manager server to come online...')
cm_server_startup_time = wait_for_port_open(primary_node.ip_address,
CM_SERVER_PORT, timeout_sec=1800)
logger.info("Detected Cloudera Manager server after %.2f seconds.", cm_server_startup_time)
cm_server_web_ui_host_port = get_host_port_binding(primary_node.container_id,
CM_SERVER_PORT)

 

And when I run the clusterdock_run, it still gives me the same error. 

 

INFO:clusterdock.cluster:Successfully started node-2.cluster (IP address: 192.168.123.3).
INFO:clusterdock.cluster:Successfully started node-1.cluster (IP address: 192.168.123.2).
INFO:clusterdock.cluster:Started cluster in 39.65 seconds.
INFO:clusterdock.topologies.cdh.actions:Changing server_host to node-1.cluster in /etc/cloudera-scm-agent/config.ini...
INFO:clusterdock.topologies.cdh.actions:Restarting CM agents...
cloudera-scm-agent is already stopped
Stopping cloudera-scm-agent: [ OK ]
Starting cloudera-scm-agent: [ OK ]
Starting cloudera-scm-agent: [ OK ]
INFO:clusterdock.topologies.cdh.actions:Waiting for Cloudera Manager server to come online...
Traceback (most recent call last):
File "bin/start_cluster", line 70, in <module>
main()
File "bin/start_cluster", line 63, in main
actions.start(args)
File "/root/clusterdock/clusterdock/topologies/cdh/actions.py", line 108, in start
CM_SERVER_PORT, timeout_sec=180)
File "/root/clusterdock/clusterdock/utils.py", line 52, in wait_for_port_open
timeout_sec, address, port
Exception: Timed out after 180 seconds waiting for 192.168.123.2:7180 to be open.

 

I tried with restarting docker, restarting the server and also I increase the number of cores to 8 and now it has 64 GB RAM. So I doubt this is due to any resource issue. 

 

I also connect the container while I'm running the command and I don't see any errors while starting cdh manager. I'm attaching the log file for your reference. 

 

 

 

Re: Getting cannot open port 7180 when starting multi node cluster with clusterdock

Explorer

After removing the github folder and re-cloning it, somehow I was able to go over that step. Now it fails at service validation step. I can log in to containers and I can see the services. How can I log in to cloudera manager ? I tried <ip_address_of_host_machine>:7180, but it says it is not running. I tried telnet as well, and it gives me the same error. 

 

When I log in to primary node and check the status of cloudera scm server, it says "running". 

 

 

Re: Getting cannot open port 7180 when starting multi node cluster with clusterdock

Explorer

I was able to make it work. It was a problem with port mapping from host machine. I followed https://stackoverflow.com/questions/19897743/exposing-a-port-on-a-live-docker-container and able to access cloudera manager from browser. 

 

Thank you for all your help.