Reply
Explorer
Posts: 10
Registered: ‎03-11-2015

cloudera director bootstrap failed on "hostname: Unknown host"

I am using "AWS Quick Start" in AWS to launch my cluster. 

The quick start works fine, I got two instances: ClusterLauncher Instance and NAT Instance;

 

 

The next step is to change the configuration in the launcher instance. I just added privateKey pem file and change the subnet to public subnet, and run the bootstrap command. 

 

But on console the bootstrap stopped after printing 7 lines on the console. The console output is: 

 

Process logs can be found at /home/ec2-user/cloudera/cloudera-director-client-1.1.0/logs/application.log

Cloudera Director 1.1.0 initializing ...

Installing Cloudera Manager ...

* Starting ..... done

* Requesting an instance for Cloudera Manager ......................... done

* Running custom bootstrap script on 172.40.2.159 ...... done

* Inspecting capabilities of 172.40.2.159 ...

 

 

And I can see an error in the application.log file line 199:

    186 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: # Generated by iptables-save v1.4.7 on Fri Mar 20 01:41:33 2015

    187 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: *filter

    188 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: :INPUT ACCEPT [0:0]

    189 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: :FORWARD ACCEPT [0:0]

    190 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: :OUTPUT ACCEPT [282:32137]

    191 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT

    192 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: -A INPUT -p icmp -j ACCEPT

    193 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: -A INPUT -i lo -j ACCEPT

    194 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: -A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT

    195 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: -A INPUT -j REJECT --reject-with icmp-host-prohibited

    196 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: -A FORWARD -j REJECT --reject-with icmp-host-prohibited

    197 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: COMMIT

    198 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: # Completed on Fri Mar 20 01:41:33 2015

    199 [2015-03-19 21:41:34] INFO  [io-thread-1] - ssh:172.40.2.159: hostname: Unknown host

 

Can some somebody tell me what is the issue?

If the installation is going smoothly, this line in the log should be something like this:  

  Server: 172.40.0.2

  Address: 172.40.0.2#53

 

thanks a lot!

Bin

Cloudera Employee
Posts: 119
Registered: ‎10-13-2014

Re: cloudera director bootstrap failed on "hostname: Unknown host"

What do you mean by chaging the subnet to public subnet? That may be the root of this problem. The preconfigured one should just work. 

 

Director needs proper forward and reverse DNS resolution for hostnames on cluster instances. 

Explorer
Posts: 10
Registered: ‎03-11-2015

Re: cloudera director bootstrap failed on "hostname: Unknown host"

I think the issue is the proxy. 

We have existing VPC, and use proxy to connect to internet.

The installation scripts need to  download RPMs from internet, so that cause the isseu. 

I added "--lp.proxy.http.host=xxx --lp.proxy.http.port=xxx" to the command : cloudera-director bootstrap and it is working now.

 

But after the CM instance is created and running, we have one more issue, the bootstrap failed on step "Creating cluster ..."

 

[2015-03-21 11:22:26] INFO [main] - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating parcel URL and version compatibility
[2015-03-21 11:22:28] INFO [main] - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating parcels and version compatibility
[2015-03-21 11:22:28] INFO [main] - c.c.l.b.v.GenericDeploymentTemplateValidator: Validating parcel URL and version compatibility 

 

I am guessing the problem is that we set the proxy to environment variables like "export http_proxy=xxx" and it works for curl, wget command, but not java native call. The failure could be the java command could not use the proxy.

 

Is there anybody having the same issue? what is the solution?

 

thanks

Bin 

Cloudera Employee
Posts: 119
Registered: ‎10-13-2014

Re: cloudera director bootstrap failed on "hostname: Unknown host"

I believe you should look into adding a bootstrap script to configure the proxy for the package manager on the new instances. Parcel validation should not generate a failure - it should be able to use the proxy you specified via the command line options. 

Expert Contributor
Posts: 61
Registered: ‎03-06-2015

Re: cloudera director bootstrap failed on "hostname: Unknown host"

Hi

 

What does your hosts configuration look like?  Also, what is in your nsswitch?  It could be an easy fix as it sounds like the same issues when I first started playing with Cloudera.

 

 

Explorer
Posts: 10
Registered: ‎03-11-2015

Re: cloudera director bootstrap failed on "hostname: Unknown host"

[ Edited ]

finally I found out the issue is proxy issue. 

The java does not use the proxy setting as I provided in the command line. 

Cloudera Employee
Posts: 119
Registered: ‎10-13-2014

Re: cloudera director bootstrap failed on "hostname: Unknown host"

Did that work as expected and resulted in a healthy cluster? Similar options are available for the Cloudera Director server. 

Explorer
Posts: 9
Registered: ‎04-03-2015

Re: cloudera director bootstrap failed on "hostname: Unknown host"

This sounds like the problem I am dealing with, can you let me know what you changed?

Highlighted
Explorer
Posts: 10
Registered: ‎03-11-2015

Re: cloudera director bootstrap failed on "hostname: Unknown host"

I found out the proxy setting is not picked up by the java somehow, so I modified /usr/lib64/cloudera-director/client/bin/cloudera-director by adding proxy settings to the call to java:

-Dhttp.proxyHost=[your_proxy_host] \
-Dhttp.proxyPort=[your_proxy_port] \
-Dhttps.proxyHost=[your_proxy_host] \
-Dhttps.proxyPort=[your_proxy_port] \
-Dhttp.nonProxyHosts="localhost|127.*|[::1]|172.30.2.*" \

Expert Contributor
Posts: 61
Registered: ‎03-06-2015

Re: cloudera director bootstrap failed on "hostname: Unknown host"

Did that resolve the issue skylark?