New Node Addition Failure process hangs

[#1]

Hello all,

Currently  am stuck at adding a new node to my cluster.

It is stuck at doing this  and is running in a loop.

Please guide me as to what can I do to get this moving on.

Any help will be highly appriciated.

[14/Apr/2014 18:59:27 +0000] 6073 Thread-13 downloader   INFO     Starting download of: http://host2:7180/cmf/parcel/download/IMPALA-1.2.4-1.p0.110-el6.parcel
[14/Apr/2014 18:59:27 +0000] 6073 Thread-13 downloader   ERROR    HTTP error during download
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/src/cmf/downloader.py", line 237, in _download
    dlname, op.headers = self.opener.retrieve(url, tempname, cb)
  File "/usr/lib64/python2.6/urllib.py", line 239, in retrieve
    fp = self.open(url, data)
  File "/usr/lib64/python2.6/urllib.py", line 207, in open
    return getattr(self, name)(url)
  File "/usr/lib64/python2.6/urllib.py", line 362, in open_http
    return self.http_error(url, fp, errcode, errmsg, headers)
  File "/usr/lib64/python2.6/urllib.py", line 379, in http_error
    return self.http_error_default(url, fp, errcode, errmsg, headers)
  File "/usr/lib64/cmf/agent/src/cmf/downloader.py", line 58, in http_error_default
    raise urllib2.HTTPError(url, code, msg, headers, fp)
HTTPError: HTTP Error 502: Bad Gateway

[#2 · Accepted solution]

Parcels are normally downloaded first to CM server, then distributed to each agent. Agents always download parcels from the CM server. It looks like your configuration is preventing this from working.

[#3]

Does this host have access to the internet?  Specifically can it reach the URL http://archive.cloudera.com/cdh5/ ?

It will need to be able to reach our parcels repo in order to add new hosts.  If your machine is not internet connected, you will need to follow these instructions (those are for CM 5..0.0) to create your own local repo.  Also, I notice that you are trying to install Impala version 1.2.4, please note that you must be on CM 4.8 or better to use that version as per this announcement.

HTH

[#4]

Thanks for your prompt response.

My new node have same confuguration as of all ther nodes in cluster.

Every thing is fine from our end

Still this is happening.

We have CM 4.8

Please suggest what needs to be done

[#5]

Its not the case that I am having trouble with only Impala installation.

I am having issue in adding the new node completely.

Any help will be highly appreciated.

Complete logs are

[16/Apr/2014 16:37:23 +0000] 6073 Thread-13 downloader   INFO     Starting download of: http://hadoop02.abc.com:7180/cmf/parcel/download/CDH-4.6.0-1.cdh4.6.0.p0.26-el6.parcel
[16/Apr/2014 16:37:23 +0000] 6073 Thread-13 downloader   ERROR    HTTP error during download
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/src/cmf/downloader.py", line 237, in _download
    dlname, op.headers = self.opener.retrieve(url, tempname, cb)
  File "/usr/lib64/python2.6/urllib.py", line 239, in retrieve
    fp = self.open(url, data)
  File "/usr/lib64/python2.6/urllib.py", line 207, in open
    return getattr(self, name)(url)
  File "/usr/lib64/python2.6/urllib.py", line 362, in open_http
    return self.http_error(url, fp, errcode, errmsg, headers)
  File "/usr/lib64/python2.6/urllib.py", line 379, in http_error
    return self.http_error_default(url, fp, errcode, errmsg, headers)
  File "/usr/lib64/cmf/agent/src/cmf/downloader.py", line 58, in http_error_default
    raise urllib2.HTTPError(url, code, msg, headers, fp)
HTTPError: HTTP Error 502: Bad Gateway
[16/Apr/2014 16:37:23 +0000] 6073 Thread-13 downloader   INFO     Starting download of: http://hadoop02.abc.com:7180/cmf/parcel/download/IMPALA-1.2.4-1.p0.110-el6.parcel
[16/Apr/2014 16:37:23 +0000] 6073 Thread-13 downloader   ERROR    HTTP error during download
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/src/cmf/downloader.py", line 237, in _download
    dlname, op.headers = self.opener.retrieve(url, tempname, cb)
  File "/usr/lib64/python2.6/urllib.py", line 239, in retrieve
    fp = self.open(url, data)
  File "/usr/lib64/python2.6/urllib.py", line 207, in open
    return getattr(self, name)(url)
  File "/usr/lib64/python2.6/urllib.py", line 362, in open_http
    return self.http_error(url, fp, errcode, errmsg, headers)
  File "/usr/lib64/python2.6/urllib.py", line 379, in http_error
    return self.http_error_default(url, fp, errcode, errmsg, headers)
  File "/usr/lib64/cmf/agent/src/cmf/downloader.py", line 58, in http_error_default
    raise urllib2.HTTPError(url, code, msg, headers, fp)
HTTPError: HTTP Error 502: Bad Gateway

[#6]

Looks like a basic networking error. SSH into the machine and make sure you can access the URL it is trying to reach using curl and/or wget.

Make sure things like iptables are not getting in the way.

[#7]

Why is my new host trying to download from the box where CM is installed using http protocol ??
It could have just checked the version and tried to download from cloudera repo.

This is not resolved ..

[#8 · Accepted solution]

Parcels are normally downloaded first to CM server, then distributed to each agent. Agents always download parcels from the CM server. It looks like your configuration is preventing this from working.

[#9 · Accepted solution]

Finally resolved the Issue.

Due to space crunch on root mount space we have created a soft  link for /opt/cloudera/ to point to /data/opt/cloudera/

As a result of this the the local repo path was changed as well.

Hence while installing package it was trying to download the packages again and again.

But unable to find the same on the disk as correctly distributed by CM.

As I changed the local repo directory in CM Administration/parcel-repo path

Every thing started working as required.

Thanks every one for your support and suggestions.

Priyabrata Patnaik