CDH Activation stuck in a loop of 404

[#1]

Hi,

I rebooted the machine that host cloudera manager. But after the reboot I got a problem. 
In parcels I see that cdh: 5.9.0-1.cdh5.9.0.p0.23 is beeing distributed, but it is stuck in one machine in the activation step.

I tried to restart and hard restart the agent and the server, but I always have the same problem. The machine it is stuck on the activation is the same machine as the one that have cloudera manager, so there shouldn't be any network problem.

The agent log keep looping over the same logs :

[14/Nov/2016 14:30:08 +0000] 24070 MainThread client_configs ERROR    Failed to deploy client config <hadoop-conf,/etc/hadoop/conf.cloudera.hdfs>: No parcel provided required tags: set([u'cdh'])
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/client_configs.py", line 769, in rectify
    deploy_path = self._deploy_client_config(new_ccs[key])
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/client_configs.py", line 479, in _deploy_client_config
    env, parcels_in_use = self._adapt_cc_to_env(self.deploy_env, cc)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/client_configs.py", line 670, in _adapt_cc_to_env
    cc["optional_tags"])
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/parcel.py", line 383, in prepare_environment
    raise ParcelTagUnsatisfiedException("No parcel provided required tags: %s" % (missing_reqs,))
ParcelTagUnsatisfiedException: No parcel provided required tags: set([u'cdh'])
[14/Nov/2016 14:30:08 +0000] 24070 Thread-13 https        ERROR    Failed to retrieve/stroe URL: http://my_server:7180/cmf/parcel/download/CDH-5.9.0-1.cdh5.9.0.p0.23-trusty.parcel.torrent -> /opt/cloudera/parcel-cache/CDH-5.9.0-1.cdh5.9.0.p0.23-trusty.parcel.torrent HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 175, in fetch_to_file
    resp = self.open(req_url)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 170, in open
    return self.opener(*pargs, **kwargs)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 205, in http_error_default
    raise e
HTTPError: HTTP Error 404: Not Found
[14/Nov/2016 14:30:08 +0000] 24070 Thread-13 downloader   ERROR    Failed fetching torrent: HTTP Error 404: Not Found
Traceback (most recent call last):
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/downloader.py", line 263, in download
    cmf.https.ssl_url_opener.fetch_to_file(torrent_url, torrent_file)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 175, in fetch_to_file
    resp = self.open(req_url)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 170, in open
    return self.opener(*pargs, **kwargs)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 410, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 523, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 448, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/usr/lib/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.9.0-py2.7.egg/cmf/https.py", line 205, in http_error_default
    raise e
HTTPError: HTTP Error 404: Not Found
[14/Nov/2016 14:30:23 +0000] 24070 MainThread parcel       INFO     The following requested parcels are not available: {u'CDH': u'5.9.0-1.cdh5.9.0.p0.23'}

 Any idea ? 

thanks

[#2]

Hi,

Did you get past this issue? I am facing the same issues. In my case the newly added nodes are on RHEL 7.2 while Cloudera Manager host is running on RHEL 6.6. Not sure if this is causing the issue because, similar type of nodes were previously added successfully.

HTTPError: HTTP Error 404: Not Found
[01/Mar/2017 11:49:41 +0000] 20716 Thread-13 downloader ERROR Failed fetching torrent: HTTP Error 404: Not Found
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.3-py2.7.egg/cmf/downloader.py", line 263, in download
cmf.https.ssl_url_opener.fetch_to_file(torrent_url, torrent_file)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.3-py2.7.egg/cmf/https.py", line 175, in fetch_to_file
resp = self.open(req_url)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.3-py2.7.egg/cmf/https.py", line 170, in open
return self.opener(*pargs, **kwargs)
File "/usr/lib64/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python2.7/urllib2.py", line 475, in error
return self._call_chain(*args)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.8.3-py2.7.egg/cmf/https.py", line 205, in http_error_default
raise e
HTTPError: HTTP Error 404: Not Found

Thanks,

Soundy

[#3]

I am also having the same issue.

I'm trying to add a new node (CentOS 7.2) to my existing cluster which has CentOS 6.7. While going through the 'Add New Host Cluster' Wizard, activation of CDH parcel is stuck with 404: Not Found Error (below):


[06/Apr/2017 18:08:22 +0000] 32130 Thread-13 downloader ERROR Failed fetching torrent: HTTP Error 404: Not Found
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/downloader.py", line 263, in download
cmf.https.ssl_url_opener.fetch_to_file(torrent_url, torrent_file)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/https.py", line 175, in fetch_to_file
resp = self.open(req_url)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/https.py", line 170, in open
return self.opener(*pargs, **kwargs)
File "/usr/lib64/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib64/python2.7/urllib2.py", line 437, in open
response = meth(req, response)
File "/usr/lib64/python2.7/urllib2.py", line 550, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib64/python2.7/urllib2.py", line 475, in error
return self._call_chain(*args)
File "/usr/lib64/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.10.0-py2.7.egg/cmf/https.py", line 205, in http_error_default
raise e
HTTPError: HTTP Error 404: Not Found

Any solutions or workaround for this?

Thanks!

[#4]

To get around this, I removed the failing parcels from the servers using the Cloudera Manager GUI, then re-downloaded, distributed and activated them.

Hope this helps.