Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

After reboot, parcels are gone. CM shows all downloadable parcels as "unavailable"

After reboot, parcels are gone. CM shows all downloadable parcels as "unavailable"

Explorer

After rebooting a small dev cluster (for kernel update) Cloudera manager showed most services in "critical" state. HBASE, HDFS, MapReduce and Hive all have stale configurations. The change CM wants to deploy makes the value of the parcels variable empty:

 

Client Configs Metadata                         hbase(1) Show
... ...
@@ -1,6 +1,6 @@
alt_link=/etc/hbase/conf
alt_name=hbase-conf
-parcels=CDH-5.3.0-1.cdh5.3.0.p0.30
+parcels=
priority=90
root_dir_name=/etc/hbase
src_name=hbase-conf

 

 

Attempting to deploy this config fails.

 

Parcels are missing from /opt/cloudera/parcels and parcel-cache on all nodes:

 

ll /opt/cloudera/parcels
total 0

ll /opt/cloudera/parcel-cache/
total 0

 

In CM in the "parcels" tab (under "hosts") it lists all downloadable parcels as "unavailable":

 

 

CDH
5.3.10-1.cdh5.3.10.p0.19
Unavailable

 

and it shows in the dev cluster below that it is "distributing" the parcels, but this never completes:

 

CDH
5.3.0-1.cdh5.3.0.p0.30
Distributing

 

In the cloudera agent logs it says the parcel is not available:

 

[25/Apr/2016 17:37:47 +0000] 3259 MainThread agent INFO Activating Process 2283-deploy-client-config
[25/Apr/2016 17:37:47 +0000] 3259 MainThread agent INFO Created /var/run/cloudera-scm-agent/process/2283-deploy-client-config
[25/Apr/2016 17:37:47 +0000] 3259 MainThread agent INFO Chowning /var/run/cloudera-scm-agent/process/2283-deploy-client-config to root (0) root (0)
[25/Apr/2016 17:37:47 +0000] 3259 MainThread agent INFO Chmod'ing /var/run/cloudera-scm-agent/process/2283-deploy-client-config to 0751
[25/Apr/2016 17:37:47 +0000] 3259 MainThread parcel INFO prepare_environment begin: {u'CDH': u'5.3.0-1.cdh5.3.0.p0.30'}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']
[25/Apr/2016 17:37:47 +0000] 3259 MainThread parcel INFO The following requested parcels are not available: {u'CDH': u'5.3.0-1.cdh5.3.0.p0.30'}
[25/Apr/2016 17:37:47 +0000] 3259 MainThread agent ERROR Failed to activate {u'refresh_files': [], u'config_generation': 0, u'auto_restart': False, u'running': True, u'required_tags': [u'cdh'], u'user': u'root', u'special_file_info': [], u'group': u'root', u'id': 2283, u'status_links': {}, u'name': u'deploy-client-config', u'run_generation': 1, u'environment': {u'ROOT_DIR_NAME': u'/etc/hadoop', u'CDH_VERSION': u'5', u'ALT_LINK': u'/etc/hadoop/conf', u'PRIORITY': u'90', u'ALT_NAME': u'hadoop-conf', u'SERVICE_TYPE': u'HDFS', u'DIRECTORY_NAME': u'hadoop-conf'}, u'optional_tags': [u'cdh-plugin', u'hdfs-plugin'], u'program': u'client/deploy-cc.sh', u'arguments': [u'hdfs-clientconfig.zip', u'conf.cloudera.hdfs'], u'parcels': {u'CDH': u'5.3.0-1.cdh5.3.0.p0.30'}, u'resources': [], u'one_off': True}
Traceback (most recent call last):
File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 1121, in handle_heartbeat_response
new_process.activate()
File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2197, in activate
self.update_process_environment_and_parcels()
File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 2362, in update_process_environment_and_parcels
self.raw['optional_tags'])
File "/usr/lib64/cmf/agent/src/cmf/parcel.py", line 300, in prepare_environment
raise ParcelTagUnsatisfiedException("No parcel provided required tags: %s" % (missing_reqs,))
ParcelTagUnsatisfiedException: No parcel provided required tags: set([u'cdh'])

 

 

In CM clicking hosts --> parcels --> "check for new parcels" seems to do nothing. This gets logged in cloudera-scm-server.log when check for new parcels is clicked:

 

2016-04-28 15:27:52,931 INFO 2123867933@scm-web-823:com.cloudera.cmf.service.ServiceHandlerRegistry: Executing command ProcessStalenessCheckCommand BasicCmdArgs{args=[]}.
2016-04-28 15:27:55,659 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: (4 skipped) Invalid parcel in manifest : KEYTRUSTEE-5.7.0-5.KEYTRUSTEE5.7.0.p0.11-el7.parcel
2016-04-28 15:27:56,323 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: (4 skipped) Invalid parcel in manifest : SQOOP_TERADATA_CONNECTOR-1.5c5-el7.parcel
2016-04-28 15:27:56,324 WARN ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: (4 skipped) Invalid parcel in manifest : SQOOP_NETEZZA_CONNECTOR-1.3c5-el7.parcel
2016-04-28 15:27:56,671 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Parcel DbParcel{id=231, filename=CDH-5.3.10-1.cdh5.3.10.p0.19-trusty.parcel} no longer exists on source repo. {} therefore deleted.
2016-04-28 15:27:56,674 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Parcel DbParcel{id=228, filename=CDH-5.3.10-1.cdh5.3.10.p0.19-el6.parcel} no longer exists on source repo. {} therefore deleted.
2016-04-28 15:27:56,677 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Parcel DbParcel{id=229, filename=CDH-5.3.10-1.cdh5.3.10.p0.19-precise.parcel} no longer exists on source repo. {} therefore deleted.
2016-04-28 15:27:56,680 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Parcel DbParcel{id=227, filename=CDH-5.3.10-1.cdh5.3.10.p0.19-el5.parcel} no longer exists on source repo. {} therefore deleted.
2016-04-28 15:27:56,684 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Parcel DbParcel{id=230, filename=CDH-5.3.10-1.cdh5.3.10.p0.19-sles11.parcel} no longer exists on source repo. {} therefore deleted.
2016-04-28 15:27:56,686 INFO ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Parcel DbParcel{id=232, filename=CDH-5.3.10-1.cdh5.3.10.p0.19-wheezy.parcel} no longer exists on source repo. {} therefore deleted.
2016-04-28 15:27:59,740 INFO ProcessStalenessDetector-0:com.cloudera.cmf.service.config.components.ProcessStalenessDetector: Staleness check done. Duration: PT6.652S
2016-04-28 15:28:03,677 INFO 2123867933@scm-web-823:com.cloudera.server.web.cmf.ParcelController: Synchronizing repos based on user request admin
2016-04-28 15:28:03,679 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files CDH-4.7.0-1.cdh4.7.0.p0.40-el6.parcel,CDH-5.3.0-1.cdh5.3.0.p0.30-el6.parcel under /opt/cloudera/parcel-repo
2016-04-28 15:28:04,889 INFO 461674099@scm-web-827:com.cloudera.server.web.cmf.ParcelController: Synchronizing repos based on user request admin
2016-04-28 15:28:06,124 INFO 461674099@scm-web-827:com.cloudera.server.web.cmf.ParcelController: Synchronizing repos based on user request admin
2016-04-28 15:28:06,934 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files CDH-4.7.0-1.cdh4.7.0.p0.40-el6.parcel,CDH-5.3.0-1.cdh5.3.0.p0.30-el6.parcel under /opt/cloudera/parcel-repo
2016-04-28 15:28:08,402 INFO 840582814@scm-web-829:com.cloudera.server.web.cmf.ParcelController: Synchronizing repos based on user request admin
2016-04-28 15:28:10,760 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files CDH-4.7.0-1.cdh4.7.0.p0.40-el6.parcel,CDH-5.3.0-1.cdh5.3.0.p0.30-el6.parcel under /opt/cloudera/parcel-repo
2016-04-28 15:28:14,244 INFO ParcelUpdateService:com.cloudera.parcel.components.LocalParcelManagerImpl: Found files CDH-4.7.0-1.cdh4.7.0.p0.40-el6.parcel,CDH-5.3.0-1.cdh5.3.0.p0.30-el6.parcel under /opt/cloudera/parcel-repo

 

I am not sure where to go from here. I cannot even figure out where the "parcels" variable that cloudera manager wants to deploy to all hosts (changing the value from "CDH-5.3.0-1.cdh5.3.0.p0.30" to null) exists on the system. Any help would be greatly appreciated.

 

 

4 REPLIES 4

Re: After reboot, parcels are gone. CM shows all downloadable parcels as "unavailable"

Explorer

When checking why all downloadable parcels are unavailable I see the following error message:

 

One or more hosts did not report their OS Distribution. This is likely due to running an outdated version of the Cloudera Manager Agent. Please run the Host Inspector to check the Agent versions..

 

I am running the same agent on all nodes. 

 

The host inspector does not indicate any problems. The agent version on all machines is:

 

cloudera-manager-agent-5.2.0-1.cm520.p0.60.el6.x86_64

 

 

Highlighted

Re: After reboot, parcels are gone. CM shows all downloadable parcels as "unavailable"

Explorer

Here is my host inspector output. The java version difference is because I am using the cloudera supplied JVM on the cloudera manager node and an RPM (slightly different version) on the other nodes. It has run this way for over a year with no issues. 

 

{"agentUserErrors":[],"allHostsDnsMessages":[],"componentSetByCluster":{"DEV_idtracker":[{"componentInfo":[{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"parquet"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"impala"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"yarn"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hdfs"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hue_plugins"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"sqoop2"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"kms"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hbase"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"sqoop"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"oozie"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"zookeeper"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hue"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"spark"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"mr1"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"pig"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"crunch"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"llama"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"httpfs"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hadoop"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hive"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hcatalog"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"sentry"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"mr2"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hbase_indexer"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"solr"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"flume"},{"cdhVersion":"NOT_APPLICABLE","componentRelease":"1.cm520.p0.60","componentVersion":"5.2.0","name":"cm_daemons"},{"cdhVersion":"NOT_APPLICABLE","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"java6"},{"cdhVersion":"NOT_APPLICABLE","componentRelease":"NO_SCRIPT","componentVersion":"java version \"1.7.0_71\" Java(TM) SE Runtime Environment (build 1.7.0_71-b14) Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode) ","name":"java7"},{"cdhVersion":"NOT_APPLICABLE","componentRelease":"1.cm520.p0.60.el6","componentVersion":"5.2.0","name":"cm_agent"}],"hosts":["lnx605.mm-corp.net","lnx606.mm-corp.net","lnx604.mm-corp.net"]},{"componentInfo":[{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"parquet"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"impala"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"yarn"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"spark"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hdfs"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hue_plugins"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"sqoop2"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"kms"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hbase"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"sqoop"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"oozie"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"zookeeper"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hue"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hbase_indexer"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"mr1"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"pig"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"crunch"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"llama"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"httpfs"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hadoop"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hive"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"hcatalog"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"sentry"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"mr2"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"solr"},{"cdhVersion":"NO_SCRIPT","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"flume"},{"cdhVersion":"NOT_APPLICABLE","componentRelease":"1.cm520.p0.60","componentVersion":"5.2.0","name":"cm_daemons"},{"cdhVersion":"NOT_APPLICABLE","componentRelease":"NO_SCRIPT","componentVersion":"NO_SCRIPT","name":"java6"},{"cdhVersion":"NOT_APPLICABLE","componentRelease":"NO_SCRIPT","componentVersion":"java version \"1.7.0_67\" Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode) ","name":"java7"},{"cdhVersion":"NOT_APPLICABLE","componentRelease":"1.cm520.p0.60.el6","componentVersion":"5.2.0","name":"cm_agent"}],"hosts":["lnx603.mm-corp.net"]}]},"componentSetNoCluster":[],"etcHostsMessages":[],"etcKrbConfErrors":[],"extantInitdErrors":[{"args":["hue","lnx603.mm-corp.net"],"messageId":"message.inspector.extantInitd.enabledComponent"}],"failedHostsWithError":{},"failedLocalhostSanityMessages":[],"hostDnsChecksSuccesses":4,"hostDnsMessages":[],"huePyVersionErrors":[],"inspections":"DEFAULT","jceStrengthErrors":[],"kernelVersionErrors":[],"otherErrors":[],"parcelPackageConflictErrors":[],"successfulSubcommands":4,"swappinessErrors":[],"thpErrors":[],"timeInspectionMessage":null,"timeInspectionTimezoneMessage":null,"userGroupErrors":[],"versionHostsByCDHVersion":[{"hostCDHVersion":"NONE","hosts":["lnx605.mm-corp.net","lnx606.mm-corp.net","lnx603.mm-corp.net","lnx604.mm-corp.net"]}],"versionMessages":[{"annotation":"CONCERNING","args":["4","NONE"],"messageId":"message.inspector.version.nHostsConcerningCdhVersion"},{"annotation":"CONCERNING","args":[],"messageId":"message.inspector.version.moreThanOneVersionComponentSet"},{"annotation":"CONCERNING","args":[],"messageId":"message.inspector.version.javaVersionsInconsistent"},{"annotation":"GOOD","args":[],"messageId":"message.inspector.version.daemonsVersionsMatchServer"},{"annotation":"GOOD","args":[],"messageId":"message.inspector.version.agentVersionsMatchServer"}]}

 

Re: After reboot, parcels are gone. CM shows all downloadable parcels as "unavailable"

New Contributor

Hi,  Did you find a resolution to this?  I may be running in on the same issue where the parcels went missing and CDH_version are reporting inconsistency across the cluster.  Please advise.  Thanks.

Re: After reboot, parcels are gone. CM shows all downloadable parcels as "unavailable"

Rising Star
Hi, did you find any solution? I met the same issue. Thanks.
Don't have an account?
Coming from Hortonworks? Activate your account here