Support Questions

Find answers, ask questions, and share your expertise

Cloudera 5.10 - cloudera_scm_agent seems to be deleting/uninstalling parcels/packages every minute

avatar
Contributor

Need Help here!. 

Keep getting follow messages in cloudera-scm-agent log and noticed it deletes everything from 

/etc/hadoop/ and puts back the content every minute - 

This is causing some Mapreduce tasks to fail with File not found errors.


[11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       INFO     Ensuring alternatives entries are deactivated for parcel CDH-5.10.2-1.cdh5.10.2.p0.5. [11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       INFO     Executing command ['/usr/lib64/cmf/service/common/alternatives.sh', 'deactivate', 'sqoop-import', '/usr/bin/sqoop-import', 'bin/sqoop-import', '10', 'False'] [11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       ERROR    Failed to deactivate alternatives for parcel CDH-5.10.2-1.cdh5.10.2.p0.5: 2 [11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       INFO     Executing command ['/usr/lib64/cmf/service/common/alternatives.sh', 'deactivate', 'solr-conf', '/etc/solr/conf', 'etc/solr/conf.dist', '10', 'True'] [11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       ERROR    Failed to deactivate alternatives for parcel CDH-5.10.2-1.cdh5.10.2.p0.5: 2 [11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       INFO     Executing command ['/usr/lib64/cmf/service/common/alternatives.sh', 'deactivate', 'hive-hcatalog-conf', '/etc/hive-hcatalog/conf', 'etc/hive-hcatalog/conf.dist', '10', 'True'] [11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       ERROR    Failed to deactivate alternatives for parcel CDH-5.10.2-1.cdh5.10.2.p0.5: 2 [11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       INFO     Executing command ['/usr/lib64/cmf/service/common/alternatives.sh', 'deactivate', 'llama-conf', '/etc/llama/conf', 'etc/llama/conf.dist', '10', 'True'] [11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       ERROR    Failed to deactivate alternatives for parcel CDH-5.10.2-1.cdh5.10.2.p0.5: 2 [11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       INFO     Executing command ['/usr/lib64/cmf/service/common/alternatives.sh', 'deactivate', 'sqoop-codegen', '/usr/bin/sqoop-codegen', 'bin/sqoop-codegen', '10', 'False'] [11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       ERROR    Failed to deactivate alternatives for parcel CDH-5.10.2-1.cdh5.10.2.p0.5: 2 [11/Dec/2020 18:39:34 +0000] 11426 MainThread parcel       INFO     Executing command ['/usr/lib64/cmf/service/common/alternatives.sh', 'deactivate', 'sqoop-import-all-tables', '/usr/bin/sqoop-import-all-tables', 'bin/sqoop-import-all-tables', '10', 'False'] [11/Dec/2020 18:39:35 +0000] 11426 MainThread parcel       ERROR    Failed to deactivate alternatives for parcel CDH-5.10.2-1.cdh5.10.2.p0.5: 2 [11/Dec/2020 18:39:35 +0000] 11426 MainThread parcel       INFO     Executing command ['/usr/lib64/cmf/service/common/alternatives.sh', 'deactivate', 'sqoop', '/usr/bin/sqoop', 'bin/sqoop', '10', 'False'] [11/Dec/2020 18:39:35 +0000] 11426 MainThread parcel       ERROR    Failed to deactivate alternatives for parcel CDH-5.10.2-1.cdh5.10.2.p0.5: 2 [11/Dec/2020 18:39:35 +0000] 11426 MainThread parcel       INFO     Executing command ['/usr/lib64/cmf/service/common/alternatives.sh', 'deactivate', 'cli_mt', '/usr/bin/cli_mt', 'bin/cli_mt', '10', 'False'] [11/Dec/2020 18:39:35 +0000] 11426 MainThread parcel       ERROR    Failed to deactivate alternatives for parcel CDH-5.10.2-1.cdh5.10.2.p0.5: 2 [11/Dec/2020 18:39:35 +0000] 11426 MainThread parcel       INFO     Executing command ['/usr/lib64/cmf/service/common/alternatives.sh', 'deactivate', 'impalad', '/usr/bin/impalad', 'bin/impalad', '10', 'False']
1 REPLY 1

avatar
Master Guru

@Anks2411 This could be a bug since it's an older CM version but looking at the logs this seems issue with alternatives. 

 

Can you try these steps and let me know if this improves the behaviour. 

1. Stop all roles on the host.

2. Stop the agent.

3. Ensure that the agent and supervisord are fully stopped.

4. Verify the output of the follow command and ensure that it only captures information related to cloudera.

 

\ls -l /etc/alternatives/ | grep "\/opt\/cloudera"

 

5. Run the following shell script to cleanup and remove all alternatives related to cloudera parcels.

 

$ \ls -l /etc/alternatives/ | grep "\/opt\/cloudera" | awk {'print $9'} | \
while read m; do if [[ -e /var/lib/alternatives/${m} ]] ;then echo "Removing ${m}"; \
rm -fv /var/lib/alternatives/${m} ; fi; rm -fv /etc/alternatives/${m}; done

 

6. Start the agent and review the log data in /var/log/cloudera-scm-agent/cloudera-scm-agent.log. You should see the agent attempt to create new alternatives for your parcels.

 

7. Verify where the alternative point and restart the roles on the host.

Feel free to keep us updated here. 


Cheers!
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.