Skip to main content
BrianChan
BrianChan
Gold ⭐️
May 2, 2023
Solved

Parcel stuck in activation state

Hi all,

 

I have a cluster in CDP 7.1.8 private cloud based. Today I tried migrating CM server and following the instruction in https://docs.cloudera.com/cdp-private-cloud-base/7.1.8/managing-clusters/topics/cm-moving-the-cm-server-new-host2.html

 

After I manually restore the cluster following the instruction in https://docs.cloudera.com/cdp-private-cloud-base/7.1.3/configuring-clusters/topics/cm-api-import-configuration.html, the cluster cannot restart.

 

Found that in the Parcel page in CM Console, the parcels are activating but keep on loading forever.

BrianChan_0-1683022293164.png

 

Here are the log message related to parcel from cloudera-scm-agent.log:

 

[02/May/2023 03:29:55 -0400] 6258 MainThread parcel       ERROR    Error while attempting to modify permissions of file '/opt/                                                                                cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/lib/hadoop-0.20-mapreduce/sbin/Linux/task-controller'.
  File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/parcel.py", line 586, in ensure_permissions
    file = cmf.util.validate_and_open_fd(path, self.get_parcel_home(parcel))
OSError: [Errno 2] No such file or directory: '/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/lib/hadoop-0.20-mapreduc                                                                                e/sbin/Linux/task-controller'
[02/May/2023 03:29:55 -0400] 6258 MainThread downloader   INFO     Downloader path: /opt/cloudera/parcel-cache
[02/May/2023 03:29:55 -0400] 6258 MainThread parcel_cache INFO     Using /opt/cloudera/parcel-cache for parcel cache
[02/May/2023 03:29:55 -0400] 6258 MainThread throttling_logger WARNING  Failed parsing alternatives line: sqoop-export string                                                                                 index out of range  link currently points to /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/bin/sqoop-export
[02/May/2023 03:29:59 -0400] 6258 MainThread parcel_cache INFO     Deleting unmanaged parcel CDH-7.1.8-1.cdh7.1.8.p0.30990532
[02/May/2023 03:30:40 -0400] 6258 MainThread parcel_cache INFO     Deleting unmanaged parcel SPARK3-3.3.0.3.3.7180.0-274-1.p0.                                                                                31212967
[02/May/2023 03:30:40 -0400] 6258 MainThread parcel       INFO     prepare_environment begin: {}, [], []
[02/May/2023 03:30:40 -0400] 6258 MainThread parcel       INFO     No parcels activated for use
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:31:30 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:41:21 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:44:42 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:44:43 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:44:43 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 03:56:51 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 04:44:58 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 04:57:36 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {}, [], []
[02/May/2023 04:57:36 -0400] 6258 __run_queue parcel       INFO     No parcels activated for use
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:05:20 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 05:05:21 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:05:21 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel       INFO     prepare_environment begin: {u'SPARK3': u'3.3.0.3.3.7180.0-                                                                                274-1.p0.31212967', u'CFM': u'2.1.5.0-215', u'CDH': u'7.1.8-1.cdh7.1.8.p0.30990532'}, [], []
[02/May/2023 05:13:11 -0400] 6258 __run_queue parcel       INFO     Service does not request any parcels
[02/May/2023 05:30:15 -0400] 28744 MainThread agent        INFO     To override these variables, use /etc/cloudera-scm-agent/c                                                                                onfig.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[02/May/2023 05:30:19 -0400] 28744 MainThread agent        INFO     Previously active parcels: {}
[02/May/2023 05:30:19 -0400] 28744 MainThread agent        INFO     Using parcels directory from server provided value: /opt/c                                                                                loudera/parcels
[02/May/2023 05:30:19 -0400] 28744 MainThread parcel       INFO     Agent does create users/groups
[02/May/2023 05:30:19 -0400] 28744 MainThread parcel       INFO     Agent does parcel permissions
[02/May/2023 05:30:19 -0400] 28744 MainThread downloader   INFO     Downloader path: /opt/cloudera/parcel-cache
[02/May/2023 05:30:19 -0400] 28744 MainThread parcel_cache INFO     Using /opt/cloudera/parcel-cache for parcel cache
[02/May/2023 05:30:19 -0400] 28744 MainThread throttling_logger WARNING  Failed parsing alternatives line: sqoop-export string                                                                                 index out of range  link currently points to /opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/bin/sqoop-export
[02/May/2023 05:31:59 -0400] 29223 MainThread agent        INFO     To override these variables, use /etc/cloudera-scm-agent/c                                                                                onfig.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[02/May/2023 05:32:03 -0400] 29223 MainThread agent        INFO     Previously active parcels: {}
[02/May/2023 05:32:03 -0400] 29223 MainThread agent        INFO     Using parcels directory from server provided value: /opt/c                                                                                loudera/parcels

 

  

Please let me know if I need to provide any further information. Thank you.

BrianChan
BrianChanAnswer
Gold ⭐️
May 9, 2023

Thank you for your help @kolli_sandeep . Finally I made an API call to interrupt the Parcels activation, then remove the corresponding Parcels file and the ./flood folder in /opt/cloudera/parcels.

 

The parcels can successfully redistribute and activate after that. 

View in thread

6 replies

kolli_sandeep
kolli_sandeep
Silver ⭐️
May 3, 2023

Hi @BrianChan

Hope the /opt file system permissions are correct. Also make sure you got enough space.

[Errno 2] No such file or directory: '/opt/cloudera/parcels/CDH-7.1.8-1.cdh7.1.8.p0.30990532/

 

Translate
BrianChan
BrianChanAuthorAnswer
Gold ⭐️
May 9, 2023

Thank you for your help @kolli_sandeep . Finally I made an API call to interrupt the Parcels activation, then remove the corresponding Parcels file and the ./flood folder in /opt/cloudera/parcels.

 

The parcels can successfully redistribute and activate after that. 

Translate
wjsandman
wjsandman
Gold ⭐️
February 27, 2024

Could you post the API call you used please? I'm having the. same issue and see nothing in the API documenttion that would allow interrupting the Parcel Activation. 

Translate
    DianaTorres
    Community Manager
    DianaTorres
    Community Manager
    February 27, 2024

    @wjsandman As this is an older post, you would have a better chance of receiving a resolution by starting a new thread. This will also be an opportunity to provide details specific to your environment that could aid others in assisting you with a more accurate answer to your question. You can link this thread as a reference in your new post. Thanks.

    Translate
    Regards, Diana Torres, Senior Community Moderator
      wjsandman
      wjsandman
      Gold ⭐️
      February 29, 2024

      I had the exact issue described here.

      What ended up working for me was

       

      curl -k -u userid:password -X POST https://yourhost-yourdomain.com:7183/api/v50/clusters/sandboxCluster/parcels/products/CFM/versions/2.1.6.0-323/commands/deactivate

      This, obviously, was for Cloudera Flow Manafer 2.1.6.0-323

       

      Translate
      Terms & ConditionsAccessibility statement