Skip to main content
Jessica_cisco
Jessica_cisco
Gold ⭐️
June 3, 2022
Solved

Data03 node is decomissioned, i want it back in working condition.

Hi Team,

From last few days, data03 node was not working, so we moved it to decomissioned state and restarted all our services. 
Now i need to recomission and start the services here as well.
I selected the host and under actions clicked on End maintenace, but it is failing.

Please help me out with what shall i do to resolve this?


clo.png
rki_
rki_Answer
Gold ⭐️
June 7, 2022

Hello,

Try to install the agent package on this host and let me know if that solves the issue.

  • Make a copy of your /etc/cloudera-scm-agent/config.ini.
  • Uninstall the cloudera-manager-agent package
  • Install the cloudera-manager-agent package
  • Copy the /etc/cloudera-scm-agent/config.ini back
  • Start the cloudera-scm-agent service
View in thread

17 replies

rki_
Gold ⭐️
rki_
Gold ⭐️
June 3, 2022

It could be possible that the Datanode is still in decommissioning state. Can you start to Stop that Datanode from the UI and then try to exit the Maintenance mode and check.

Translate
Jessica_cisco
Jessica_ciscoAuthor
Gold ⭐️
June 6, 2022

Hi @rki_ ,
Thanks for your response.
I tried that stopping all roles on this host and tried to end maintenance but still it is giving the same error.
"Role not started due to unhealthy host data-03."

Also, the log file to which it is referring is also empty.

Any suggestions, what shall i do next?

 

Translate
rki_
Gold ⭐️
rki_
Gold ⭐️
June 6, 2022

Hi, Do you see any errors in the Cloudera-scm-agent logs? If the agent looses connection to the CM Server, it reports the host as unhealthy. Have you tried restarting the Cloudera-scm-agent on this host and check if that helps.

Translate
Jessica_cisco
Jessica_ciscoAuthor
Gold ⭐️
June 6, 2022

Hi,
I could see that cloudera-scm-agent logs on this host are till last month that time it showed like data03 node has been out of contact with the cloudera manager for too long.

Also when i had run the below command to check the state -

    service cloudera-scm-agent status
It is giving below error-
     "Unit cloudera-scm-agent.service could not be found."

Please help me with a way forward.

 

Translate
Jessica_cisco
Jessica_ciscoAuthor
Gold ⭐️
June 6, 2022

Please find attached the screenshot of Cloudera-scm-agent log haviing the errors related to impala.


datanode_error.jpg
Translate
rki_
Gold ⭐️
rki_
Gold ⭐️
June 6, 2022

Hi,

Can you check if agent package is indeed present on this host. You can compare it with a working host.

# rpm -qa | grep cloudera

 

Translate
    Jessica_cisco
    Jessica_ciscoAuthor
    Gold ⭐️
    June 7, 2022

    Hi @rki_ ,
    Yes cloudera-manager-agent package is missing on data-03 node.

    Translate
    rki_
    Gold ⭐️
    rki_Answer
    Gold ⭐️
    June 7, 2022

    Hello,

    Try to install the agent package on this host and let me know if that solves the issue.

    • Make a copy of your /etc/cloudera-scm-agent/config.ini.
    • Uninstall the cloudera-manager-agent package
    • Install the cloudera-manager-agent package
    • Copy the /etc/cloudera-scm-agent/config.ini back
    • Start the cloudera-scm-agent service
    Translate
      Jessica_cisco
      Jessica_ciscoAuthor
      Gold ⭐️
      June 7, 2022

      Hi @rki_ ,

      I am doing this for the very first time.
      Can you help me with the detailed steps.


      Translate
      rki_
      Gold ⭐️
      rki_
      Gold ⭐️
      June 7, 2022

      Hello @Jessica_cisco ,

       

      You can check if cloudera-manager.repo is present on this host under /etc/yum.repos.d/. If not, copy this repo from a working node. 

      If you run the below command on this host, it should show you the repo from where it will download the agent package.

       

      # yum whatprovides cloudera-manager-agent

      cloudera-manager-agent-7.6.1-24046616.el7.x86_64 : The Cloudera Manager Agent
      Repo : @cloudera-manager

       

      Once the above is confirmed, You can use the below doc for instructions. 

      https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/installation/topics/cdpdc-manually-install-cm-agent-packages.html

      Translate
      Jessica_cisco
      Jessica_ciscoAuthor
      Gold ⭐️
      June 7, 2022

      Hi @rki_ ,

      I tried searching for cloudera-manager.repo on this host as well as working datanode, but it is not found anywhere.

      Also on the working node i could see that below cloudera-manager-agent is present when i tried running this command :    rpm -qa 'cloudera-manager-*'

                  cloudera-manager-agent-6.3.0-1281944.el7.x86_64

      Translate
      Terms & ConditionsAccessibility statement