question Re: Data03 node is decomissioned, i want it back in working condition. in Support Questions

Data03 node is decomissioned, i want it back in working condition.

Jessica_cisco — Fri, 16 Sep 2022 14:46:15 GMT

Hi Team,

From last few days, data03 node was not working, so we moved it to decomissioned state and restarted all our services.
Now i need to recomission and start the services here as well.
I selected the host and under actions clicked on End maintenace, but it is failing.

Please help me out with what shall i do to resolve this?

Re: Data03 node is decomissioned, i want it back in working condition.

rki_ — Fri, 03 Jun 2022 09:53:13 GMT

It could be possible that the Datanode is still in decommissioning state. Can you start to Stop that Datanode from the UI and then try to exit the Maintenance mode and check.

Re: Data03 node is decomissioned, i want it back in working condition.

Jessica_cisco — Mon, 06 Jun 2022 10:15:46 GMT

Hi @rki_ ,
Thanks for your response.
I tried that stopping all roles on this host and tried to end maintenance but still it is giving the same error.
"Role not started due to unhealthy host data-03."

Also, the log file to which it is referring is also empty.

Any suggestions, what shall i do next?

Re: Data03 node is decomissioned, i want it back in working condition.

rki_ — Mon, 06 Jun 2022 10:57:09 GMT

Hi, Do you see any errors in the Cloudera-scm-agent logs? If the agent looses connection to the CM Server, it reports the host as unhealthy. Have you tried restarting the Cloudera-scm-agent on this host and check if that helps.

Re: Data03 node is decomissioned, i want it back in working condition.

Jessica_cisco — Mon, 06 Jun 2022 11:14:21 GMT

Hi,
I could see that cloudera-scm-agent logs on this host are till last month that time it showed like data03 node has been out of contact with the cloudera manager for too long.

Also when i had run the below command to check the state -

service cloudera-scm-agent status
It is giving below error-
"Unit cloudera-scm-agent.service could not be found."

Please help me with a way forward.

Re: Data03 node is decomissioned, i want it back in working condition.

Jessica_cisco — Mon, 06 Jun 2022 11:17:56 GMT

Please find attached the screenshot of Cloudera-scm-agent log haviing the errors related to impala.

Re: Data03 node is decomissioned, i want it back in working condition.

rki_ — Mon, 06 Jun 2022 13:51:40 GMT

Hi,

Can you check if agent package is indeed present on this host. You can compare it with a working host.

# rpm -qa | grep cloudera

Re: Data03 node is decomissioned, i want it back in working condition.

Jessica_cisco — Tue, 07 Jun 2022 04:55:36 GMT

Hi @rki_ ,
Yes cloudera-manager-agent package is missing on data-03 node.

Re: Data03 node is decomissioned, i want it back in working condition.

rki_ — Tue, 07 Jun 2022 07:51:10 GMT

Hello,

Try to install the agent package on this host and let me know if that solves the issue.

Make a copy of your /etc/cloudera-scm-agent/config.ini.
Uninstall the cloudera-manager-agent package
Install the cloudera-manager-agent package
Copy the /etc/cloudera-scm-agent/config.ini back
Start the cloudera-scm-agent service

Re: Data03 node is decomissioned, i want it back in working condition.

Jessica_cisco — Tue, 07 Jun 2022 08:03:58 GMT

Hi @rki_ ,

I am doing this for the very first time.
Can you help me with the detailed steps.

Re: Data03 node is decomissioned, i want it back in working condition.

rki_ — Tue, 07 Jun 2022 08:30:53 GMT

Hello @Jessica_cisco ,

You can check if cloudera-manager.repo is present on this host under /etc/yum.repos.d/. If not, copy this repo from a working node.

If you run the below command on this host, it should show you the repo from where it will download the agent package.

# yum whatprovides cloudera-manager-agent

cloudera-manager-agent-7.6.1-24046616.el7.x86_64 : The Cloudera Manager Agent
Repo : @cloudera-manager

Once the above is confirmed, You can use the below doc for instructions.

https://docs.cloudera.com/cdp-private-cloud-base/7.1.6/installation/topics/cdpdc-manually-install-cm-agent-packages.html

Re: Data03 node is decomissioned, i want it back in working condition.

Jessica_cisco — Tue, 07 Jun 2022 09:06:38 GMT

Hi @rki_ ,

I tried searching for cloudera-manager.repo on this host as well as working datanode, but it is not found anywhere.

Also on the working node i could see that below cloudera-manager-agent is present when i tried running this command : rpm -qa 'cloudera-manager-*'

cloudera-manager-agent-6.3.0-1281944.el7.x86_64

Re: Data03 node is decomissioned, i want it back in working condition.

Jessica_cisco — Tue, 07 Jun 2022 17:09:52 GMT

Hi @rki_ ,
Thanks for helping out. I followed the above mentioned steps and now data03 node is in commissioned state and roles are started.
But still i am getting some error for Hbase. (Please refer to the screenshot attached)

Could you please me with your inputs on this.

Regards,

Jessica Jindal

Re: Data03 node is decomissioned, i want it back in working condition.

rki_ — Tue, 07 Jun 2022 18:11:27 GMT

Hi @Jessica_cisco ,

As per the screenshot, it reports that the Region server on this host has failed to start. You can login to that host and confirm if there is a Region server process running.

#ps -ef | grep regionserver

If you don't see any process, try to restart this Region server from CM and if it still fails, please check the stderr and the role log of this Region server for more clues.

Re: Data03 node is decomissioned, i want it back in working condition.

Jessica_cisco — Wed, 08 Jun 2022 11:56:04 GMT

Hi @rki_ ,

This region server issue is resolved. It was because this host time was not in sync with the other nodes.
So after the time sync and restarting the region server. It is resolved.

Now, to get the process time which will than further be used in oozie workflows, i had run the below commands on the cm-hue node -

kinit -kt /usr/local/etl/etl.keytab oozie/data-02.novalocal@CIM.IVSG.AUTH
hdfs dfs -cat /user/oozie/oozie/res/config.properties| sed -nr '/processstart/ s/processstart=|\\//g p'

But it is giving the output as below :

2022-04-19T17:02+0530

It is the same day when it stopped working.
Now my doubt is like it should give the current date and time right?

If so what shall i check now i mean any specific logs or so?

Regards,
Jessica Jindal

Re: Data03 node is decomissioned, i want it back in working condition.

Jessica_cisco — Thu, 09 Jun 2022 06:20:17 GMT

Hi @rki_ ,

I created the etl jobs.
But workflows are getting killed continuously. With the below email -

The workflow 0000085-220601112151640-oozie-oozi-W is killed, error message: RuntimeException: couldn't retrieve HBase table (cim.Locations) info:

Failed after attempts=4, exceptions:

Thu Jun 09 11:41:14 IST 2022, RpcRetryingCaller{globalStartTime=1654755074760, pause=100, maxAttempts=4}, org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: hbase:meta,,1 is not online on data-02.novalocal,16020,1652959331850

at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:3328)

at org.apach.

I checked the oozie yarn logs as well for this workflow. Please refer the attachement for that)

Please suggest me some way out.

Regards,

Jessica Jindal

Re: Data03 node is decomissioned, i want it back in working condition.

rki_ — Thu, 09 Jun 2022 07:58:47 GMT

Hi @Jessica_cisco,

The hbase:meta ( system table ) is not online and thus the Hbase Master has not come out of initialisation phase. We will need to assign the region for the hbase:meta table, as this table contains the mapping regarding which region is hosted on a specific Region server. The workflow is failing as this hbase:meta table is not assigned/online.

We need to use hbck2 jar ( we get this via support ticket) to assign this region for which the best way is to open a Cloudera Support ticket. https://docs.cloudera.com/runtime/7.2.10/troubleshooting-hbase/topics/hbase_fix_issues_hbck.html

Else, ( Doesn't work everytime )

You can try to restart the Region server data-02.novalocal followed by a restart of the Hbase Master to see if the Master is able to assign the meta table.

Regards,

Robin

Re: Data03 node is decomissioned, i want it back in working condition.

Jessica_cisco — Thu, 09 Jun 2022 10:14:37 GMT

Hi @rki_ ,

Thanks for this. I tried restarting regionserver and master, but that didn't work.
I will raise a support ticket than.

Regards,

Jessica Jindal

Regards,

Jessica Jindal