Member since
03-14-2019
12
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 6895 | 09-23-2020 03:51 AM |
10-06-2020
01:59 AM
Hi @GangWar , Thank you very much for your prompt response. However there is no issue with the curl in our servers. I had investigated further and found out that the /var/log/hadoop/hdfs/namenode.log is indicating the following error. 2020-10-06 16:19:01,877 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode. org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /master/hadoop/hdfs/namenode is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first. at org.apache.hadoop.hdfs.server.namenode.FSImage.checkUpgrade(FSImage.java:411) at org.apache.hadoop.hdfs.server.namenode.FSImage.checkUpgrade(FSImage.java:418) at org.apache.hadoop.hdfs.server.namenode.FSImage.doUpgrade(FSImage.java:438) at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:310) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1090) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:714) at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:632) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:694) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710) 2020-10-06 16:19:01,879 INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: org.apache.hadoop.hdfs.server.common.InconsistentFSStateException: Directory /master/hadoop/hdfs/namenode is in an inconsistent state: previous fs state should not exist during upgrade. Finalize or rollback first. 2020-10-06 16:19:01,879 INFO timeline.HadoopTimelineMetricsSink (AbstractTimelineMetricsSink.java:getCurrentCollectorHost(291)) - No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2020-10-06 16:19:01,880 INFO namenode.NameNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at H8CC0328.stb.gov.sg/10.21.21.85 Also I tried to Finalize my upgrade but finalize HDFS is failing.The error messages are attached. Please help. Thanks and Regards, Ananya
... View more
10-05-2020
11:35 PM
Hi Experts, Please help here.I am stuck in between my upgrade. Thanks and Regards, Ananya
... View more
10-05-2020
01:56 AM
Hi Experts, We are currently in middle of upgrade from HDP 2.6.5 to HDP 3.1.5. Currently we are just before finalizing the upgrade state. As instructed we wanted to make sure everything is fine before we finalize the upgrade. However when we checked we found that the HDFS namenode is not starting. We have HA configuration for our HDFS and both the ZKFAILOVERCONTROLLER is running but HDFS namenode or standby namenode is not starting and it keep on trying without any success. I am attaching the logs and screenshots for your analysis.Please help. Thanks and Regards, Ananya
... View more
Labels:
- Labels:
-
HDFS
-
Hortonworks Data Platform (HDP)
09-23-2020
03:51 AM
1 Kudo
Hi Experts, My issue is resolved now. I had raised a ticket with cloudera and they provided my with a modified yumrpm.py file which I replaced with my existing file. This new python had option to skip in case it encounters a blank package. Following are the steps to resolve my issue. 1) verify ambari-agent version is 2.6.2.2 by `ambari-agent --version` on problematic node 2) stop ambari-agent on problematic node 3)go to /usr/lib/ambari-agent/lib/resource_management/core/providers/package/ ~> cd /usr/lib/ambari-agent/lib/resource_management/core/providers/package/ 4) take bacup of yumrpm.py ~> mv yumrpm.py /tmp 5) download the attached yumrpm.py and keep it in /user/home/ananya 6) do an diff of ~> diff /tmp/yumrpm.py /user/home/ananya/yumrpm.py >>>(diff loooks similar too this) 263,267c263,264 < if len(name) == 0: < Logger.info("Skipping installation of empty package") < else: < Logger.info("Installing package %s ('%s')" % (name, string_cmd_from_args_list(cmd))) < self.checked_call_with_retries(cmd, sudo=True, logoutput=self.get_logoutput()) --- > Logger.info("Installing package %s ('%s')" % (name, string_cmd_from_args_list(cmd))) > self.checked_call_with_retries(cmd, sudo=True, > logoutput=self.get_logoutput()) <<<< 7) move the yumrpm.py to /usr/lib/ambari-agent/lib/resource_management/core/providers/package/ mv /user/home/ananya/yumrpm.py /usr/lib/ambari-agent/lib/resource_management/core/providers/package/ 😎 start ambari-agent Thanks and Regards, Ananya
... View more
09-17-2020
03:59 AM
Hi @Bender Frenec, Really appreciate your prompt response. I have tried your suggestion. 1. I tried to deregister the 2.6.5 version but the deregister button is disabled and it is saying can not deregister as the version already installed on some nodes. 2. I checked and found no old transaction by running following command on my problematic node. ls -lart /var/lib/yum/transaction* Anything else I can check. Thanks and Regards, Ananya
... View more
09-16-2020
04:06 AM
Hi @Bender Frenec, I am on the installation binary phase and the upgrade is not yet started. If I check on the host where the installation is failing as well as where installation is successful. Both servers still got the packages for HDP 2.3.4. Is it okay to remove them before the upgrade is completed and we finalize the upgrade. Please advise. Thanks and Regards, Ananya
... View more
09-15-2020
10:34 PM
Hi Experts, Any idea on this issue please. We are currently stuck with the upgrade and can not proceed further. Thanks and Regards, Ananya
... View more
09-14-2020
10:57 PM
Hi , We are currently upgrading from HDP 2.3.4 to HDP 2.6.5 following the following reference document as suggested in the following link. https://docs.cloudera.com/HDPDocuments/Ambari-2.6.2.2/bk_ambari-upgrade/content/ambari_upgrade_guide.html Ours is an environment without internet connection so we downloaded and created the offline repository. We already upgraded the ambari from 2.4.2 to Ambari 2.6.2.2 successfully and performed all the prerequisites for HDP upgrade. We are currently got stuck in the "Register and Install Target Version" phase. We have registered the local repository for HDP 2.6.5 and from ambari>stacks and versions>manage version we are trying to install the new softwares which is failing with the following error message.This issue is only happening for only one of the hosts in cluster.Rest all other hosts the installation was successful. "Package Manager failed to install packages. Error: Execution of '/usr/bin/yum -d 0 -e 0 -y install ''' returned 1. Error: Nothing to do" Please help. Somehow it is detecting a blank package. I have attached the output and error logs. Thanks and Regards, Ananya
... View more
Labels:
- Labels:
-
Hortonworks Data Platform (HDP)