Re: How to start namenode with filenotfoundexcepti...

scott1 · ‎02-11-2019

It seems that the issue was not about hbase, but instead related to the namenode. Performing a bootstrapStanby after a backup resolved the issue. The biggest concern is that this was done without putting the cluster into safe mode, since the second namenode could not start up.

scott1 · ‎02-11-2019

This is a production cluster that is kerberized. There have been a bunch of ongoing problems over the last week, of which this is one. I have 2 hbase masters in the cluster at the moment. I don't have any hbase dependent services, just an internally developed service that we can easily recreate the data for. Given that I have been having issues all week I am hesitant to restart the cluster. The only error I found in the base log is 2019-02-08 14:53:35,713 WARN [Thread-18] wal.WALProcedureStore: Unable to read tracker for hdfs://cluster/apps/hbase/data/MasterProcWALs/pv2-00000000000000000336.logorg.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat$InvalidWALDataException: Missing trailer: size=19 startPos=19 at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFormat.readTrailer(ProcedureWALFormat.java:183) at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFile.readTrailer(ProcedureWALFile.java:93) at org.apache.hadoop.hbase.procedure2.store.wal.ProcedureWALFile.readTracker(ProcedureWALFile.java:100) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLog(WALProcedureStore.java:1386) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.initOldLogs(WALProcedureStore.java:1335) at org.apache.hadoop.hbase.procedure2.store.wal.WALProcedureStore.recoverLease(WALProcedureStore.java:416) at org.apache.hadoop.hbase.procedure2.ProcedureExecutor.init(ProcedureExecutor.java:714) at org.apache.hadoop.hbase.master.HMaster.createProcedureExecutor(HMaster.java:1398) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:857) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2225) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:568) at java.lang.Thread.run(Thread.java:745)

scott1 · ‎02-08-2019

Interestingly restarting hbase deletes the directory and file that I created. If I create it and only restart the namenode it still says the file is missing, even though I see it with hdfs dfs -ls. Maybe some issue with a 0 length file?

scott1 · ‎02-08-2019

@Geoffrey Shelton Okot Sorry that it was unclear, there is a specific file in that directory it complains about. So in that case do you suggest hdfs dfs -touch thefilename then changing the owner?

scott1 · ‎02-08-2019

@Geoffrey Shelton Okot Sorry that it was unclear, there is a specific file in that directory it complains about. So in that case do you suggest `hdfs dfs -touch thefilename` then changing the owner?

scott1 · ‎02-08-2019

I am running into an issue with starting my namenode (HDP 3.1) which failes on a filenotfoundexception. It complains about a file in /apps/hbase/data/WALs/. I ran hdfs fsck / and the report shows that the filesystem is healthy. I am not sure why this file doesn't exist, or why the namenode cares about it existing. Is there a way to force the namenode to start with the file missing?

scott1 · ‎01-23-2019

I just upgraded ambari to 2.7.3 on my cluster. The cluster had previously been configured with kerberos, and running correctly. Now whenever I try to modify components I get an "admin session expired" dialog. I enter my admin credentials (kerberos) and get an invalid KDC admin error returned. I verified that the principal I entered has admin rights by logging into kadmin on the command line. The principal I am entering is the one I used when I initially configured the cluster to use kerberos. Why would this not work anymore, and how can I fix it?

scott1 · ‎02-22-2018

Additional details: If I switch from HTTPS_ONLY to HTTP_AND_HTTPS the metrics work. If I then switch back to HTTPS_ONLY the metrics are all unavailable again.

scott1 · ‎02-20-2018

After enabling ssl and setting hadoop to https_only a bunch of the metrics in ambari are unavailable. Specifically those related to HDFS. Is there a setting that needs to be changed to have ambari metrics worth with the https endpoints?

scott1 · ‎02-06-2018

Yes. I uninstalled and reinstalled and it worked this time. Not sure why.

scott1 · ‎02-05-2018

The base URL possibilities in HDP-2.6 are: HDP-2.6 HDP-2.6-GPL HDP-UTILS-1.1.0.22 I must be missing something.

scott1 · ‎02-05-2018

Yes, I followed the doc you reference. I installed the mpack, registered HDF 3.1 with the correct base URL, and attempted to add the nifi service. The above error is what I ran into. It looks like HDP 2.6.4 has nifi 1.2 listed, and HDF 3.1 has 1.5. I then realized that I had never installed the HDF stack onto my cluster, but when I try that through the web interface the button doesn't actually do anything. When I inspect it in the browser it links to javascript:void(null) as the command to execute, so I don't think hdf 3.1 is actually correctly installed because of that. To verify Step 4 should be changing the base for HDF in HDF 3.1 not in HDP 2.6.4, correct?

scott1 · ‎02-05-2018

I installed the mpack for HDF 3.1 on my existing cluster managed with Ambari (2.6.1.0) on Ubuntu 16.04. I registed HDF 3.1 in the 'versions' section of cluster management. This allows me to select NiFi as a service to install, however when I go to actually deploy NiFi it fails. I get the following error. E: Unable to locate package nifi-3-0-* E: Couldn't find any package by glob 'nifi-3-0-*' E: Couldn't find any package by regex 'nifi-3-0-*' Update: I think there is an issue with the "Install On" button in managing stack versions. I can not install the HDF stack or a different HDP stack to my cluster. I think this is why it is looking for 1.2 even though 1.5 of nifi is what apt can actually find.

scott1 · ‎02-02-2018

Online	Offline
Last Visited	‎02-11-2019 05:01 PM

Date Registered	‎02-02-2018 08:26 PM
Last Visited	‎02-11-2019 05:01 PM
Total Messages Posted	16

Re: How to start namenode with filenotfoundexcepti...

Re: How to start namenode with filenotfoundexcepti...

Re: How to start namenode with filenotfoundexcepti...

Re: How to start namenode with filenotfoundexcepti...

Re: How to start namenode with filenotfoundexcepti...

How to start namenode with filenotfoundexception?

invalid KDC administrator credentials after upgrad...

Re: Ambari metrics N/A after enabling SSL for HDFS...

Ambari metrics N/A after enabling SSL for HDFS and...

Re: NiFi install failure. HDP 2.6.4.0 and HDF 3.1...

Re: NiFi install failure. HDP 2.6.4.0 and HDF 3.1...

Re: NiFi install failure. HDP 2.6.4.0 and HDF 3.1...

NiFi install failure. HDP 2.6.4.0 and HDF 3.1.0

Is there a HDF stack addon for Ubuntu 16.04? The ...