Support Questions
Find answers, ask questions, and share your expertise

Namenode can't be started due to PEM may be corrupt

Namenode can't be started due to PEM may be corrupt

New Contributor

I'm new to HDP and I'm using HDP 3.1.

I was able to run namenode last week and I turn down the servers during the weekend. However, I was unable to start namenode today.

Logs from namenode:

2019-01-07 04:09:38,264 INFO  http.HttpRequestLog (HttpRequestLog.java:getRequestLog(81)) - Http request log for http.requests.namenode is not defined
2019-01-07 04:09:38,271 INFO  http.HttpServer2 (HttpServer2.java:addGlobalFilter(968)) - Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$Quoting
InputFilter)
2019-01-07 04:09:38,275 INFO  http.HttpServer2 (HttpServer2.java:addFilter(941)) - Added filter authentication (class=org.apache.hadoop.security.authentication.server.
AuthenticationFilter) to context hdfs
2019-01-07 04:09:38,275 INFO  http.HttpServer2 (HttpServer2.java:addFilter(951)) - Added filter authentication (class=org.apache.hadoop.security.authentication.server.
AuthenticationFilter) to context static
2019-01-07 04:09:38,275 INFO  http.HttpServer2 (HttpServer2.java:addFilter(951)) - Added filter authentication (class=org.apache.hadoop.security.authentication.server.
AuthenticationFilter) to context logs
2019-01-07 04:09:38,275 INFO  security.HttpCrossOriginFilterInitializer (HttpCrossOriginFilterInitializer.java:initFilter(49)) - CORS filter not enabled. Please set ha
doop.http.cross-origin.enabled to 'true' to enable it
2019-01-07 04:09:38,315 INFO  http.HttpServer2 (NameNodeHttpServer.java:initWebHdfs(100)) - Added filter 'org.apache.hadoop.hdfs.web.AuthFilter' (class=org.apache.hado
op.hdfs.web.AuthFilter)
2019-01-07 04:09:38,316 INFO  http.HttpServer2 (HttpServer2.java:addJerseyResourcePackage(787)) - addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.n
amenode.web.resources;org.apache.hadoop.hdfs.web.resources, pathSpec=/webhdfs/v1/*
2019-01-07 04:09:38,319 INFO  http.HttpServer2 (HttpServer2.java:addInternalServlet(864)) - Adding Kerberos (SPNEGO) filter to fsck
2019-01-07 04:09:38,319 INFO  http.HttpServer2 (HttpServer2.java:addInternalServlet(864)) - Adding Kerberos (SPNEGO) filter to imagetransfer
2019-01-07 04:09:38,324 INFO  http.HttpServer2 (HttpServer2.java:bindListener(1185)) - Jetty bound to port 50470
2019-01-07 04:09:38,326 INFO  server.Server (Server.java:doStart(351)) - jetty-9.3.24.v20180605, build timestamp: 2018-06-05T13:11:56-04:00, git hash: 84205aa28f11a4f3
1f2a3b86d1bba2cc8ab69827
2019-01-07 04:09:38,367 INFO  server.AuthenticationFilter (AuthenticationFilter.java:constructSecretProvider(240)) - Unable to initialize FileSignerSecretProvider, fal
ling back to use random secrets.
2019-01-07 04:09:38,371 INFO  server.KerberosAuthenticationHandler (KerberosAuthenticationHandler.java:init(158)) - Using keytab /etc/security/keytabs/spnego.service.k
eytab, for principal HTTP/master1.int.com@INT.COM
2019-01-07 04:09:38,390 INFO  ssl.SslContextFactory (SslContextFactory.java:load(290)) - x509=X509@5b1f29fa(caroot,h=[],w=[int.com]) for SslContextFactory@aeab9a1(file
:///etc/security/serverKeys/server-keystore.jks,file:///etc/security/clientKeys/all.jks)
2019-01-07 04:09:38,390 INFO  ssl.SslContextFactory (SslContextFactory.java:load(290)) - x509=X509@40f70521(localhost,h=[master1.int.com],w=[]) for SslContextFactory@a
eab9a1(file:///etc/security/serverKeys/server-keystore.jks,file:///etc/security/clientKeys/all.jks)
2019-01-07 04:09:38,443 INFO  server.AbstractConnector (AbstractConnector.java:doStart(278)) - Started ServerConnector@1af146{SSL,[ssl, http/1.1]}{master1.int.com:5047
0}
2019-01-07 04:09:38,451 INFO  server.AbstractConnector (AbstractConnector.java:doStop(318)) - Stopped ServerConnector@1af146{SSL,[ssl, http/1.1]}{master1.int.com:50470
}
2019-01-07 04:09:38,453 INFO  handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.s.ServletContextHandler@4b45dcb8{/logs,file:///var/log/hadoop/hd
fs/,UNAVAILABLE}
2019-01-07 04:09:38,457 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping NameNode metrics system...
2019-01-07 04:09:38,458 INFO  impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted.
2019-01-07 04:09:38,459 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - NameNode metrics system stopped.
2019-01-07 04:09:38,460 INFO  impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - NameNode metrics system shutdown complete.
2019-01-07 04:09:38,461 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode.
java.io.IOException: Problem starting http server
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1165)
        at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:177)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:869)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:691)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)
Caused by: javax.servlet.ServletException: CertificateException - PEM may be corrupt
        at org.apache.hadoop.security.authentication.util.CertificateUtil.parseRSAPublicKey(CertificateUtil.java:60)
        at org.apache.hadoop.security.authentication.server.JWTRedirectAuthenticationHandler.init(JWTRedirectAuthenticationHandler.java:128)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:194)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:180)
        at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)
        at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)
        at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:349)
        at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)
        at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)
        at org.eclipse.jetty.server.Server.start(Server.java:427)
        at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)
        at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
        at org.eclipse.jetty.server.Server.doStart(Server.java:394)
        at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
        at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1134)
        ... 7 more
Caused by: java.security.cert.CertificateParsingException: signed fields invalid
        at sun.security.x509.X509CertImpl.parse(X509CertImpl.java:1791)
        at sun.security.x509.X509CertImpl.<init>(X509CertImpl.java:195)
        at sun.security.provider.X509Factory.engineGenerateCertificate(X509Factory.java:102)
        at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:339)
        at org.apache.hadoop.security.authentication.util.CertificateUtil.parseRSAPublicKey(CertificateUtil.java:50)
        ... 32 more
2019-01-07 04:09:38,465 INFO  util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: java.io.IOException: Problem starting http server
2019-01-07 04:09:38,509 INFO  namenode.NameNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at master1.int.com/10.52.179.102
************************************************************/

At my first thought, I think server-keystore.jks and all.jks may be corrupt. So I recreated them. But it doesn't work at all.

Can anyone help with this? Thanks a lot!