Support Questions
Find answers, ask questions, and share your expertise

Datanode is failing to start.... Its a Kerberosed cluster. It was working earlier.. After enabling DPS its failing...

Datanode is failing to start.... Its a Kerberosed cluster. It was working earlier.. After enabling DPS its failing...

Explorer

2019-03-19 04:48:41,934 INFO server.AuthenticationFilter (AuthenticationFilter.java:constructSecretProvider(240)) - Unable to initialize FileSignerSecretProvider, falling back to use random secrets.

2019-03-19 04:48:41,940 INFO server.KerberosAuthenticationHandler (KerberosAuthenticationHandler.java:init(158)) - Using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/ppc64le-rhel75-50g-1.novalocal@EXAMPLE.COM

2019-03-19 04:48:41,951 INFO server.AbstractConnector (AbstractConnector.java:doStart(278)) - Started ServerConnector@2c5529ab{HTTP/1.1,[http/1.1]}{ppc64le-rhel75-50g-1.novalocal:50070}

2019-03-19 04:48:41,960 INFO server.AbstractConnector (AbstractConnector.java:doStop(318)) - Stopped ServerConnector@2c5529ab{HTTP/1.1,[http/1.1]}{ppc64le-rhel75-50g-1.novalocal:50070}

2019-03-19 04:48:41,963 INFO handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.s.ServletContextHandler@77258e59{/logs,file:///var/log/hadoop/hdfs/,UNAVAILABLE}

2019-03-19 04:48:41,967 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping NameNode metrics system...

2019-03-19 04:48:41,968 INFO impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted.

2019-03-19 04:48:41,969 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - NameNode metrics system stopped.

2019-03-19 04:48:41,970 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - NameNode metrics system shutdown complete.

2019-03-19 04:48:41,970 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode.

java.io.IOException: Problem starting http server

at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1165)

at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:177)

at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:869)

at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:691)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937)

at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910)

at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643)

at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710)

Caused by: javax.servlet.ServletException: Public key for signature validation must be provisioned.

at org.apache.hadoop.security.authentication.server.JWTRedirectAuthenticationHandler.init(JWTRedirectAuthenticationHandler.java:125)

at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:194)

at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:180)

at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139)

at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873)

at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:349)

at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778)

at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262)

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)

at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161)

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113)

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)

at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131)

at org.eclipse.jetty.server.Server.start(Server.java:427)

at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105)

at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)

at org.eclipse.jetty.server.Server.doStart(Server.java:394)

at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)

at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1134)

... 7 more

2019-03-19 04:48:41,972 INFO timeline.HadoopTimelineMetricsSink (AbstractTimelineMetricsSink.java:getCurrentCollectorHost(291)) - No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times.

2019-03-19 04:48:41,973 INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: java.io.IOException: Problem starting http server

2019-03-19 04:48:41,977 INFO namenode.NameNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG:


1 REPLY 1
Highlighted

Re: Datanode is failing to start.... Its a Kerberosed cluster. It was working earlier.. After enabling DPS its failing...

New Contributor

Any luck on this @Santoshkumar Hiremath? I am facing a similar issue. Any help/pointer is appreciated. Thanks.


Don't have an account?