Member since
04-26-2017
5
Posts
0
Kudos Received
0
Solutions
03-19-2019
10:22 PM
2019-03-19 04:48:41,934 INFO server.AuthenticationFilter (AuthenticationFilter.java:constructSecretProvider(240)) - Unable to initialize FileSignerSecretProvider, falling back to use random secrets. 2019-03-19 04:48:41,940 INFO server.KerberosAuthenticationHandler (KerberosAuthenticationHandler.java:init(158)) - Using keytab /etc/security/keytabs/spnego.service.keytab, for principal HTTP/ppc64le-rhel75-50g-1.novalocal@EXAMPLE.COM 2019-03-19 04:48:41,951 INFO server.AbstractConnector (AbstractConnector.java:doStart(278)) - Started ServerConnector@2c5529ab{HTTP/1.1,[http/1.1]}{ppc64le-rhel75-50g-1.novalocal:50070} 2019-03-19 04:48:41,960 INFO server.AbstractConnector (AbstractConnector.java:doStop(318)) - Stopped ServerConnector@2c5529ab{HTTP/1.1,[http/1.1]}{ppc64le-rhel75-50g-1.novalocal:50070} 2019-03-19 04:48:41,963 INFO handler.ContextHandler (ContextHandler.java:doStop(910)) - Stopped o.e.j.s.ServletContextHandler@77258e59{/logs,file:///var/log/hadoop/hdfs/,UNAVAILABLE} 2019-03-19 04:48:41,967 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(210)) - Stopping NameNode metrics system... 2019-03-19 04:48:41,968 INFO impl.MetricsSinkAdapter (MetricsSinkAdapter.java:publishMetricsFromQueue(141)) - timeline thread interrupted. 2019-03-19 04:48:41,969 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:stop(216)) - NameNode metrics system stopped. 2019-03-19 04:48:41,970 INFO impl.MetricsSystemImpl (MetricsSystemImpl.java:shutdown(607)) - NameNode metrics system shutdown complete. 2019-03-19 04:48:41,970 ERROR namenode.NameNode (NameNode.java:main(1715)) - Failed to start namenode. java.io.IOException: Problem starting http server at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1165) at org.apache.hadoop.hdfs.server.namenode.NameNodeHttpServer.start(NameNodeHttpServer.java:177) at org.apache.hadoop.hdfs.server.namenode.NameNode.startHttpServer(NameNode.java:869) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:691) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:937) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:910) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1643) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1710) Caused by: javax.servlet.ServletException: Public key for signature validation must be provisioned. at org.apache.hadoop.security.authentication.server.JWTRedirectAuthenticationHandler.init(JWTRedirectAuthenticationHandler.java:125) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.initializeAuthHandler(AuthenticationFilter.java:194) at org.apache.hadoop.security.authentication.server.AuthenticationFilter.init(AuthenticationFilter.java:180) at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:139) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:873) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:349) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:778) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:262) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:161) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:113) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:131) at org.eclipse.jetty.server.Server.start(Server.java:427) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:105) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61) at org.eclipse.jetty.server.Server.doStart(Server.java:394) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.hadoop.http.HttpServer2.start(HttpServer2.java:1134) ... 7 more 2019-03-19 04:48:41,972 INFO timeline.HadoopTimelineMetricsSink (AbstractTimelineMetricsSink.java:getCurrentCollectorHost(291)) - No live collector to send metrics to. Metrics to be sent will be discarded. This message will be skipped for the next 20 times. 2019-03-19 04:48:41,973 INFO util.ExitUtil (ExitUtil.java:terminate(210)) - Exiting with status 1: java.io.IOException: Problem starting http server 2019-03-19 04:48:41,977 INFO namenode.NameNode (LogAdapter.java:info(51)) - SHUTDOWN_MSG:
... View more
Labels:
08-14-2018
10:59 AM
@Facundo Bianco Please check disk space on the master node
... View more
05-11-2017
11:21 AM
Hi, Has anyone found a fix for this? I am noticing same issues Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 150, in <module>
AmsCollector().execute()
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 313, in execute
method(env)
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 48, in start
self.configure(env, action = 'start') # for security
File "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py", line 116, in locking_configure
original_configure(obj, *args, **kw)
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/metrics_collector.py", line 43, in configure
hbase('master', action)
File "/usr/lib/python2.6/site-packages/ambari_commons/os_family_impl.py", line 89, in thunk
return fn(*args, **kwargs)
File "/var/lib/ambari-agent/cache/common-services/AMBARI_METRICS/0.1.0/package/scripts/hbase.py", line 244, in hbase
recursive_ownership = True
File "/usr/lib/python2.6/site-packages/resource_management/core/base.py", line 155, in __init__
self.env.run()
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 160, in run
self.run_action(resource, action)
File "/usr/lib/python2.6/site-packages/resource_management/core/environment.py", line 124, in run_action
provider_action()
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 199, in action_create
recursion_follow_links=self.resource.recursion_follow_links, safemode_folders=self.resource.safemode_folders)
File "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py", line 73, in _ensure_metadata
sudo.chown_recursive(path, _user_entity, _group_entity, recursion_follow_links)
File "/usr/lib/python2.6/site-packages/resource_management/core/sudo.py", line 53, in chown_recursive
os.lchown(os.path.join(root, name), uid, gid)
OSError: [Errno 2] No such file or directory: '/var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD/e71dde671eea1c7b89dc851965d74348/0/9a18f23dada64e92b3455e73376a8fdb'
[root@hdp26clmn 0]# ls -l
ls: cannot access 9a18f23dada64e92b3455e73376a8fdb: No such file or directory
total 0
?????????? ? ? ? ? ? 9a18f23dada64e92b3455e73376a8fdb
[root@hdp26clmn 0]#pwd
/var/lib/ambari-metrics-collector/hbase/data/default/METRIC_RECORD/e71dde671eea1c7b89dc851965d74348/0
[ams@hdp26clmn 0]
... View more
04-26-2017
12:26 PM
Hi, I have setup single node and multinode (1MN+3DN) cluster with HDP 2.6. I am using xfs filesystem on RHEL72. After deployment the setup looks good but after some time few nodes stops responding with the below error (can be seen in the console). sda5 is a root disk, hence I cant login and fix the filesystem. Reboot fixes the issue. Did check the disks and dint notice any issue with the disks. [16613.582270] systemd[14024]: Failed at step EXEC spawning /sbin/agetty: Input/output error
[16613.831856] systemd[14027]: Failed at step EXEC spawning /sbin/agetty: Input/output error
[16614.082272] systemd[14030]: Failed at step EXEC spawning /sbin/agetty: Input/output error
[16614.332070] systemd[14033]: Failed at step EXEC spawning /sbin/agetty: Input/output error
[16614.581558] systemd[14036]: Failed at step EXEC spawning /sbin/agetty: Input/output error
[16616.436262] XFS (sda5): xfs_log_force: error -5 returned.
[16646.515588] XFS (sda5): xfs_log_force: error -5 returned.
[16676.594914] XFS (sda5): xfs_log_force: error -5 returned.
... View more
- Tags:
- FileSystem
