Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Error on datanode after deployment

Error on datanode after deployment

Explorer

On some of my datanodes there are strange errors. Unable to extract JSON from JMX response. Similar to this: Unable to extract JSON from JMX response. However i do not use kerberos. I used cloudbreak 1.14.0 and 1.14.4 which lead consistantly to this error. My cloud platform is openstack newton and i use and existing subnet.

6 REPLIES 6
Highlighted

Re: Error on datanode after deployment

Explorer

Btw. my HDP version is 2.6 however i am pretty sure i saw the same issues with 2.5.

I have the same error with my nodemanagers as well. On a 5 slave node cluster i only 3/5 are alive (same number with datanodes).

Highlighted

Re: Error on datanode after deployment

Super Mentor

@Marius Dieckmann

The message "Unable to extract JSON from JMX response. Similar to this: Unable to extract JSON from JMX response", Simply says that the output form the jmx endpoint is in form of json and that son data is not in correct format. But it does not tell exactly what is wrong and why the JSON data is not coming in proper format.

Usually this happens when the DataNode is not healthy. So can you please check the DataNode logs first if there are any errors? Please share the DataNode logs as well.

Please check if you are able to access the JMX URL for your DataNode? (Assuming that erie1.example.com is your DataNode hostname)

http://erie1.example.com:50075/jmx

.

Highlighted

Re: Error on datanode after deployment

Explorer

@Jay SenSharma

Hey thanks for replying.

sry for double posting if i did so. I can not see or edit my old posts and i am not sure if something went wrong.

I guess the problem is that the fqdn can not be resolved. I cant ping or get the jxm json. However if i use marius-1-master-0.[organisation].local it works fine. I think i have to change the domains handed over to ambari but i have no idea how. marius-1-master-0.[organisation].local is also the fqdn displayed in the openstack ports UI.

Cheers

Marius

This is my datanode log:

2017-06-02 07:40:13,485 WARN datanode.DataNode (BPOfferService.java:getBlockPoolId(177)) - Block pool ID needed, but service not yet registered with NN java.lang.Exception: trace at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:177) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.getActorInfoMap(BPServiceActor.java:164) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBPServiceActorInfo(DataNode.java:3151) at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193) at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175) at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117) at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83) at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:140) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:109) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:394) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:72) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:87) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:57) at java.util.Collections.list(Collections.java:5240) at io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:17) at io.prometheus.jmx.shaded.io.prometheus.client.exporter.MetricsServlet.doGet(MetricsServlet.java:41) at io.prometheus.jmx.shaded.javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at io.prometheus.jmx.shaded.javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at io.prometheus.jmx.shaded.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:648) at io.prometheus.jmx.shaded.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) at io.prometheus.jmx.shaded.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.Server.handle(Server.java:365) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988) at io.prometheus.jmx.shaded.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) at io.prometheus.jmx.shaded.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at io.prometheus.jmx.shaded.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) at io.prometheus.jmx.shaded.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) at io.prometheus.jmx.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at io.prometheus.jmx.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) 2017-06-02 07:40:13,958 WARN datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(222)) - Problem connecting to server: marius-1-master-0.openstacklocal:8020

Highlighted

Re: Error on datanode after deployment

Explorer

@Jay SenSharma

Ok i found something. My instances in ambari appear to be named like [instancename].example.com or [instancename].openstacklocal however if i look into my ports settings i found the name of the domain to be different. [instancename].xyz.local.

This is the log from one of the datanodes:

2017-06-01 15:09:56,807 WARN datanode.DataNode (BPOfferService.java:getBlockPoolId(177)) - Block pool ID needed, but service not yet registered with NN java.lang.Exception: trace at org.apache.hadoop.hdfs.server.datanode.BPOfferService.getBlockPoolId(BPOfferService.java:177) at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.getActorInfoMap(BPServiceActor.java:164) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBPServiceActorInfo(DataNode.java:3151) at sun.reflect.GeneratedMethodAccessor114.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193) at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175) at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117) at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54) at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) at com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83) at com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647) at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.scrapeBean(JmxScraper.java:140) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxScraper.doScrape(JmxScraper.java:109) at io.prometheus.jmx.shaded.io.prometheus.jmx.JmxCollector.collect(JmxCollector.java:394) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.findNextElement(CollectorRegistry.java:72) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:87) at io.prometheus.jmx.shaded.io.prometheus.client.CollectorRegistry$MetricFamilySamplesEnumeration.nextElement(CollectorRegistry.java:57) at java.util.Collections.list(Collections.java:5240) at io.prometheus.jmx.shaded.io.prometheus.client.exporter.common.TextFormat.write004(TextFormat.java:17) at io.prometheus.jmx.shaded.io.prometheus.client.exporter.MetricsServlet.doGet(MetricsServlet.java:41) at io.prometheus.jmx.shaded.javax.servlet.http.HttpServlet.service(HttpServlet.java:735) at io.prometheus.jmx.shaded.javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at io.prometheus.jmx.shaded.org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:648) at io.prometheus.jmx.shaded.org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1072) at io.prometheus.jmx.shaded.org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:382) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1006) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.Server.handle(Server.java:365) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:485) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:926) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:988) at io.prometheus.jmx.shaded.org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:635) at io.prometheus.jmx.shaded.org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at io.prometheus.jmx.shaded.org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at io.prometheus.jmx.shaded.org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) at io.prometheus.jmx.shaded.org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) at io.prometheus.jmx.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at io.prometheus.jmx.shaded.org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:745) 2017-06-01 15:09:59,043 WARN datanode.DataNode (BPServiceActor.java:retrieveNamespaceInfo(222)) - Problem connecting to server: marius-1-master-0.openstacklocal:8020

This error appears mutliple times. Every thing else in the log looked fine.

Highlighted

Re: Error on datanode after deployment

Super Mentor

@Marius Dieckmann

Proper FQDN setting through out the cluster nodes are the one very basic requirement of HDP cluster.

I have written an article sometime back showing the issue that we can encounter in Cloud based environments like Openstack.

https://community.hortonworks.com/articles/42872/why-ambari-host-might-have-different-public-host-n....

.

It will also be good to doublecheck the datanode hostname & IP Address mapping inside the "/etc/hosts" file of all the cluster nodes.

https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.1.0/bk_ambari-installation/content/set_the_host...

.

Highlighted

Re: Error on datanode after deployment

Explorer

@Jay SenSharma

Thanks for your suggestions. I tried it and it did not work. However it started working 1 week later when we changed some DNS settings. Probably this was the issue.

But many thanks regardless.

Marius

Don't have an account?
Coming from Hortonworks? Activate your account here