Member since
11-04-2019
2
Posts
0
Kudos Received
0
Solutions
12-01-2019
10:52 AM
I'm trying to access Yarn UI (ResourceManager UI) over knox (https://<knox-host>:8443/gateway/default/yarn but I don't get it to work. Ambari, ranger and atlas is working correctly but trying to access yarn I get the following error message HTTP ERROR 403 Problem accessing /gateway/default/yarn. Reason: java.lang.IllegalArgumentException My environment is HDP 3.1, the environment is kerberized and I've configured the advanced topology with the following information <service> <role>YARN</role> <url>http://<rm-host>:8088</url> </service> <service> <role>YARNUI</role> <url>http://<rm-host>:8088</url> </service> Does anyone have any suggestions on what I should do to get it to work?
... View more
11-04-2019
11:48 AM
Running Ambari 2.7.3 and HDP 3.1.0.0-78.
The cluster is kerberized and the Timeline Service v2.0 Reader starts but we get an alert that doesn't go away.
ATSv2 HBase Application
Traceback (most recent call last):
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/alerts/alert_ats_hbase.py", line 183, in execute
ats_hbase_app_info = make_valid_json(output)
File "/var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/alerts/alert_ats_hbase.py", line 226, in make_valid_json
raise Fail("Couldn't validate the received output for JSON parsing.")
Fail: Couldn't validate the received output for JSON parsing.
The following is written in the resourcemanager log
2019-11-04 19:54:16,482 ERROR org.apache.hadoop.hbase.client.AsyncProcess: Failed to get region location java.io.IOException: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-secure2/meta-region-server at org.apache.hadoop.hbase.client.ConnectionImplementation.get(ConnectionImplementation.java:2002) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateMeta(ConnectionImplementation.java:762) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:729) at org.apache.hadoop.hbase.client.ConnectionImplementation.relocateRegion(ConnectionImplementation.java:707) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegionInMeta(ConnectionImplementation.java:911) at org.apache.hadoop.hbase.client.ConnectionImplementation.locateRegion(ConnectionImplementation.java:732) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:281) at org.apache.hadoop.hbase.client.AsyncProcess.submit(AsyncProcess.java:236) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.backgroundFlushCommits(BufferedMutatorImpl.java:321) at org.apache.hadoop.hbase.client.BufferedMutatorImpl.flush(BufferedMutatorImpl.java:285) at org.apache.hadoop.yarn.server.timelineservice.storage.common.TypedBufferedMutator.flush(TypedBufferedMutator.java:66) at org.apache.hadoop.yarn.server.timelineservice.storage.HBaseTimelineWriterImpl.flush(HBaseTimelineWriterImpl.java:566) at org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector.flushBufferedTimelineEntities(TimelineCollector.java:173) at org.apache.hadoop.yarn.server.timelineservice.collector.TimelineCollector.putEntities(TimelineCollector.java:150) at org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.putEntity(TimelineServiceV2Publisher.java:459) at org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher.access$100(TimelineServiceV2Publisher.java:73) at org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:494) at org.apache.hadoop.yarn.server.resourcemanager.metrics.TimelineServiceV2Publisher$TimelineV2EventHandler.handle(TimelineServiceV2Publisher.java:483) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:197) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:126) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-secure2/meta-region-server at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:164) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:321) ... 1 more
ats-hbase is running as a system service and we can see that the ats-hbase service is running using 'yarn app -list' as the yarn user.
We can not get the status of the service using 'yarn app -status ats-hbase' but if we specify the appId instead we get the status.
Looking in ZooKeeper we have no node for neither atsv2-hbase-secure nor atsv2-hbase-secure2.
No json-file is created under /user/yarn/.yarn/services/ats-hbase in hdfs.
Anyone having an idea about what could be wrong?
... View more
Labels: