HDP3.0: timeline service V2 reader cannot create zookeeper nodes.

[#1]

Hi,

I am using HDP3.0 and ambari 2.7 blueprint to install my cluster. Yarn's timeline service V2 reader cannot create zookeeper node:

at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:168) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:323) at java.lang.Thread.run(Thread.java:745) 2018-08-14 17:59:55,827 INFO [main] client.RpcRetryingCallerImpl: Call exception, tries=6, retries=6, started=4142 ms ago, cancelled=false, msg=org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/meta-region-server, details=row 'prod.timelineservice.entity' on table 'hbase:meta' at null

I observed yarn is using timeline service 1.5 and timeline service reader v2. I am not sure if this is expected. My blueprint is using:

{
    "name": "APP_TIMELINE_SERVER"
},

Adding hbase client per this post did not help. Any idea? Thanks.

[#2]

Looks like this is also caused by noexec /tmp like this post:

Suppressed: java.lang.UnsatisfiedLinkError: /tmp/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_644869269367588546881.so: /tmp/liborg_apache_hbase_thirdparty_netty_transport_

native_epoll_x86_644869269367588546881.so: failed to map segment from shared object: Operation not permitted

                at java.lang.ClassLoader$NativeLibrary.load(Native Method)

                at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

                at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

                at java.lang.Runtime.load0(Runtime.java:809)

                at java.lang.System.load(System.java:1086)

                at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:498)

                at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)

                at java.security.AccessController.doPrivileged(Native Method)

                at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)

                at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)

                ... 27 more

No sure what setting can be used to make yarn not use netty.

[#3]

@Lian Jiang

In your blueprint you have added "APP_TIMELINE_SERVER", So can you please check if you have also added the "TIMELINE_READER" as following?

        {
          "name" : "APP_TIMELINE_SERVER"
        },
        {
          "name" : "TIMELINE_READER"
        },

.

[#4]

I also hit the same issue. But modify hbase_java_io_tmpdir does not work for me...

Any other suggestion?

[#5 · Accepted solution]

@Lian Jiang

Also please find the below properties values from your Blueprint file and change the values to a path which has proper permission:

Specially inside "yarn-hbase-env"

hbase_java_io_tmpdir
yarn_hbase_java_io_tmpdir

.

[#6]

Thanks @Jay Kumar SenSharma

This setting worked:

"yarn-hbase-env": {
        "properties": {
          "hbase_java_io_tmpdir": "/u01/tmp"
        }
      }

These setting made TS reader v2 start. Progress!

However, after some time, it stopped again due to the issue described in this post. The reason is that the yarn hbase regional server cannot start (though the yarn hbase master starts). I guess it is still related to the fact that regional server is using /tmp.

In ambari UI, I cannot find a place to set yarn_hbase_java_io_tmpdir. Any idea?

[#7]

@Lian Jiang

Good to know that with the help of "hbase_java_io_tmpdir" your issue partially solved.

Regarding the "yarn_hbase_java_io_tmpdir" property, you can find it inside the "Advanced yarn-hbase-env" as following:

Ambari UI --> Yarn --> Configs --> Advanced --> "Advanced yarn-hbase-env" --> "hbase-env template"

      export HBASE_OPTS="$HBASE_OPTS -XX:+UseConcMarkSweepGC -XX:ErrorFile=$HBASE_LOG_DIR/hs_err_pid%p.log -Djava.io.tmpdir={{yarn_hbase_java_io_tmpdir}}"

.

So just replace the value of {{yarn_hbase_java_io_tmpdir}} with your desired value. The Default value of the "yarn_hbase_java_io_tmpdir" is calculated as following:

# grep 'yarn_hbase_java_io_tmpdir' /var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/params_linux.py
yarn_hbase_java_io_tmpdir = default("/configurations/yarn-hbase-env/hbase_java_io_tmpdir", "/tmp")

[#8]

It turned out hbase_java_io_tmpdir is enough. The hbase regional server not starting issue is because our security setting disabled the yarn-ats user whose uid is out of a range. After creating this user in the desired range, ts reader v2 worked.

[#9]

@Lian Jiang

Correct we do not need to worry about the "yarn_hbase_java_io_tmpdir" property as it is basically being controlled by the property "hbase_java_io_tmpdir" hence if we do not set the hbase_java_io_tmpdir property on our own then the default value for both the property will be "/tmp"

yarn_hbase_java_io_tmpdir = default("/configurations/yarn-hbase-env/hbase_java_io_tmpdir", "/tmp")

As your reader seems to be working fine now. hence it will be great if you ca mark this thread as answered so that it will be useful for other HCC users to know the details about these properties and can quickly browse the answers.

[#10]

I also hit the same issue. But modify hbase_java_io_tmpdir does not work for me...

Any other suggestion?