Support Questions
Find answers, ask questions, and share your expertise

HDP3.0: timeline service V2 reader cannot create zookeeper nodes.

Contributor

Hi,

I am using HDP3.0 and ambari 2.7 blueprint to install my cluster. Yarn's timeline service V2 reader cannot create zookeeper node:

at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$ZKTask$1.exec(ReadOnlyZKClient.java:168) at org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient.run(ReadOnlyZKClient.java:323) at java.lang.Thread.run(Thread.java:745) 2018-08-14 17:59:55,827 INFO [main] client.RpcRetryingCallerImpl: Call exception, tries=6, retries=6, started=4142 ms ago, cancelled=false, msg=org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /atsv2-hbase-unsecure/meta-region-server, details=row 'prod.timelineservice.entity' on table 'hbase:meta' at null

I observed yarn is using timeline service 1.5 and timeline service reader v2. I am not sure if this is expected. My blueprint is using:

{
    "name": "APP_TIMELINE_SERVER"
},

Adding hbase client per this post did not help. Any idea? Thanks.

1 ACCEPTED SOLUTION

Accepted Solutions

Super Mentor

@Lian Jiang

Also please find the below properties values from your Blueprint file and change the values to a path which has proper permission:

Specially inside "yarn-hbase-env"

hbase_java_io_tmpdir
yarn_hbase_java_io_tmpdir

.

View solution in original post

13 REPLIES 13

Contributor

Looks like this is also caused by noexec /tmp like this post:







Suppressed: java.lang.UnsatisfiedLinkError: /tmp/liborg_apache_hbase_thirdparty_netty_transport_native_epoll_x86_644869269367588546881.so: /tmp/liborg_apache_hbase_thirdparty_netty_transport_

native_epoll_x86_644869269367588546881.so: failed to map segment from shared object: Operation not permitted

                at java.lang.ClassLoader$NativeLibrary.load(Native Method)

                at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)

                at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)

                at java.lang.Runtime.load0(Runtime.java:809)

                at java.lang.System.load(System.java:1086)

                at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)

                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)

                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

                at java.lang.reflect.Method.invoke(Method.java:498)

                at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)

                at java.security.AccessController.doPrivileged(Native Method)

                at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)

                at org.apache.hbase.thirdparty.io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)

                ... 27 more

No sure what setting can be used to make yarn not use netty.

Super Mentor

@Lian Jiang

In your blueprint you have added "APP_TIMELINE_SERVER", So can you please check if you have also added the "TIMELINE_READER" as following?

        {
          "name" : "APP_TIMELINE_SERVER"
        },
        {
          "name" : "TIMELINE_READER"
        },

.

New Contributor

I also hit the same issue. But modify hbase_java_io_tmpdir does not work for me...

Any other suggestion?

Super Mentor

@Lian Jiang

Also please find the below properties values from your Blueprint file and change the values to a path which has proper permission:

Specially inside "yarn-hbase-env"

hbase_java_io_tmpdir
yarn_hbase_java_io_tmpdir

.

View solution in original post

Contributor

Thanks @Jay Kumar SenSharma

This setting worked:

"yarn-hbase-env": {
        "properties": {
          "hbase_java_io_tmpdir": "/u01/tmp"
        }
      }

These setting made TS reader v2 start. Progress!

However, after some time, it stopped again due to the issue described in this post. The reason is that the yarn hbase regional server cannot start (though the yarn hbase master starts). I guess it is still related to the fact that regional server is using /tmp.

In ambari UI, I cannot find a place to set yarn_hbase_java_io_tmpdir. Any idea?

Super Mentor

@Lian Jiang

Good to know that with the help of "hbase_java_io_tmpdir" your issue partially solved.

Regarding the "yarn_hbase_java_io_tmpdir" property, you can find it inside the "Advanced yarn-hbase-env" as following:

Ambari UI --> Yarn --> Configs --> Advanced --> "Advanced yarn-hbase-env" --> "hbase-env template"

      export HBASE_OPTS="$HBASE_OPTS -XX:+UseConcMarkSweepGC -XX:ErrorFile=$HBASE_LOG_DIR/hs_err_pid%p.log -Djava.io.tmpdir={{yarn_hbase_java_io_tmpdir}}"

.

So just replace the value of {{yarn_hbase_java_io_tmpdir}} with your desired value. The Default value of the "yarn_hbase_java_io_tmpdir" is calculated as following:

# grep 'yarn_hbase_java_io_tmpdir' /var/lib/ambari-agent/cache/stacks/HDP/3.0/services/YARN/package/scripts/params_linux.py
yarn_hbase_java_io_tmpdir = default("/configurations/yarn-hbase-env/hbase_java_io_tmpdir", "/tmp")

Contributor

It turned out hbase_java_io_tmpdir is enough. The hbase regional server not starting issue is because our security setting disabled the yarn-ats user whose uid is out of a range. After creating this user in the desired range, ts reader v2 worked.

Super Mentor

@Lian Jiang

Correct we do not need to worry about the "yarn_hbase_java_io_tmpdir" property as it is basically being controlled by the property "hbase_java_io_tmpdir" hence if we do not set the hbase_java_io_tmpdir property on our own then the default value for both the property will be "/tmp"

yarn_hbase_java_io_tmpdir = default("/configurations/yarn-hbase-env/hbase_java_io_tmpdir", "/tmp")

As your reader seems to be working fine now. hence it will be great if you ca mark this thread as answered so that it will be useful for other HCC users to know the details about these properties and can quickly browse the answers.

New Contributor

I also hit the same issue. But modify hbase_java_io_tmpdir does not work for me...

Any other suggestion?