Hive beeline throws ClassNotFoundException after installing atlas

[#1]

Hello Team,

We have a HDP 2.6 cluster with HIVE in HA using zookeeper discovery enabled. Before installing atlas on cluster All comands related to hive through beeline was working.

Post installing atlas through ambari when i run simple show tables command it throws below error.

[ajay@vijayhdf-1 ~]$ beeline Beeline version 1.2.1000.2.6.3.0-235 by Apache Hive beeline> !connect jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Connecting to jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Enter username for jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ajay Enter password for jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ******** Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235) Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://vijayblue-1.novalocal:2181,vi> show tables; Error: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(org.apache.atlas.hive.hook.HiveHook) (state=08S01,code=12)

Kindly help me to fix the issue.

- Vijay Mishra

[#2]

@Vijay Mishra,

Did you enable Atlas Hook under hive. If not, then please enable Hive hook and retry.

Go to Ambari -> Hive -> Configs -> Advanced -> Advanced hive-env

Please refer screenshot

Thanks,

Aditya

[#3]

atlas-hook-hive.png

@ Aditya

Its enabled, Attached screenshot for the same.

- Vijay Mishra

[#4]

@Vijay Mishra,

Can you please run the command and attach the output here

[root@sss ~]# ll /usr/bin/beeline
lrwxrwxrwx. 1 root root 40 Dec  6 12:29 /usr/bin/beeline -> /usr/hdp/current/hive-client/bin/beeline

[root@sss ~]# cat /usr/hdp/current/hive-client/bin/beeline (cat the content of your symlink)

Thanks,

Aditya

[#5]

@Aditya,

[root@vijayhdf-2 hive]# ll /usr/bin/beeline lrwxrwxrwx 1 root root 40 Jan 30 08:47 /usr/bin/beeline -> /usr/hdp/current/hive-client/bin/beeline [root@vijayhdf-2 hive]# cat /usr/hdp/current/hive-client/bin/beeline #!/bin/bash if [ -d "/usr/hdp/2.6.3.0-235/atlas/hook/hive" ]; then if [ -z "${HADOOP_CLASSPATH}" ]; then export HADOOP_CLASSPATH=/usr/hdp/2.6.3.0-235/atlas/hook/hive/* else export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:/usr/hdp/2.6.3.0-235/atlas/hook/hive/* fi fi BIGTOP_DEFAULTS_DIR=${BIGTOP_DEFAULTS_DIR-/etc/default} [ -n "${BIGTOP_DEFAULTS_DIR}" -a -r ${BIGTOP_DEFAULTS_DIR}/hbase ] && . ${BIGTOP_DEFAULTS_DIR}/hbase export HIVE_HOME=${HIVE_HOME:-/usr/hdp/2.6.3.0-235/hive} export HADOOP_HOME=${HADOOP_HOME:-/usr/hdp/2.6.3.0-235/hadoop} export ATLAS_HOME=${ATLAS_HOME:-/usr/hdp/2.6.3.0-235/atlas} HCATALOG_JAR_PATH=/usr/hdp/2.6.3.0-235/hive-hcatalog/share/hcatalog/hive-hcatalog-core-1.2.1000.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/hive-hcatalog/share/hcatalog/hive-hcatalog-server-extensions-1.2.1000.2.6.3.0-235.jar:/usr/hdp/2.6.3.0-235/hive-hcatalog/share/webhcat/java-client/hive-webhcat-java-client-1.2.1000.2.6.3.0-235.jar if [ -z "${HADOOP_CLASSPATH}" ]; then export HADOOP_CLASSPATH=${HCATALOG_JAR_PATH} else export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:${HCATALOG_JAR_PATH} fi exec "${HIVE_HOME}/bin/beeline.distro" "$@" [root@vijayhdf-2 hive]#

[#6]

@Aditya,

For verify purpose i just disable the atlas hook from hive and ran the beeline and its working.

[ajay@vijayhdf-1 ~]$ beeline Beeline version 1.2.1000.2.6.3.0-235 by Apache Hive beeline> !connect jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Connecting to jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Enter username for jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ajay Enter password for jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ******** Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235) Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://vijayblue-1.novalocal:2181,vi> show tables; +-----------+--+ | tab_name | +-----------+--+ +-----------+--+ No rows selected (2.28 seconds) 0: jdbc:hive2://vijayblue-1.novalocal:2181,vi>

Post which i reenable atlas hook into hive and ran the beeline again and got the same error.

[ajay@vijayhdf-1 ~]$ beeline Beeline version 1.2.1000.2.6.3.0-235 by Apache Hive beeline> !connect jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Connecting to jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Enter username for jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ajay Enter password for jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ******** Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235) Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://vijayblue-1.novalocal:2181,vi> show tables; Error: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(org.apache.atlas.hive.hook.HiveHook) (state=08S01,code=12) 0: jdbc:hive2://vijayblue-1.novalocal:2181,vi>

[#7]

@Vijay Mishra,

Do you have atlas client installed on the node from where you are running beeline. If not, can you please run the command and try again.

curl -k -u {ambari-username}:{ambari-password} -H "X-Requested-By:ambari" -i -X PUT -d '{"HostRoles": {"state": "INSTALLED"}}' http://{ambari-host}:{ambari-port}/api/v1/clusters/{clustername}/hosts/{hostname}/host_components/AT...

Replace ambari-username, ambari-password, ambari-host, ambari-port, clustername with their respective values in the url.

Replace {hostname} with the hostname where you are running beeline client.

Thanks,

Aditya

[#8]

@Aditya,

Atlas client installed on all 4 nodes of cluster. Still i ran the command provided by you. Below snap for the same.

[root@vijayhdf-1 hive]# curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X PUT -d '{"HostRoles": {"state": "INSTALLED"}}' http://vijayhdf-1.novalocal:8080/api/v1/clusters/hdpdev/hosts/vijayhdf-1.novalocal/host_components/A... HTTP/1.1 202 Accepted X-Frame-Options: DENY X-XSS-Protection: 1; mode=block X-Content-Type-Options: nosniff Cache-Control: no-store Pragma: no-cache Set-Cookie: AMBARISESSIONID=n8czyjmtzvhzevwgenc583h5;Path=/;HttpOnly Expires: Thu, 01 Jan 1970 00:00:00 GMT User: admin Content-Type: text/plain Vary: Accept-Encoding, User-Agent Content-Length: 149 { "href" : "http://vijayhdf-1.novalocal:8080/api/v1/clusters/hdpdev/requests/122", "Requests" : { "id" : 122, "status" : "Accepted" } }[root@vijayhdf-1 hive]#

I restarted Hive and atlas and ran the beeline but it throws error again.

[ajay@vijayhdf-1 ~]$ hostname -f vijayhdf-1.novalocal [ajay@vijayhdf-1 ~]$ beeline Beeline version 1.2.1000.2.6.3.0-235 by Apache Hive beeline> !connect jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Connecting to jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Enter username for jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ajay Enter password for jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ******** Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235) Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://vijayblue-1.novalocal:2181,vi> show tables; Error: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(org.apache.atlas.hive.hook.HiveHook) (state=08S01,code=12) 0: jdbc:hive2://vijayblue-1.novalocal:2181,vi>

- Vijay Mishra

[#9]

@Aditya

I explored few more thing.

Updated below property as mentioned below and restarted hive and atlas.

From:

hive.exec.post.hooks-org.apache.hadoop.hive.ql.hooks.ATSHook,org.apache.atlas.hive.hook.HiveHook

To:

hive.exec.post.hooks-org.apache.hadoop.hive.ql.hooks.ATSHook

All commands in beeline started getting executed but in Atlas UI i am unable to see hive table.

[ajay@vijayhdf-1 ~]$ beeline Beeline version 1.2.1000.2.6.3.0-235 by Apache Hive beeline> !connect jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Connecting to jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2 Enter username for jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ajay Enter password for jdbc:hive2://vijayblue-1.novalocal:2181,vijayhdf-3.novalocal:2181,vijayhdf-2.novalocal:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2: ******** Connected to: Apache Hive (version 1.2.1000.2.6.3.0-235) Driver: Hive JDBC (version 1.2.1000.2.6.3.0-235) Transaction isolation: TRANSACTION_REPEATABLE_READ 0: jdbc:hive2://vijayblue-1.novalocal:2181,vi> show tables; +-----------+--+ | tab_name | +-----------+--+ | emp | +-----------+--+ 1 row selected (2.232 seconds) 0: jdbc:hive2://vijayblue-1.novalocal:2181,vi> select * from emp; +----------+-----------+-------------+------------------+--+ | emp.eid | emp.name | emp.salary | emp.destination | +----------+-----------+-------------+------------------+--+ +----------+-----------+-------------+------------------+--+ No rows selected (1.519 seconds)

0: jdbc:hive2://vijayblue-1.novalocal:2181,vi> LOAD DATA INPATH 'hdfs://hdpdev/user/ajay/sample.txt' INTO TABLE emp;

0: jdbc:hive2://vijayblue-1.novalocal:2181,vi> select * from emp; +----------+--------------+-------------+------------------+--+ | emp.eid | emp.name | emp.salary | emp.destination | +----------+--------------+-------------+------------------+--+ | 1201 | Gopal | 45000 | Technical | | 1202 | Manisha | 45000 | Proof | | 1203 | Masthanvali | 40000 | Technical | | 1204 | Kiran | 40000 | Hr | | 1205 | Kranthi | 30000 | Op | +----------+--------------+-------------+------------------+--+

But when i did basic search in atlas UI for hive_table it shows no result.

Kindly confirm what i did above is correct, if Yes then how can i fix Atlas not able to view hive tables?

- Vijay Mishra

[#10]

@Vijay Mishra,

This is same as disabling the atlas hive hook