Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Apache Atlas not capturing metadata for Hive?

Apache Atlas not capturing metadata for Hive?

Rising Star

Hello,

I have downloaded Atlas-Ranger technical preview machines from below link and it's running fine.

Link: http://hortonworks.com/solutions/security-and-governance/

I followed the Atlas-Ranger Tutorial as well from the link:

1) http://hortonworks.com/hadoop-tutorial/tag-based-policies-atlas-ranger/(For Tag-based policy).

2) http://hortonworks.com/hadoop-tutorial/cross-component-lineage-apache-atlas/(For Cross Component lineage).

both the tutorials are worked fine on virtual machine.

The problem is,whenerver I create any hive tables or performing some hive operations on existing tables those are not getting reflecting into atlas.According to my knowledge,whenever we create hive table ,atlas usually captures metadata for it.

But this is not happening on my machine/VM.

--------------------------------------------------------------------------------------------------------------------------------------------------------------

One more thing,I am getting two alerts inside ambari regarding Hive:

1) Alert(Hive Metaserver Process)

Metastore on localhost.localdomain failed (Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/sbin/:/usr/hdp/current/hive-metastore/bin'"'"' ; export HIVE_CONF_DIR='"'"'/usr/hdp/current/hive-metastore/conf/conf.server'"'"' ; hive --hiveconf hive.metastore.uris=thrift://localhost.localdomain:9083 --hiveconf hive.metastore.client.connect.retry.delay=1 --hiveconf hive.metastore.failure.retries=1 --hiveconf hive.metastore.connect.retries=1 --hiveconf hive.metastore.client.socket.timeout=14 --hiveconf hive.execution.engine=mr -e '"'"'show databases;'"'"''' was killed due timeout after 30 seconds)

-----------------------------------------------------------------------------------------------------------------------------------------------------

2) Alert(HiveServer2 Process)

Connection failed on host localhost.localdomain:10000 (Execution of 'ambari-sudo.sh su ambari-qa -l -s /bin/bash -c 'export PATH='"'"'/usr/sbin:/sbin:/usr/lib/ambari-server/*:/sbin:/usr/sbin:/bin:/usr/bin:/var/lib/ambari-agent:/bin/:/usr/bin/:/usr/lib/hive/bin/:/usr/sbin/'"'"' ; ! beeline -u '"'"'jdbc:hive2://localhost.localdomain:10000/;transportMode=binary'"'"' -e '"'"''"'"' 2>&1| awk '"'"'{print}'"'"'|grep -i -e '"'"'Connection refused'"'"' -e '"'"'Invalid URL'"'"''' was killed due timeout after 30 seconds)

Is this is the problem for not getting metadata inside atlas.

Thanks in advance,

Please share you thoughts with me ,It will be really helpful to me.

4 REPLIES 4

Re: Apache Atlas not capturing metadata for Hive?

Hi @Manoj Dhake

Are you using Hive CLI or Beeline (or something else)?

Re: Apache Atlas not capturing metadata for Hive?

There are mostly two reason.

1)May be this is the issue because there is not enough resources in your system i.e memory and CPU.

2)Second reason for this issue is that the Hive Metastore cannot access the Hive database.

to check is the access login to mysql

and run command

$show grants for hive;

if nothing returns then you need to run the grants command

grant all on *.* to "hive_user"@"%" identified by "hive_user_password";

flush priviliges;

And then check again grants

show grants for hive;

If this is resolved your issue please let us know it will help to others.

Re: Apache Atlas not capturing metadata for Hive?

@Manoj Dhake I guess as per your comment, you are able to create hive tables and modify them using beeline or hive cli but metadata is not propagated to atlas.

Assuming that hiveserver2, metastore and atlas are in running state. Can you paste the hiveserver2.log content when you create a table?

Re: Apache Atlas not capturing metadata for Hive?

New Contributor

How can I update HiveServer2 Process check timeout values? In my scenes there were in test environment, So I don't care it check for long time. 30 seconds seems not enough.