Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

How to config Sqoop Hook for Atlas?

Highlighted

How to config Sqoop Hook for Atlas?

Explorer

@Ayub Pathan

After configured the Sqoop hook, I import table from MySQL into Hive by using Sqoop. But I check the Atlas UI, click the sqoop_process TYPE, I can't check any information. The Sqoop hook doesn't work.

The doc http://atlas.apache.org/Bridge-Sqoop.html is not clear. And it doesn't show the specific method of configuring sqoop hook.

So I did some research and configured it as shown following:

Step 1: Set the <sqoop-conf>/sqoop-site.xml

<property>  
<name>sqoop.job.data.publish.class</name>    
<value>org.apache.atlas.sqoop.hook.SqoopHook</value>
</property>

Step 2: Copy the <atlas-conf>/atlas-application.properties to <sqoop-conf>

Step 3: Link <atlas-home>/hook/sqoop/*.jar in sqoop lib. Because I don't know how to link them, I copy these *.jar to <sqoop-lib>.

I am wondering that, is these some way to link these *.jar other than copy them? set a environment variable to point *.jar ?

After configured the Sqoop hook, I import table from MySQL into Hive by using Sqoop.

sqoop import -connect
jdbc:mysql://master:3306/hive -username root -password admin -table
TBLS -hive-import -hive-table sqoophook1 

But I check the Atlas UI, click the sqoop_process TYPE, I can't check any information.

Why the Sqoop hook doesn't work? and my configuration is wrong ?

My Hadoop version is 2.6.0, sqoop version is 1.4.6, hive version is 1.2.1, atlas version is 0.7.

3 REPLIES 3

Re: How to config Sqoop Hook for Atlas?

Re: How to config Sqoop Hook for Atlas?

Explorer

Thanks @Divakar Annapureddy

This link doesn't help me.

Because I don't use HDP, and I install atlas, hive, sqoop respectively.

I don't think the sqoop lack some jar, but some configuration steps of Sqoop hook are wrong.

The tutorial of http://atlas.apache.org/Bridge-Sqoop.html is not clear, please tell me the specific method to configure sqoop hook. Thank you

Re: How to config Sqoop Hook for Atlas?

Contributor

@Ethan Hsieh

Could you also confirm if the sqoop-site.xml has the rest address for atlas server configured ?

Sample configuration is available here

Don't have an account?
Coming from Hortonworks? Activate your account here