Support Questions
Find answers, ask questions, and share your expertise

How to config Sqoop Hook for Atlas?

Explorer

@Ayub Pathan

After configured the Sqoop hook, I import table from MySQL into Hive by using Sqoop. But I check the Atlas UI, click the sqoop_process TYPE, I can't check any information. The Sqoop hook doesn't work.

The doc http://atlas.apache.org/Bridge-Sqoop.html is not clear. And it doesn't show the specific method of configuring sqoop hook.

So I did some research and configured it as shown following:

Step 1: Set the <sqoop-conf>/sqoop-site.xml

<property>  
<name>sqoop.job.data.publish.class</name>    
<value>org.apache.atlas.sqoop.hook.SqoopHook</value>
</property>

Step 2: Copy the <atlas-conf>/atlas-application.properties to <sqoop-conf>

Step 3: Link <atlas-home>/hook/sqoop/*.jar in sqoop lib. Because I don't know how to link them, I copy these *.jar to <sqoop-lib>.

I am wondering that, is these some way to link these *.jar other than copy them? set a environment variable to point *.jar ?

After configured the Sqoop hook, I import table from MySQL into Hive by using Sqoop.

sqoop import -connect
jdbc:mysql://master:3306/hive -username root -password admin -table
TBLS -hive-import -hive-table sqoophook1 

But I check the Atlas UI, click the sqoop_process TYPE, I can't check any information.

Why the Sqoop hook doesn't work? and my configuration is wrong ?

My Hadoop version is 2.6.0, sqoop version is 1.4.6, hive version is 1.2.1, atlas version is 0.7.

3 REPLIES 3

Re: How to config Sqoop Hook for Atlas?

Re: How to config Sqoop Hook for Atlas?

Explorer

Thanks @Divakar Annapureddy

This link doesn't help me.

Because I don't use HDP, and I install atlas, hive, sqoop respectively.

I don't think the sqoop lack some jar, but some configuration steps of Sqoop hook are wrong.

The tutorial of http://atlas.apache.org/Bridge-Sqoop.html is not clear, please tell me the specific method to configure sqoop hook. Thank you

Re: How to config Sqoop Hook for Atlas?

Contributor

@Ethan Hsieh

Could you also confirm if the sqoop-site.xml has the rest address for atlas server configured ?

Sample configuration is available here