Created on 08-07-2019 09:02 PM - edited 08-17-2019 04:18 PM
Using sqoop on hdp 3.1 to import Oracle DB data into HDFS as parquet and expecting it to appear in atlas as an entity (I assume it should be the default sqoop_process
type). However, after running the sqoop job, nothing appears in atlas. Sqoop job looks like...
sqoop import -Dmapreduce.map.memory.mb=3144 -Dmapreduce.map.java.opts=-Xmx1048m -Dyarn.app.mapreduce.am.log.level=DEBUG -Dmapreduce.map.log.level=DEBUG -Dmapreduce.reduce.log.level=DEBUG -Dmapred.job.name="Ora import table $tablename" -Djava.security.egd=file:///dev/urandom -Djava.security.egd=file:///dev/urandom -Doraoop.timestamp.string=false -Dmapreduce.map.max.attempts=10 $oracle_cnxn_str --as-parquetfile --target-dir $importdir -query "select a.*, current_date as etl_date from $tablename a where 1=1 AND \$CONDITIONS" --split-by $splitby --where "1=1" --num-mappers 12 --delete-target-dir
Looking in the hdp ambari interface for atlas, I can see that the sqoop plugin for atlas is configured(did not initially do any manual tweaking to it after installing atlas according to the docs). Adding the property
based on another Hortonworks post and attempting the sqoop job again still shows nothing in the atlas UI. Anyone with more experience with hdp and atlas have any debugging suggestions or fixes?