I have a general question about how sqoop works. When you run an import, does sqoop take a snapshot of the table at that time and import that result? Or does it keep an active window of the data? If the table is rapidly changing, would that cause issues with the data import? My import query looks like this: SELECT * FROM (SELECT *, ROW_NUMBER() OVER (ORDER BY COL1) R FROM TABLE) A WHERE \$CONDITIONS Upon doing some QA on the imported data to HDFS, I am seeing the overall row count is close to the source table, but the data is inconsistent. For example, there are plenty of missing rows and duplicated rows. This is a large table sitting at ~560 million rows. I am calculating the row number in order to get a perfect split-by, but I think that may be where the problem is coming from. It takes a few minutes to calculate the row numbers and my hunch is that while that's happening, the table is being updated at the source and it is throwing the row numbers and subsequent import off. Any thoughts? ... View more

Trying to run a basic oozie workflow that completes a hive action and i keep getting this error: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.HiveMain not found Here is the XML workflow: <workflow-app xmlns = "uri:oozie:workflow:0.5" name = "test_workflow"> <start to = "test_query" /> <!--Step 1 --> <action name = "test_query"> <hive xmlns = "uri:oozie:hive-action:0.5"> <job-tracker>jobtracker</job-tracker> <name-node>nameNode</name-node> <configuration> <property> <name>oozie.use.system.libpath</name> <value>true</value> </property> <property> <name>oozie.libpath</name> <value>nameNode/user/oozie/share/lib/lib_20161215134654</value> </property> </configuration> <script>hdfspath/hive.hql</script> </hive> <ok to = "end" /> <error to = "kill_job" /> </action> <kill name = "kill_job"> <message>Job Failed</message> </kill> <end name = "end" /> </workflow-app> ... View more

I have hive tables with various special characters in the field names, such as - / # etc... Is there a way to properly escape them when loading into a pig script? I'm using HCatloader to load from the tables example: e-mail_address (it thinks that e is a field and mail_address is a field) ... View more

Our cluster is set to produce .deflate files, i have a pig script that produces a data set to an HDFS folder that i have an external table located on. Can the hive table read .deflate files? The .deflate file is in the directory, but when i query the table i get no results. ... View more

I have a sqoop import job in which I tell it to use '\\N' for both the --null-string and --null-non-string values. It is importing from an SAP HANA database directly to hive. I noticed that for a particular table I import, there are two columns of type NUMC in SAP that are being converted to string automatically, but instead of having null values it is defaulting to a value of 00000000. Why would sqoop be ignoring the --null-non-string argument? Or am I doing it wrong? ... View more

I have a pig script stored in HDFS that i need to run from the local filesystem. I'm trying this but it doesn't work: pig -exectype tez -file /path/to/script/script.pig It returns an error saying the script does not exist. Since I am executing in Tez mode, i would assume that it looks in HDFS for the script and not the local filesystem. ... View more

Whenever I run a sqoop job with more than one mapper, it always creates files that have no data in them. I understand that in the absence of a uniformly distributed field to split by, the data will be skewed, but that does not explain why there would be files with literally nothing in them... Here's an example: sqoop --options-file opt.txt --table table --hive-import --hive-overwrite --hive-database db --num-mappers 8 --split-by FISCPER This produced files that look like this: part-m-00000 Size: 102.2GB part-m-00001 Size: 0.1kB part-m-00002 Size: 0.1kB part-m-00003 Size: 0.1kB part-m-00004 Size: 0.1kB part-m-00005 Size: 0.1kB part-m-00006 Size: 0.1kB part-m-00007 Size: 121.0GB Each of the 0.1kB files is empty...all of the data is contained in the two large files. ... View more

Many times when I run a sqoop command it ignores the number of mappers i tell it to use. Common examples are: --num-mappers 2 and it uses 4 instead --num-mappers 8 and it uses 10 instead Also, sqoop sometimes creates files that have no data in them. Any ideas about these issues? ... View more

So i'm having trouble loading the files that sqoop creates when you import from a DB into a Hive table. I sqooped the data here: /user/hive/sqoop_hdfs_hive_test/tmpdir I then run this query: LOAD DATA INPATH '/user/hive/sqoop_hdfs_hive_test/tmpdir' overwrite into table temp.temp It executes with no errors, but doesn't actually move the data to the table. I have created tables from .csv files before, but not from the files that sqoop creates (they look like this part-m-00000) ... View more

This is my script: sqoop --options-file sqoop-opt.txt --table HDFS_MATERIAL_MASTER --hive-import --hive-overwrite --hive-database SAP This is the error I get: Error: java.io.IOException: SQLException in nextKeyValue at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:277) at org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.nextKeyValue(MapTask.java:556) at org.apache.hadoop.mapreduce.task.MapContextImpl.nextKeyValue(MapContextImpl.java:80) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.nextKeyValue(WrappedMapper.java:91) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145) at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [257]: sql syntax error: incorrect syntax near ",": line 1 col 479 (at pos 479) at com.sap.db.jdbc.exceptions.SQLExceptionSapDB.createException(SQLExceptionSapDB.java:345) at com.sap.db.jdbc.exceptions.SQLExceptionSapDB.generateDatabaseException(SQLExceptionSapDB.java:185) at com.sap.db.jdbc.packet.ReplyPacket.buildExceptionChain(ReplyPacket.java:100) at com.sap.db.jdbc.ConnectionSapDB.execute(ConnectionSapDB.java:1130) at com.sap.db.jdbc.CallableStatementSapDB.sendCommand(CallableStatementSapDB.java:1810) at com.sap.db.jdbc.StatementSapDB.sendSQL(StatementSapDB.java:981) at com.sap.db.jdbc.CallableStatementSapDB.doParse(CallableStatementSapDB.java:183) at com.sap.db.jdbc.CallableStatementSapDB.constructor(CallableStatementSapDB.java:142) at com.sap.db.jdbc.CallableStatementSapDB.<init>(CallableStatementSapDB.java:87) at com.sap.db.jdbc.CallableStatementSapDBFinalize.<init>(CallableStatementSapDBFinalize.java:31) at com.sap.db.jdbc.ConnectionSapDB.prepareStatement(ConnectionSapDB.java:1379) at com.sap.db.jdbc.trace.Connection.prepareStatement(Connection.java:347) at org.apache.sqoop.mapreduce.db.DBRecordReader.executeQuery(DBRecordReader.java:101) at org.apache.sqoop.mapreduce.db.DBRecordReader.nextKeyValue(DBRecordReader.java:235) ... 12 more ... View more

My pig script is throwing ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1066: Unable to open iterator for alias E This is where i'm defining the alias E: A = JOIN CONTACT_ACTIVITY BY Sold_To, KNA1_SAMPLE BY KNA1_Sold_To; B = JOIN A BY KNA1_Sold_To, KNVV_SAMPLE BY KNVV_Sold_To; C = JOIN B BY KNVV_District_Number, DISTRICT BY District_Number; D = JOIN C BY KNA1_Industry_Code, INDUSTRY BY Industry_Code; E = JOIN D BY KNA1_CoT_Number, CLASS_OF_TRADE BY CoT_Number; dump E; Any help would be much appreciated! I'm running this on TEZ ... View more