Member since
04-20-2016
35
Posts
6
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1639 | 09-14-2016 08:05 PM |
04-03-2017
06:33 PM
Unfortunately there is no timestamp or any other good candidate for a split-by, which is why I was trying to generate row numbers on the fly
... View more
04-03-2017
03:45 PM
I have a general question about how sqoop works. When you run an import, does sqoop take a snapshot of the table at that time and import that result? Or does it keep an active window of the data? If the table is rapidly changing, would that cause issues with the data import? My import query looks like this: SELECT * FROM (SELECT *, ROW_NUMBER() OVER (ORDER BY COL1) R FROM TABLE) A WHERE \$CONDITIONS Upon doing some QA on the imported data to HDFS, I am seeing the overall row count is close to the source table, but the data is inconsistent. For example, there are plenty of missing rows and duplicated rows. This is a large table sitting at ~560 million rows. I am calculating the row number in order to get a perfect split-by, but I think that may be where the problem is coming from. It takes a few minutes to calculate the row numbers and my hunch is that while that's happening, the table is being updated at the source and it is throwing the row numbers and subsequent import off. Any thoughts?
... View more
Labels:
- Labels:
-
Apache Sqoop
03-27-2017
07:12 PM
When i run this i get: oozie admin -oozie http://host:11000/oozie -shareliblist [Available ShareLib] hive distcp mapreduce-streaming spark oozie hcatalog hive2 sqoop pig spark_orig
... View more
03-27-2017
06:23 PM
@tli 2017-03-27 12:28:39,482 WARN HiveActionExecutor:523 - SERVER[] USER[hdfs] GROUP[-] TOKEN[] APP[test_workflow] JOB[0000049-170319180732598-oozie-oozi-W] ACTION[0000049-170319180732598-oozie-oozi-W@test_query] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.HiveMain], exception invoking main(), java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.HiveMain not found
2017-03-27 12:28:39,483 WARN HiveActionExecutor:523 - SERVER[] USER[hdfs] GROUP[-] TOKEN[] APP[test_workflow] JOB[0000049-170319180732598-oozie-oozi-W] ACTION[0000049-170319180732598-oozie-oozi-W@test_query] Launcher exception: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.HiveMain not found
java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.HiveMain not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2214)
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:240)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:453)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.HiveMain not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2120)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2212)
... 9 more
... View more
03-27-2017
05:50 PM
Trying to run a basic oozie workflow that completes a hive action and i keep getting this error: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.HiveMain not found Here is the XML workflow: <workflow-app xmlns = "uri:oozie:workflow:0.5" name = "test_workflow"> <start to = "test_query" />
<!--Step 1 -->
<action name = "test_query"> <hive xmlns = "uri:oozie:hive-action:0.5"> <job-tracker>jobtracker</job-tracker> <name-node>nameNode</name-node> <configuration>
<property> <name>oozie.use.system.libpath</name>
<value>true</value>
</property> <property>
<name>oozie.libpath</name>
<value>nameNode/user/oozie/share/lib/lib_20161215134654</value>
</property>
</configuration> <script>hdfspath/hive.hql</script> </hive>
<ok to = "end" /> <error to = "kill_job" />
</action>
<kill name = "kill_job">
<message>Job Failed</message>
</kill>
<end name = "end" /> </workflow-app>
... View more
Labels:
- Labels:
-
Apache Oozie
09-14-2016
08:05 PM
2 Kudos
Found the culprit, it's the CSVSerde, it casts all fields as strings.
... View more
09-14-2016
07:50 PM
I just tried following the method shown here: https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_dataintegration/content/moving_data_from_hdfs_to_hive_external_table_method.html I created the ORC table and it still stores everything as a string
... View more
09-14-2016
07:16 PM
1 Kudo
I am creating an external table with some fields that are numeric decimals. I have tried declaring them as floats, doubles, and decimal formats, but hive is casting them as strings regardless. Here is a sample of the query: create external table sapbw.copa_ods(currency_type string, cost_for_fi decimal,
gross_margin_fi decimal, sales_for_fi decimal)
ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
stored as textfile
location '/data_LZ/BW/COPA_ODS'; Why is this happening?
... View more
- Tags:
- Data Processing
- Hive
Labels:
- Labels:
-
Apache Hive
06-16-2016
07:22 PM
1 Kudo
So I found that if i force a query using --query it will work, but if I let sqoop generate the query it does not. Example --query "select * from \"BI0/TCUSTOMER\"" will work, but referring to the table in the same way by using the --table \"BI0/TCUSTOMER\" argument does not work. So the problem must be when sqoop generates the SQL query on its own, it is not syntactically correct. If there was a way to see the query it was generating that would be very helpful to troubleshoot it, but as far as I know there is not.
... View more
06-13-2016
06:21 PM
Any updates on this?
... View more
06-09-2016
01:18 PM
I tried that and got this error: ERROR manager.SqlManager: Error executing statement: com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [257]: sql syntax error: incorrect syntax near "AS": line 1 col 19 (at pos 19)
com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [257]: sql syntax error: incorrect syntax near "AS": line 1 col 19 (at pos 19)
... View more
06-08-2016
09:22 PM
Got the same error: 16/06/08 16:20:49 ERROR tool.ImportTool: Encountered IOException running import job: java.io.FileNotFoundException: /tmp/sqoop-hdfs/compile/2fb644a9751cabd5c187540154e2e696/SAPR3."/BI0/TCUSTOMER".jar (No such file or directory)
... View more
06-08-2016
09:13 PM
Yes I tried the above. That somewhat fixed my problem, it seems to accept the syntax now, but I believe it is now literally interpreting the quotes as part of the table name, so when it goes to SAP to find that table, it can't find it because the quotes aren't actually there in the SAP table name... 16/06/08 15:10:31 ERROR tool.ImportTool: Encountered IOException running import job: java.io.FileNotFoundException: /tmp/sqoop-hdfs/compile/d26f755016cd4c734711de4a2550ca46/SCHEMA."/BI0/TCUSTOMER".jar (No such file or directory)
... View more
06-07-2016
09:32 PM
That still doesn't work. Here's my script: sqoop --options-file options.txt \ --table "SCHEMA\".\"/BI0/TCUSTOMER" --hive-import --hive-overwrite --split-by CUSTOMER --hive-database DATABASE Produces this error: com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [257]: sql syntax error: incorrect syntax near "/": line 1 col 26 (at pos 26)
... View more
06-07-2016
08:59 PM
Yes, doing that gave the same error.
... View more
06-07-2016
08:45 PM
Is there a way to make sqoop accept / characters in table and field names? I have tried the following: leaving the name as-is: --table /BI0/TCUSTOMER wrapping in quotes: --table "/BI0/TCUSTOMER" escaping each /: --table "\/BI0\/TCUSTOMER" It produces this error each time for all three options above: com.sap.db.jdbc.exceptions.JDBCDriverException: SAP DBTech JDBC: [257]: sql syntax error: incorrect syntax near "\": line 1 col 17 (at pos 17)
... View more
Labels:
- Labels:
-
Apache Sqoop