Member since
04-20-2016
35
Posts
6
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
2792 | 09-14-2016 08:05 PM |
03-28-2017
07:27 AM
@Josh Persinger Sometimes it happens when Oozie is having trouble referencing the proper oozie-sharelib-hive-<HDP Version>.jar
You should try to regenerate the oozie sharelib 1. Get a listing of hive sharelib content:
# oozie admin -oozie http://${OOZIE_HOST}:11000/oozie -shareliblist hive* > /tmp/hive_shareliblist_OLD.txt 2>&1
2. Now we will recreate the sharelib # /usr/hdp/<HDP Version>/oozie/bin/oozie-setup.sh sharelib create -fs hdfs://${NAMENODE}
3. Again list the hive sharelib contents now. # oozie admin -oozie http://${OOZIE_HOST}:11000/oozie -shareliblist hive* > /tmp/hive_shareliblist_NEW.txt 2>&1
4. For double verification list all hive sharelib directories in HDFS # hdfs dfs -ls -R /user/oozie/share/lib/*/hive/* > /tmp/hive_libs_on_hdfs.txt 2>&1
5. Edit the "workflow.xml" and remove the line: <property>
<name>oozie.libpath</name>
<value>${nameNodeHost:8020}/user/oozie/share/lib/lib_20170116233431</value>
</property>
6. Modify the "job.properties" file and set the following property to true. The restart the oozie/job oozie.use.system.libpath=true . .
... View more
09-14-2016
08:05 PM
2 Kudos
Found the culprit, it's the CSVSerde, it casts all fields as strings.
... View more
07-08-2016
11:38 AM
1 Kudo
I can confirm what
@Josh Persinger is saying. The only way to get tables with forward slashes ('/') (and actually colons too) in the tablename from SAP into hadoop hdfs/hive is by using the --query statement.
Some other things I found out when importing from SAP HANA:
a table name can be something like 'MSG\TABLENAME' or even worse: '[SCHEMA]::database.[TABLENAME]'. Just make sure you put the complete tablename between escaped double quotes:
eg:
\"/SOMETING/TABLENAME\" or \"[SCHEMA]::database.[TABLENAME]\"
we needed to add there where clause '\$CONDITIONS' even though we did a select * without any filters.
when limiting the result with a where clause the values have to be between single quotes:
eg. WHERE DDLANGUAGE='E'
SAP columns can contain empty values called SPACE (not the same as NULL) (shown as a '?' in the webIDE). If you want to exclude them use the where clause <>'' (just two singlequotes following each other):
WHERE DDLANGUAGE<>''
When making your command more readible I had to keep one extra parameter after the --query parameter. When I moved the --hive-import to the next line the command would fail (I think due to the ending quotes of the query.
The result should look something like this:
sqoop import --connect "jdbc:sap://[SERVER]:30015/?currentschema=[SCHEMA]" \ --username [USERNAME] \ --password-file file:///[PATH]/.pwd_file \ --driver com.sap.db.jdbc.Driver \ --query "select * from \"[/TABLENAME/WITH/FORWARDSLASHES]\" WHERE DDLANGUAGE='E' and [COLUMNNAME]<>'' and \$CONDITIONS" --hive-import \ --hive-database [HIVEDATABASE] \ --hive-table [TABLENAME] \ --hive-delims-replacement ' ' \ --fields-terminated-by '|' \ --lines-terminated-by '\n' \ --hive-overwrite
--num-mappers 1
... View more