Member since
08-11-2020
7
Posts
1
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1745 | 09-02-2020 05:47 AM |
04-24-2023
12:54 AM
Hi, Unfortunatly --query is not an Sqoop export argument... I'm still facing this issue
... View more
04-18-2023
02:00 AM
Hi, Do I need specific JDBC driver in Sqoop librairies to manage postgresl/postgis database ? tkx
... View more
04-17-2023
05:12 AM
Thanks for hint. I customed it has following to make it running but facing new issue... Here with right syntaxe : --map-column-java geom=String (need uppercase for first datatype) Looks better but facing this output error : Error: column « geom » has type geometry but expression type is character varying Hint: You should rewrite expression or make type transformation. Tried few queries but not working... 1) --map-column-java geom=Geometry \ Had output : ERROR orm.ClassWriter: No ResultSet method for Java type Geometry ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.IllegalArgumentException: No ResultSet method for Java type Geometry java.lang.IllegalArgumentException: No ResultSet method for Java type Geometry 2) --map-column-java geom=String,value=Geometry \ ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.IllegalArgumentException: No column by the name valuefound while importing data; expecting one of [Y_REEL, INCERTITUDE_Z, DATE_POSE, geom, COTE_FIL_EAU, INCERTITUDE_XY, ANTI_ODEUR, OBSERVATION_RES, USER_VALID, DATE_GEOLOC, PRESTATAIRE, WM_VERSION, USER_CRE, USER_MAJ, LTT_FEATURE_ID, id, WM_USERNAME, id_metier, WM_ROWID, MARQUE_MATERIEL, X_REEL, Z_REEL, SOURCE_MAJ, FORMAT_ORIGINE_XY, WM_CREATEWORKSPACEID, LTT_STATUS, DATE_CRE, DATE_MAJ, COMMUNE, FORMAT_ORIG_ZRAD, NUM_SERIE_MATERIEL, WM_CREATETIME, TYPE_AVALOIR, ENTREPRISE_RELEVE, WM_RETIRETIME, COTE_TN, QUALITE_GEOLOC_XY, WM_WORKSPACE, FORMAT_ORIG_ZTN, RESEAU, MAITRE_OUVRAGE, QUALITE_GEOLOC_Z, QUAL_ANNEE, niveau, WM_OPTYPE, LTT_CURRENT_REVSET_ID, ID1, COTE_RADIER, WM_RETIREWORKSPACEID, EXPLOITANT] java.lang.IllegalArgumentException: No column by the name valuefound while importing data; expecting one of [Y_REEL, INCERTITUDE_Z, DATE_POSE, geom, COTE_FIL_EAU, INCERTITUDE_XY, ANTI_ODEUR, OBSERVATION_RES, USER_VALID, DATE_GEOLOC, PRESTATAIRE, WM_VERSION, USER_CRE, USER_MAJ, LTT_FEATURE_ID, id, WM_USERNAME, id_metier, WM_ROWID, MARQUE_MATERIEL, X_REEL, Z_REEL, SOURCE_MAJ, FORMAT_ORIGINE_XY, WM_CREATEWORKSPACEID, LTT_STATUS, DATE_CRE, DATE_MAJ, COMMUNE, FORMAT_ORIG_ZRAD, NUM_SERIE_MATERIEL, WM_CREATETIME, TYPE_AVALOIR, ENTREPRISE_RELEVE, WM_RETIRETIME, COTE_TN, QUALITE_GEOLOC_XY, WM_WORKSPACE, FORMAT_ORIG_ZTN, RESEAU, MAITRE_OUVRAGE, QUALITE_GEOLOC_Z, QUAL_ANNEE, niveau, WM_OPTYPE, LTT_CURRENT_REVSET_ID, ID1, COTE_RADIER, WM_RETIREWORKSPACEID, EXPLOITANT]
... View more
04-17-2023
01:45 AM
Using Sqoop export on CDP cluster (hdfs), I'm trying to export data geometry to Postgresql table but I'm facing a Sqoop ERROR orm.ClassWriter: No Java type for SQL type 1111 for column 'geom'. The 'geom' column definition on PostGis is : geometry (Point,3945). Here is my Sqoop export query : ``` sqoop export --connect "jdbc:postgresql://$database_server/$database_source_name" \ --username $username --password $password_alias \ --table $database_table_name \ --export-dir "${DIR_OUT_HDFS}" \ --fields-terminated-by '|' --lines-terminated-by "\n" \ -- --schema $database_table_name 1>>${LOG_FILE} 2>>${LOG_FILE} ``` Here is Sqoop output error : ``` ERROR orm.ClassWriter: No Java type for SQL type 1111 for column geom ERROR orm.ClassWriter: No Java type for SQL type 1111 for column geom ERROR orm.ClassWriter: No Java type for SQL type 1111 for column geom ERROR orm.ClassWriter: No Java type for SQL type 1111 for column geom ERROR sqoop.Sqoop: Got exception running Sqoop: java.lang.NullPointerException java.lang.NullPointerException at org.apache.sqoop.orm.ClassWriter.parseColumn(ClassWriter.java:1417) at org.apache.sqoop.orm.ClassWriter.myGenerateParser(ClassWriter.java:1555) at org.apache.sqoop.orm.ClassWriter.generateParser(ClassWriter.java:1518) at org.apache.sqoop.orm.ClassWriter.generateClassForColumns(ClassWriter.java:1979) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1764) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:77) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:113) at org.apache.sqoop.Sqoop.run(Sqoop.java:151) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:187) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:241) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:250) at org.apache.sqoop.Sqoop.main(Sqoop.java:259) ```
... View more
Labels:
- Labels:
-
Apache Sqoop
09-02-2020
07:02 AM
Hi Ian, Thanks for taking time to have a look on it. I actually tried to add backslash ( \' ) but did not worked. I also tried to add 2 quotes (/!\ not double quotes) same, it did not worked. The only way it worked is to put the query behind double quotes : -q "my_query where col1 IN ('arg1','arg2')" PS: As a reminder my script is on a Shell script file .sh Thanks again Alex
... View more
09-02-2020
05:47 AM
Fixed by using double quote for the query "the query with where clause" (vs 'the query with where clause') Full cmd : impala-shell -B -q "select col1,col2,Col3 from MyDB.MyTable where col1 IN ('argument1','argument2','argument3','argument4','argument5') order by col1,col2" --delimited --output_delimiter='|' -o ${DIR_OUT}/MyTable_$(date +"%Y%m%d-%H%M%S").csv --print_header 2>${DIR_LOG}/MyTable_$(date +"%Y%m%d-%H%M%S").log; That's all folks !
... View more
09-02-2020
03:07 AM
Hi guys, This is basic question but not able to solve it... I created a scheduled workflow in Oozie to run a Shell script witch runs some impala-shell cmd. Everything worked fine since I added a where clause for one of them. I'm wondering if it's possible to pass a where clause on a query impala-shell ? Here is my basic cmd in shell script : impala-shell -B -q 'select col1,col2,Col3 from MyDB.MyTable where col1 IN ('argument1','argument2','argument3','argument4','argument5') order by col1,col2' --delimited --output_delimiter='|' -o ${DIR_OUT}/MyTable_$(date +"%Y%m%d-%H%M%S").csv --print_header 2>${DIR_LOG}/MyTable_$(date +"%Y%m%d-%H%M%S").log; It works fine without "where col1 IN ('argument1','argument2','argument3','argument4','argument5')" Is there a way to do it without using a file ? (with -f option ???) Thanks for your advise Alex
... View more
Labels:
- Labels:
-
Apache Impala