How do I put the sql connector jar in /var/lib/sqoop? Thanks ... View more

cloudera/cloudera worked for me, as of Sep 30, 2019. ... View more

Any solution to this issue? using --append in place of --lastmodified is not the correct solution as it won't update the record but create new record in hive.   --delete-target-dir defeats the purpose to update data as it will create new directory everytime which is same as importing entire source table into hdfs-hive everytime.   I tried using --merge-key but it gives following error:   19/03/20 07:07:41 ERROR tool.ImportTool: Import failed: java.io.IOException: Could not load jar /tmp/sqoop-gfctwnsg/compile/c63dd58c7ae7aa383d4fe8e795fd8604/FRESH.EMPLOYEERUSHI.jar into JVM. (Could not find class FRESH.EMPLOYEERUSHI.) at org.apache.sqoop.util.ClassLoaderStack.addJarFile(ClassLoaderStack.java:92) at com.cloudera.sqoop.util.ClassLoaderStack.addJarFile(ClassLoaderStack.java:36) at org.apache.sqoop.tool.ImportTool.loadJars(ImportTool.java:120) at org.apache.sqoop.tool.ImportTool.lastModifiedMerge(ImportTool.java:456) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:522) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:621) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) Caused by: java.lang.ClassNotFoundException: FRESH.EMPLOYEERUSHI at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.net.FactoryURLClassLoader.loadClass(URLClassLoader.java:814) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at org.apache.sqoop.util.ClassLoaderStack.addJarFile(ClassLoaderStack.java:88)     My sqoop command is as follows:   sqoop import \ --connect "jdbc:oracle:thin:@oraasmwd17-scan.nam.nsroot.net:8889/GENIFRD" \ --username FRESH \ --password C1T12016 \ --table FRESH.EMPLOYEERUSHI \ --merge-key id \ --target-dir /data/gfctwnsg/staging/hive/gfctwnsg_staging/rp86813/sqoopimportdir \ --incremental lastmodified \ --check-column MODIFIED_DATE \ --last-value '2019-03-20 06:43:59.0' \   My source Oracle table is as follows: 1 Rushi Pradhan engineer 30000 18-MAR-19 2 abc xyz doctor 20000 18-MAR-19   I changed the salary of id =1 and updated corresponding date manually. Now I want to reflect this change at hive end also. But there it is not allowing me to update the record but to only append. ... View more

Hi I am targeting ojdbc7.jar.  I have both ojdbc6 and ojdbc7 in my /opt/cloudera/parcels/CDH/lib/sqoop/lib/ I ran sqoop import --libjars ...../sqoop/lib/ojdbc7.jar --driver "oracle.jdbc.driver.OracleDriver" --connect "jdbc:oracle:thin:@(description=(ADDRESS=(PROTOCOL=TCP)(HOST=XXXXX.XXXX.com)(PORT=XXXX))(CONNECT_DATA=(SERVER=DEDICATED)(service_name=XXXXX.XXXXX.com)))" --username XXXXXX --password XXXXXX --query 'select * from XXXXXXX where rownum < 100 and $CONDITIONS' --fields-terminated-by  "\001" --target-dir /XXX/XXXX/XXXXX/ --num-mappers 1 --verbose --delete-target-dir The job worked but I can't determine if it used driver ojdbc7 or odbc6 as i see both in log  DEBUG mapreduce.JobBase: Adding to job classpath: file:..... ..../sqoop/lib/ojdbc6.jar  DEBUG mapreduce.JobBase: Adding to job classpath: file:..... ..../sqoop/lib/ojdbc7.jar is there a way to target only to use the ojdbc7.jar and not any other driver? trying to test this new ojdbc7 and I am not sure if it is still using old 6? also I am reaing compatability issues with 9 and the ojdbc7, should I remove ojdbc6 all together and replace with ojdbc6 or keep both? If I keep both how do I target one driver vs. the other        ... View more

Hello Audi et al,   I have seen the error below when the lock files in the Derby database directory are owned by a user other than Sqoop2.  This can happen if you manually connect to the database using the Derby "ij" tool.   Server startup failure org.apache.sqoop.common.SqoopException: JDBCREPO_0007:Unable to lease link at org.apache.sqoop.repository.JdbcRepositoryTransaction.begin(JdbcRepositoryTransaction.java:64) <SNIP> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection. at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:1Caused by: java.util.NoSuchElementException: Could not create a validated object, cause: A read-only user or a user in a read-only database is not permitted to disable read-only mode on a connection.   # ls -l total 24 -rw-r--r-- 1 root root 4 Aug 10 11:28 dbex.lck <====== -rw-r--r-- 1 root root 38 Aug 10 11:28 db.lck <======= drwxr-xr-x 2 sqoop2 sqoop2 4096 Aug 10 10:15 log drwxr-xr-x 2 sqoop2 sqoop2 4096 Aug 10 10:15 seg0 -rw-r--r-- 1 sqoop2 sqoop2 853 Aug 10 10:15 service.properties drwxr-xr-x 2 sqoop2 sqoop2 4096 Aug 10 13:49 tmp   To get past this issue do the following: Ensure that there are no connections to the database Address the lock files Option 1: chown sqoop2:sqoop2 *.lck Option 2: rm -rf *.lck Start Sqoop2   Hope this helps, Markus Kemper - Cloudera Support ... View more

Thanks @Romainr for reply. Is there any other way to fix apart from changing the db..? ... View more

for this requirment you can write join quiry withing sqoop import statement..coz of this u can import only required data in hadoop ecosystem.  ... View more

What ever is available in your CDH distribution. If you're using parcels, take a look at /opt/cloudera/parcles/CDH/jars. If you're using packages, take a look at /usr/lib/kite. You can upload that jar to any directory in HDFS and make it available to your workflow by including it using the "file" element (https://oozie.apache.org/docs/4.0.0/WorkflowFunctionalSpec.html#a3.2.2.1_Adding_Files_and_Archives_for_the_Job). That would probably be the easiest way to test. ... View more

We're still working on better facilities to look at Sqoop2 logs, but there are currently a few locations you can look (depending on your error). I'm assuming Sqoop2 starts fine, but a job you're running is failing. If that is the case, then your logs should be under /var/log/sqoop2/ and in its associated task/attempt logs. You can view these logs through the Yarn UIs (usually on port 8088. e.g. http://example.com:8088) or through Hue (http://gethue.com/). ... View more

Nara,   you could try deleting the /data1/yarn/nm/usercache/hdfs directory from your datanodes - as suggested in here: http://community.cloudera.com/t5/Batch-Processing-and-Workflow/Can-t-create-directory-yarn-nm-usercache-urika-appcache/td-p/24891   That worked for me. ... View more