Created 02-11-2017 02:27 AM
The command is:
sqoop import \ --connect "jdbc:mysql://sandbox.hortonworks.com:3306/retail_db" \ --username=root \ --password=hadoop \ --table departments \ --hive-home /apps/hive/warehouse \ --hive-import \ --hive-overwrite \ --hive-table sqoop_import.departments \ --outdir java_files
The output is:
Warning: /usr/hdp/2.5.0.0-1245/accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 17/02/11 02:15:06 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.5.0.0-1245 17/02/11 02:15:06 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 17/02/11 02:15:06 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 17/02/11 02:15:06 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc. 17/02/11 02:15:06 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 17/02/11 02:15:06 INFO tool.CodeGenTool: Beginning code generation 17/02/11 02:15:06 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `departments` AS t LIMIT 1 17/02/11 02:15:06 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `departments` AS t LIMIT 1 17/02/11 02:15:06 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.5.0.0-1245/hadoop-mapreduce Note: /tmp/sqoop-root/compile/eef451e1cc95fb2071ebe74f5d9371e9/departments.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 17/02/11 02:15:08 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/eef451e1cc95fb2071ebe74f5d9371e9/departments.jar 17/02/11 02:15:08 WARN manager.MySQLManager: It looks like you are importing from mysql. 17/02/11 02:15:08 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 17/02/11 02:15:08 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 17/02/11 02:15:08 INFO mapreduce.ImportJobBase: Beginning import of departments 17/02/11 02:15:09 INFO impl.TimelineClientImpl: Timeline service address: http://sandbox.hortonworks.com:8188/ws/v1/timeline/ 17/02/11 02:15:09 INFO client.RMProxy: Connecting to ResourceManager at sandbox.hortonworks.com/172.17.0.2:8050 17/02/11 02:15:09 INFO client.AHSProxy: Connecting to Application History server at sandbox.hortonworks.com/172.17.0.2:10200 17/02/11 02:15:14 INFO db.DBInputFormat: Using read commited transaction isolation 17/02/11 02:15:14 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(`department_id`), MAX(`department_id`) FROM `departments` 17/02/11 02:15:14 INFO db.IntegerSplitter: Split size: 1; Num splits: 4 from: 2 to: 7 17/02/11 02:15:14 INFO mapreduce.JobSubmitter: number of splits:4 17/02/11 02:15:14 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1486771879029_0004 17/02/11 02:15:15 INFO impl.YarnClientImpl: Submitted application application_1486771879029_0004 17/02/11 02:15:15 INFO mapreduce.Job: The url to track the job: http://sandbox.hortonworks.com:8088/proxy/application_1486771879029_0004/ 17/02/11 02:15:15 INFO mapreduce.Job: Running job: job_1486771879029_0004 17/02/11 02:15:21 INFO mapreduce.Job: Job job_1486771879029_0004 running in uber mode : false 17/02/11 02:15:21 INFO mapreduce.Job: map 0% reduce 0% 17/02/11 02:15:29 INFO mapreduce.Job: map 50% reduce 0% 17/02/11 02:15:30 INFO mapreduce.Job: map 75% reduce 0% 17/02/11 02:15:31 INFO mapreduce.Job: map 100% reduce 0% 17/02/11 02:15:31 INFO mapreduce.Job: Job job_1486771879029_0004 completed successfully 17/02/11 02:15:31 INFO mapreduce.Job: Counters: 30 File System Counters FILE: Number of bytes read=0 FILE: Number of bytes written=652000 FILE: Number of read operations=0 FILE: Number of large read operations=0 FILE: Number of write operations=0 HDFS: Number of bytes read=481 HDFS: Number of bytes written=60 HDFS: Number of read operations=16 HDFS: Number of large read operations=0 HDFS: Number of write operations=8 Job Counters Launched map tasks=4 Other local map tasks=4 Total time spent by all maps in occupied slots (ms)=19510 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=19510 Total vcore-milliseconds taken by all map tasks=19510 Total megabyte-milliseconds taken by all map tasks=4877500 Map-Reduce Framework Map input records=6 Map output records=6 Input split bytes=481 Spilled Records=0 Failed Shuffles=0 Merged Map outputs=0 GC time elapsed (ms)=930 CPU time spent (ms)=3690 Physical memory (bytes) snapshot=548311040 Virtual memory (bytes) snapshot=7742504960 Total committed heap usage (bytes)=176160768 File Input Format Counters Bytes Read=0 File Output Format Counters Bytes Written=60 17/02/11 02:15:31 INFO mapreduce.ImportJobBase: Transferred 60 bytes in 22.7172 seconds (2.6412 bytes/sec) 17/02/11 02:15:31 INFO mapreduce.ImportJobBase: Retrieved 6 records. 17/02/11 02:15:31 INFO mapreduce.ImportJobBase: Publishing Hive/Hcat import job data to Listeners 17/02/11 02:15:31 INFO atlas.ApplicationProperties: Looking for atlas-application.properties in classpath 17/02/11 02:15:31 INFO atlas.ApplicationProperties: Loading atlas-application.properties from file:/etc/sqoop/2.5.0.0-1245/0/atlas-application.properties 17/02/11 02:15:32 ERROR security.InMemoryJAASConfiguration: Unable to add JAAS configuration for client [KafkaClient] as it is missing param [atlas.jaas.KafkaClient.loginModuleName]. Skipping JAAS config for [KafkaClient] 17/02/11 02:15:32 INFO hook.AtlasHook: Created Atlas Hook
Created 02-11-2017 03:34 AM
on your sandbox, please confirm you have service atlas running. if not please enable it and rerun sqoop command above
Created 02-11-2017 03:34 AM
on your sandbox, please confirm you have service atlas running. if not please enable it and rerun sqoop command above
Created 02-15-2017 09:18 PM
Thank you @Sunile Manjee,
It has worked after starting the dependencies of Atlas (Ambari Infra, HBase and Kafka).
Junior.
Created 06-29-2018 09:56 AM
Hi Does this mean Sqoop will never work if i do not have atlas service installed?
I used sqoop list databases and that works fine, but when I try to import i get the "Using Hive-specific delimiters for output. You can override" error