Support Questions
Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Innovation Accelerator group hub.

Unable to import data using sqoop

New Contributor

I want to import data from MySQL to remote Hive using sqoop. I have installed Sqoop on a middleware machine. When i run this command:

sqoop import --driver com.mysql.jdbc.Driver --connect jdbc:mysql://192.168.2.146:3306/fir --username root -P -m 1 --table beard_size_list --connect jdbc:hive2://192.168.2.141:10000/efir --username oracle -P -m 1 --hive-table lnd_beard_size_list --hive-import;

Is this command correct can i import data from remote MySQL to remote Hive?

When i ran this command it keeps on trying to connect to resource manager:

17/11/01 10:54:05 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.1.0-129
Enter password: 
17/11/01 10:54:10 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
17/11/01 10:54:10 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
17/11/01 10:54:10 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
17/11/01 10:54:10 INFO manager.SqlManager: Using default fetchSize of 1000
17/11/01 10:54:10 INFO tool.CodeGenTool: Beginning code generation
17/11/01 10:54:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM beard_size_list AS t WHERE 1=0
17/11/01 10:54:11 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM beard_size_list AS t WHERE 1=0
17/11/01 10:54:11 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/hdp/2.6.1.0-129/hadoop-mapreduce
Note: /tmp/sqoop-oracle/compile/d93080265a09913fbfe9e06e92d314a3/beard_size_list.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/11/01 10:54:15 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-oracle/compile/d93080265a09913fbfe9e06e92d314a3/beard_size_list.jar
17/11/01 10:54:15 INFO mapreduce.ImportJobBase: Beginning import of beard_size_list
17/11/01 10:54:15 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
17/11/01 10:54:15 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM beard_size_list AS t WHERE 1=0
17/11/01 10:54:17 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps
17/11/01 10:54:17 INFO client.RMProxy: Connecting to ResourceManager at hortonworksn2.com/192.168.2.191:8050
17/11/01 10:54:17 INFO client.AHSProxy: Connecting to Application History server at hortonworksn2.com/192.168.2.191:10200
17/11/01 10:54:19 INFO ipc.Client: Retrying connect to server: hortonworksn2.com/192.168.2.191:8050. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
17/11/01 10:54:20 INFO ipc.Client: Retrying connect to server: hortonworksn2.com/192.168.2.191:8050. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
17/11/01 10:54:21 INFO ipc.Client: Retrying connect to server: hortonworksn2.com/192.168.2.191:8050. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
17/11/01 10:54:22 INFO ipc.Client: Retrying connect to server: hortonworksn2.com/192.168.2.191:8050. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)
17/11/01 10:54:23 INFO ipc.Client: Retrying connect to server: hortonworksn2.com/192.168.2.191:8050. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)

The port it is trying to connect is 8050 but the actual port is 8033. How can i fix this. Kindly help.

1 REPLY 1

Super Mentor

@Daniyal Bashir

Your sqoop command looks good. However as we see that it is keep on retrying to connect:

17/11/01 10:54:20 INFO ipc.Client: Retrying connect to server: hortonworksn2.com/192.168.2.191:8050. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=50, sleepTime=1000 MILLISECONDS)

.

Looks like the hortonworksn2.com:8050 is your resource manager address to submit the application. Can you please confirm if that host and port is accessible from the host where you are trying to run the sqoop command?

# nc -v  hortonworksn2.com  8050

.

Also can you please check the Resource Manager logs and the UI to see if it is running healthy Or getting any error or warning?

It might be related to resource limitations as well on the Yarn side. So please check if NodeManager & Resource Managers are having enough memory.

It will be also good if you can try running the SQOOP import command in verbose mode to see if it is using correct classpath setting and other details to see why is it not able to connect. We can enbale verbose logging by adding the flag "--verbose" as following:

# sqoop import --verbose --driver com.mysql.jdbc.Driver --connect jdbc:mysql://192.168.2.146:3306/fir --username root -P -m 1 --table beard_size_list --connect jdbc:hive2://192.168.2.141:10000/efir --username oracle -P -m 1 --hive-table lnd_beard_size_list --hive-import;

.