We have pyspark program which ingests source files into hdfs and then load them into other hive tables. Program works fine on edgenode as source files are local in edgenode.Can we schedule this pyspark program through oozie with spark2-submit --deploy-mode client? ... View more

I am excuting create table using json serde "org.apache.hive.hcatalog.data.JsonSerDe"  in spark.sql on spark version 2.1.0.cloudera2   I am getting below error  java.lang.ClassNotFoundException Class org.apache.hive.hcatalog.data.JsonSerDe not found   while executing spark2-submit & pyspark2 i am using below --jars    --jars /opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar   I also tried set("sparks.jars","/opt/cloudera/parcels/CDH-5.13.3-1.cdh5.13.3.p0.2/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core.jar") inside my code . Nothing works   ... View more

  Getting below error after trying to query hive tables in impala   InvalidStorageDescriptorException: Impala does not support tables of this type. REASON: SerDe library 'org.apache.hive.hcatalog.data.JsonSerDe' is not supported.   I able to query after converting table to parquet format. Is there any way to read json files in Impala? ... View more

Does Falcon Mirror Provides below requirements? Easily configurable to allow for multiple datasets (Some datasets are more than 1TB in size) Support multiple streams of data processing Includes alerting for job failures Includes validation that data has been copied successfully. (Data should reconcile back to source) Support full and delta copies Support dynamic changes in the structure on source without requiring manual intervention on target Target data should always be available even during refresh/sync Need optimal data transfer speeds due to the volume of data Ideal solution would include near real time data sync (15 minutes) ... View more

I have CDH 5.5.0 installed on EC2   I am able to run below without any problem   sqoop list-databases \ --connect "jdbc:mysql://localhost:3306" \ --username root \ --password admin   sqoop list-tables --connect "jdbc:mysql://localhost/classicmodels" --username root --password admin   sqoop eval \ --connect "jdbc:mysql://localhost:3306/classicmodels" \ --username root \ --password admin \ --query "select count(1) from employees"     I am executing below command on namenode after installing mysql, i get error     ubuntu@ip-172-31-2-202:~$ sqoop import-all-tables   -m 1   --connect "jdbc:mysql://localhost:3306/classicmodels"   --username=root   --password=admin   --warehouse-dir=/user/hive/warehouse/sqoop_import Warning: /opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 16/05/01 15:22:08 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.0 16/05/01 15:22:08 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 16/05/01 15:22:08 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 16/05/01 15:22:09 INFO tool.CodeGenTool: Beginning code generation 16/05/01 15:22:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `customers` AS t LIMIT 1 16/05/01 15:22:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `customers` AS t LIMIT 1 16/05/01 15:22:09 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce Note: /tmp/sqoop-ubuntu/compile/47def880c4549b38bca1e9a3717b3c62/customers.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 16/05/01 15:22:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-ubuntu/compile/47def880c4549b38bca1e9a3717b3c62/customers.jar 16/05/01 15:22:10 WARN manager.MySQLManager: It looks like you are importing from mysql. 16/05/01 15:22:10 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 16/05/01 15:22:10 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 16/05/01 15:22:10 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 16/05/01 15:22:10 INFO mapreduce.ImportJobBase: Beginning import of customers 16/05/01 15:22:10 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 16/05/01 15:22:11 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 16/05/01 15:22:11 INFO client.RMProxy: Connecting to ResourceManager at ip-172-31-2-202.us-west-2.compute.internal/172.31.2.202:8032 16/05/01 15:22:12 INFO db.DBInputFormat: Using read commited transaction isolation 16/05/01 15:22:12 INFO mapreduce.JobSubmitter: number of splits:1 16/05/01 15:22:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1462108036993_0008 16/05/01 15:22:13 INFO impl.YarnClientImpl: Submitted application application_1462108036993_0008 16/05/01 15:22:13 INFO mapreduce.Job: The url to track the job: http://ip-172-31-2-202.us-west-2.compute.internal:8088/proxy/application_1462108036993_0008/ 16/05/01 15:22:13 INFO mapreduce.Job: Running job: job_1462108036993_0008 16/05/01 15:22:19 INFO mapreduce.Job: Job job_1462108036993_0008 running in uber mode : false 16/05/01 15:22:19 INFO mapreduce.Job:   map 0% reduce 0% 16/05/01 15:22:23 INFO mapreduce.Job: Task Id : attempt_1462108036993_0008_m_000000_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220) at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165) ... 9 more Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302) at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213) ... 10 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.<init>(Socket.java:425) at java.net.Socket.<init>(Socket.java:241) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308) ... 26 more   16/05/01 15:22:27 INFO mapreduce.Job: Task Id : attempt_1462108036993_0008_m_000000_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220) at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165) ... 9 more Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302) at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213) ... 10 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.<init>(Socket.java:425) at java.net.Socket.<init>(Socket.java:241) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308) ... 26 more   16/05/01 15:22:32 INFO mapreduce.Job: Task Id : attempt_1462108036993_0008_m_000000_2, Status : FAILED Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220) at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165) ... 9 more Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302) at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213) ... 10 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.<init>(Socket.java:425) at java.net.Socket.<init>(Socket.java:241) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308) ... 26 more   16/05/01 15:22:36 INFO mapreduce.Job:   map 100% reduce 0% 16/05/01 15:22:36 INFO mapreduce.Job: Job job_1462108036993_0008 failed with state FAILED due to: Task failed task_1462108036993_0008_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0   16/05/01 15:22:36 INFO mapreduce.Job: Counters: 8 Job Counters Failed map tasks=4 Launched map tasks=4 Other local map tasks=4 Total time spent by all maps in occupied slots (ms)=10988 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=10988 Total vcore-seconds taken by all map tasks=10988 Total megabyte-seconds taken by all map tasks=11251712 16/05/01 15:22:36 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 16/05/01 15:22:36 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 25.1268 seconds (0 bytes/sec) 16/05/01 15:22:36 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 16/05/01 15:22:36 INFO mapreduce.ImportJobBase: Retrieved 0 records. 16/05/01 15:22:36 ERROR tool.ImportAllTablesTool: Error during import: Import job failed! ubuntu@ip-172-31-2-202:~$ ... View more

I am able to run below without any problem   sqoop list-databases \ --connect "jdbc:mysql://localhost:3306" \ --username root \ --password admin   sqoop list-tables --connect "jdbc:mysql://localhost/classicmodels" --username root --password admin   sqoop eval \ --connect "jdbc:mysql://localhost:3306/classicmodels" \ --username root \ --password admin \ --query "select count(1) from employees"     I am executing below command on namenode after installing mysql, i get error     ubuntu@ip-172-31-2-202:~$ sqoop import-all-tables   -m 1   --connect "jdbc:mysql://localhost:3306/classicmodels"   --username=root   --password=admin   --warehouse-dir=/user/hive/warehouse/sqoop_import Warning: /opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 16/05/01 15:22:08 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.5.0 16/05/01 15:22:08 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 16/05/01 15:22:08 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 16/05/01 15:22:09 INFO tool.CodeGenTool: Beginning code generation 16/05/01 15:22:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `customers` AS t LIMIT 1 16/05/01 15:22:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `customers` AS t LIMIT 1 16/05/01 15:22:09 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /opt/cloudera/parcels/CDH/lib/hadoop-mapreduce Note: /tmp/sqoop-ubuntu/compile/47def880c4549b38bca1e9a3717b3c62/customers.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 16/05/01 15:22:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-ubuntu/compile/47def880c4549b38bca1e9a3717b3c62/customers.jar 16/05/01 15:22:10 WARN manager.MySQLManager: It looks like you are importing from mysql. 16/05/01 15:22:10 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 16/05/01 15:22:10 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 16/05/01 15:22:10 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 16/05/01 15:22:10 INFO mapreduce.ImportJobBase: Beginning import of customers 16/05/01 15:22:10 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 16/05/01 15:22:11 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 16/05/01 15:22:11 INFO client.RMProxy: Connecting to ResourceManager at ip-172-31-2-202.us-west-2.compute.internal/172.31.2.202:8032 16/05/01 15:22:12 INFO db.DBInputFormat: Using read commited transaction isolation 16/05/01 15:22:12 INFO mapreduce.JobSubmitter: number of splits:1 16/05/01 15:22:12 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1462108036993_0008 16/05/01 15:22:13 INFO impl.YarnClientImpl: Submitted application application_1462108036993_0008 16/05/01 15:22:13 INFO mapreduce.Job: The url to track the job: http://ip-172-31-2-202.us-west-2.compute.internal:8088/proxy/application_1462108036993_0008/ 16/05/01 15:22:13 INFO mapreduce.Job: Running job: job_1462108036993_0008 16/05/01 15:22:19 INFO mapreduce.Job: Job job_1462108036993_0008 running in uber mode : false 16/05/01 15:22:19 INFO mapreduce.Job:   map 0% reduce 0% 16/05/01 15:22:23 INFO mapreduce.Job: Task Id : attempt_1462108036993_0008_m_000000_0, Status : FAILED Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220) at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165) ... 9 more Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302) at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213) ... 10 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.<init>(Socket.java:425) at java.net.Socket.<init>(Socket.java:241) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308) ... 26 more   16/05/01 15:22:27 INFO mapreduce.Job: Task Id : attempt_1462108036993_0008_m_000000_1, Status : FAILED Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220) at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165) ... 9 more Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302) at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213) ... 10 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.<init>(Socket.java:425) at java.net.Socket.<init>(Socket.java:241) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308) ... 26 more   16/05/01 15:22:32 INFO mapreduce.Job: Task Id : attempt_1462108036993_0008_m_000000_2, Status : FAILED Error: java.lang.RuntimeException: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73) at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) Caused by: java.lang.RuntimeException: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220) at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165) ... 9 more Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure   The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:358) at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2489) at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2526) at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2311) at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834) at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416) at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347) at java.sql.DriverManager.getConnection(DriverManager.java:571) at java.sql.DriverManager.getConnection(DriverManager.java:215) at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302) at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213) ... 10 more Caused by: java.net.ConnectException: Connection refused at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) at java.net.Socket.connect(Socket.java:579) at java.net.Socket.connect(Socket.java:528) at java.net.Socket.<init>(Socket.java:425) at java.net.Socket.<init>(Socket.java:241) at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:256) at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:308) ... 26 more   16/05/01 15:22:36 INFO mapreduce.Job:   map 100% reduce 0% 16/05/01 15:22:36 INFO mapreduce.Job: Job job_1462108036993_0008 failed with state FAILED due to: Task failed task_1462108036993_0008_m_000000 Job failed as tasks failed. failedMaps:1 failedReduces:0   16/05/01 15:22:36 INFO mapreduce.Job: Counters: 8 Job Counters Failed map tasks=4 Launched map tasks=4 Other local map tasks=4 Total time spent by all maps in occupied slots (ms)=10988 Total time spent by all reduces in occupied slots (ms)=0 Total time spent by all map tasks (ms)=10988 Total vcore-seconds taken by all map tasks=10988 Total megabyte-seconds taken by all map tasks=11251712 16/05/01 15:22:36 WARN mapreduce.Counters: Group FileSystemCounters is deprecated. Use org.apache.hadoop.mapreduce.FileSystemCounter instead 16/05/01 15:22:36 INFO mapreduce.ImportJobBase: Transferred 0 bytes in 25.1268 seconds (0 bytes/sec) 16/05/01 15:22:36 WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead 16/05/01 15:22:36 INFO mapreduce.ImportJobBase: Retrieved 0 records. 16/05/01 15:22:36 ERROR tool.ImportAllTablesTool: Error during import: Import job failed! ubuntu@ip-172-31-2-202:~$ ... View more