Member since
05-09-2018
5
Posts
0
Kudos Received
0
Solutions
05-28-2018
01:13 PM
I installed Hadoop 3.0.0 and Sqoop 1.4.6, I started all hadoop services and while working with sqoop import command i am receiving following java exception, HNBND110:libexec GVB$ sqoop import --connect jdbc:mysql://localhost/metastore --username root -P --table emp --m 1;
2018-05-18 11:25:31,790 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Enter password:
2018-05-18 11:25:35,556 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2018-05-18 11:25:35,556 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/2.3.1/libexec/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Fri May 18 11:25:36 IST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-05-18 11:25:37,061 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `emp` AS t LIMIT 1
2018-05-18 11:25:37,112 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `emp` AS t LIMIT 1
2018-05-18 11:25:37,135 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/Cellar/Hadoop/3.0.0
Note: /tmp/sqoop-GVB/compile/f12b2451001b1996c5a961ba73dbfccc/emp.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2018-05-18 11:25:43,997 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-GVB/compile/f12b2451001b1996c5a961ba73dbfccc/emp.jar
2018-05-18 11:25:44,060 WARN manager.MySQLManager: It looks like you are importing from mysql.
2018-05-18 11:25:44,060 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
2018-05-18 11:25:44,061 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
2018-05-18 11:25:44,061 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
2018-05-18 11:25:44,067 INFO mapreduce.ImportJobBase: Beginning import of emp
2018-05-18 11:25:44,591 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-05-18 11:25:44,638 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.DistributedFileSystem could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2558)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2569)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2586)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2625)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(FileOutputFormat.java:160)
at org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:156)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/GlobalStorageStatistics$StorageStatisticsProvider
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.newInstance(Class.java:412)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 25 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.GlobalStorageStatistics$StorageStatisticsProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 30 more Please let me know how to resolve this exception. Is my sqoop compatible with hadoop version. Hadoop : 3.0.0, Hive : 2.3.1, Sqoop : 1.4.6.
... View more
Labels:
05-28-2018
05:57 AM
I installed Hadoop 3.0.0 and Sqoop 1.4.6, I started all hadoop services and while working with sqoop import command i am receiving following java exception,
CHNBND110:libexec GVB$ sqoop import --connect jdbc:mysql://localhost/metastore --username root -P --table emp --m 1;
2018-05-18 11:25:31,790 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
Enter password:
2018-05-18 11:25:35,556 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
2018-05-18 11:25:35,556 INFO tool.CodeGenTool: Beginning code generation
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/2.3.1/libexec/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
Fri May 18 11:25:36 IST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
2018-05-18 11:25:37,061 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `emp` AS t LIMIT 1
2018-05-18 11:25:37,112 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `emp` AS t LIMIT 1
2018-05-18 11:25:37,135 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/Cellar/Hadoop/3.0.0
Note: /tmp/sqoop-GVB/compile/f12b2451001b1996c5a961ba73dbfccc/emp.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
2018-05-18 11:25:43,997 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-GVB/compile/f12b2451001b1996c5a961ba73dbfccc/emp.jar
2018-05-18 11:25:44,060 WARN manager.MySQLManager: It looks like you are importing from mysql.
2018-05-18 11:25:44,060 WARN manager.MySQLManager: This transfer can be faster! Use the --direct
2018-05-18 11:25:44,061 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path.
2018-05-18 11:25:44,061 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql)
2018-05-18 11:25:44,067 INFO mapreduce.ImportJobBase: Beginning import of emp
2018-05-18 11:25:44,591 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2018-05-18 11:25:44,638 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.DistributedFileSystem could not be instantiated
at java.util.ServiceLoader.fail(ServiceLoader.java:232)
at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2558)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2569)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2586)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2625)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(FileOutputFormat.java:160)
at org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:156)
at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259)
at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673)
at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/GlobalStorageStatistics$StorageStatisticsProvider
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
at java.lang.Class.getConstructor0(Class.java:3075)
at java.lang.Class.newInstance(Class.java:412)
at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380)
... 25 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.GlobalStorageStatistics$StorageStatisticsProvider
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 30 more
Please let me know how to resolve this exception. Is my sqoop compatible with hadoop version.
Hadoop : 3.0.0, Hive : 2.3.1, Sqoop : 1.4.6.
... View more
05-28-2018
05:13 AM
I installed Hadoop 3.0.0 and Sqoop 1.4.6, I started all hadoop services and while working with sqoop import command i am receiving following java exception, CHND:Desktop GKM$ sqoop import --connect "jdbc:mysql://localhost/metastore" --username hivelogin -P --table emp -target-dir /user/hive/Warehouse -m 1 2018-05-28 15:28:05,667 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6 Enter password: 2018-05-28 15:28:08,432 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 2018-05-28 15:28:08,432 INFO tool.CodeGenTool: Beginning code generation SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/usr/local/Cellar/hive/2.3.1/libexec/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/usr/local/Cellar/hbase/1.2.6/libexec/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. Mon May 28 15:28:09 IST 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification. 2018-05-28 15:28:09,934 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `emp` AS t LIMIT 1 2018-05-28 15:28:09,951 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `emp` AS t LIMIT 1 2018-05-28 15:28:09,958 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /usr/local/Cellar/Hadoop/3.0.0 Note: /tmp/sqoop-gkm/compile/f337db0cb3ee1720a96571a3db70df1b/emp.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 2018-05-28 15:28:11,845 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-gkm/compile/f337db0cb3ee1720a96571a3db70df1b/emp.jar 2018-05-28 15:28:11,861 WARN manager.MySQLManager: It looks like you are importing from mysql. 2018-05-28 15:28:11,861 WARN manager.MySQLManager: This transfer can be faster! Use the --direct 2018-05-28 15:28:11,861 WARN manager.MySQLManager: option to exercise a MySQL-specific fast path. 2018-05-28 15:28:11,862 INFO manager.MySQLManager: Setting zero DATETIME behavior to convertToNull (mysql) 2018-05-28 15:28:11,868 INFO mapreduce.ImportJobBase: Beginning import of emp 2018-05-28 15:28:12,049 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2018-05-28 15:28:12,069 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.DistributedFileSystem could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2558) at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2569) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2586) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2625) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2607) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:167) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(FileOutputFormat.java:160) at org.apache.sqoop.mapreduce.ImportJobBase.configureOutputFormat(ImportJobBase.java:156) at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:259) at org.apache.sqoop.manager.SqlManager.importTable(SqlManager.java:673) at org.apache.sqoop.manager.MySQLManager.importTable(MySQLManager.java:118) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:497) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605) at org.apache.sqoop.Sqoop.run(Sqoop.java:143) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227) at org.apache.sqoop.Sqoop.main(Sqoop.java:236) Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/GlobalStorageStatistics$StorageStatisticsProvider at java.lang.Class.getDeclaredConstructors0(Native Method) at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671) at java.lang.Class.getConstructor0(Class.java:3075) at java.lang.Class.newInstance(Class.java:412) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 25 more Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.GlobalStorageStatistics$StorageStatisticsProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 30 more Please let me know how to resolve this exception. Is my sqoop compatible with hadoop version. Hadoop : 3.0.0, Hive : 2.3.1, Sqoop : 1.4.6.
... View more
- Tags:
- Sqoop
05-09-2018
02:51 AM
Hi, I have installed Hadoop 3.0.0 and Hive 2.3.1 in my PC. Parallely i installed mysql and working with sql commands in sql shell mode and working fine. But While executing queries in Hive shell mode, i am receiving the following error, hive> create table saurzcode(id int, name string);
FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
I configured the hive-site.xml properly and i created the hiveuser on the metastore database in mysql shell. And i granted privileges for that user and exected schemas. But while executing any queries in hive shell i am getting exception. Please let me know the solution for this exception.
... View more
Labels:
