Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Development Setup of Hadoop 3.2.0

Solved Go to solution
Highlighted

Development Setup of Hadoop 3.2.0

New Contributor

I am trying to build the latest Hadoop-trunk branch for development. To build the distribution I am using the

 mvn clean install -DskipTests


command followed by

mvn package -Pdist,native,docs -DskipTests -Dtar

The maven build passes but I face the following error when trying to run hdfs namenode -format :

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/protocol/ClientProtocol
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocol.ClientProtocol
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 19 more


I am using a Ubuntu-Xenial vagrant system. The following is the class path which is generated when --debug option is used :

CLASSPATH: /vagrant/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.2.0-SNAPSHOT/etc/hadoop:/vagrant/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.2.0-SNAPSHOT/share/hadoop/common/lib/*:/vagrant/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.2.0-SNAPSHOT/share/hadoop/common/*:/vagrant/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.2.0-SNAPSHOT/share/hadoop/hdfs:/vagrant/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.2.0-SNAPSHOT/share/hadoop/hdfs/lib/*:/vagrant/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.2.0-SNAPSHOT/share/hadoop/hdfs/*



and the HADOOP_OPTS

HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dyarn.log.dir=/vagrant/hadoop-trunk/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/vagrant/hadoop-trunk/ -Dyarn.root.logger=INFO,console -Djava.library.path=/vagrant/hadoop-trunk/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/lib/native -Dhadoop.log.dir=/vagrant/hadoop-trunk/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/vagrant/hadoop-trunk/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT -Dhadoop.id.str=vagrant -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml



How can I go about fixing this error?

EDIT : Fixed

1 ACCEPTED SOLUTION

Accepted Solutions
Highlighted

Re: Development Setup of Hadoop 3.2.0

New Contributor

Compile and install the packages as above and run all the commands from hadoop-dist/target/hadoop-3.2.0-SNAPSHOT directory and copy into etc/hadoop/conf all the config options.

Credits @stevel

View solution in original post

3 REPLIES 3
Highlighted

Re: Development Setup of Hadoop 3.2.0

New Contributor

Compile and install the packages as above and run all the commands from hadoop-dist/target/hadoop-3.2.0-SNAPSHOT directory and copy into etc/hadoop/conf all the config options.

Credits @stevel

View solution in original post

Highlighted

Re: Development Setup of Hadoop 3.2.0

New Contributor

Even, i am facing a similar kind of error. My Hadoop cluster cam up seemingly well. But while trying to start 'sqoop2-server' i am getting the JNI error. Any leads to resolving this will be appreciated -

hadoop@master:/opt/hadoop/sqoop2/bin$ sqoop2-server start Setting conf dir: /opt/hadoop/sqoop2/conf Sqoop home directory: /opt/hadoop/sqoop2 Starting the Sqoop2 server... Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 64 at java.util.jar.JarFile.match(java.base@9-internal/JarFile.java:983) at java.util.jar.JarFile.checkForSpecialAttributes(java.base@9-internal/JarFile.java:1017) at java.util.jar.JarFile.isMultiRelease(java.base@9-internal/JarFile.java:399) at java.util.jar.JarFile.getEntry(java.base@9-internal/JarFile.java:524) at java.util.jar.JarFile.getJarEntry(java.base@9-internal/JarFile.java:480) at jdk.internal.util.jar.JarIndex.getJarIndex(java.base@9-internal/JarIndex.java:114) at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:640) at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:632) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(java.base@9-internal/URLClassPath.java:631) at jdk.internal.loader.URLClassPath$JarLoader.<init>(java.base@9-internal/URLClassPath.java:606) at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:386) at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:376) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:375) at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:352) at jdk.internal.loader.URLClassPath.getResource(java.base@9-internal/URLClassPath.java:218) at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:463) at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:460) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(java.base@9-internal/BuiltinClassLoader.java:459) at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(java.base@9-internal/BuiltinClassLoader.java:406) at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@9-internal/BuiltinClassLoader.java:364) at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@9-internal/ClassLoaders.java:184) at java.lang.ClassLoader.loadClass(java.base@9-internal/ClassLoader.java:419) at sun.launcher.LauncherHelper.loadMainClass(java.base@9-internal/LauncherHelper.java:585) at sun.launcher.LauncherHelper.checkAndLoadMain(java.base@9-internal/LauncherHelper.java:497)

Regards Pankaj

,

Even, i am facing a similar kind of error. My Hadoop cluster cam up seemingly well. But while trying to start 'sqoop2-server' i am getting the JNI error. Any leads to resolving this will be appreciated -

hadoop@master:/opt/hadoop/sqoop2/bin$ sqoop2-server start Setting conf dir: /opt/hadoop/sqoop2/conf Sqoop home directory: /opt/hadoop/sqoop2 Starting the Sqoop2 server... Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 64 at java.util.jar.JarFile.match(java.base@9-internal/JarFile.java:983) at java.util.jar.JarFile.checkForSpecialAttributes(java.base@9-internal/JarFile.java:1017) at java.util.jar.JarFile.isMultiRelease(java.base@9-internal/JarFile.java:399) at java.util.jar.JarFile.getEntry(java.base@9-internal/JarFile.java:524) at java.util.jar.JarFile.getJarEntry(java.base@9-internal/JarFile.java:480) at jdk.internal.util.jar.JarIndex.getJarIndex(java.base@9-internal/JarIndex.java:114) at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:640) at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:632) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(java.base@9-internal/URLClassPath.java:631) at jdk.internal.loader.URLClassPath$JarLoader.<init>(java.base@9-internal/URLClassPath.java:606) at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:386) at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:376) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:375) at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:352) at jdk.internal.loader.URLClassPath.getResource(java.base@9-internal/URLClassPath.java:218) at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:463) at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:460) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(java.base@9-internal/BuiltinClassLoader.java:459) at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(java.base@9-internal/BuiltinClassLoader.java:406) at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@9-internal/BuiltinClassLoader.java:364) at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@9-internal/ClassLoaders.java:184) at java.lang.ClassLoader.loadClass(java.base@9-internal/ClassLoader.java:419) at sun.launcher.LauncherHelper.loadMainClass(java.base@9-internal/LauncherHelper.java:585) at sun.launcher.LauncherHelper.checkAndLoadMain(java.base@9-internal/LauncherHelper.java:497)

Regards Pankaj

Highlighted

Re: Development Setup of Hadoop 3.2.0

New Contributor

you must copy hadoop-hdfs-client.jar to hadoop/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.4-SNAPSHOT/share/hadoop/common dir, it works,i try it success.

Don't have an account?
Coming from Hortonworks? Activate your account here