Development Setup of Hadoop 3.2.0

[#1]

I am trying to build the latest Hadoop-trunk branch for development. To build the distribution I am using the

 mvn clean install -DskipTests

command followed by

mvn package -Pdist,native,docs -DskipTests -Dtar

The maven build passes but I face the following error when trying to run hdfs namenode -format :

Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hdfs/protocol/ClientProtocol
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hdfs.protocol.ClientProtocol
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 19 more


I am using a Ubuntu-Xenial vagrant system. The following is the class path which is generated when --debug option is used :

CLASSPATH: /vagrant/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.2.0-SNAPSHOT/etc/hadoop:/vagrant/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.2.0-SNAPSHOT/share/hadoop/common/lib/*:/vagrant/hadoop-trunk/hadoop-common-project/hadoop-common/target/hadoop-common-3.2.0-SNAPSHOT/share/hadoop/common/*:/vagrant/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.2.0-SNAPSHOT/share/hadoop/hdfs:/vagrant/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.2.0-SNAPSHOT/share/hadoop/hdfs/lib/*:/vagrant/hadoop-trunk/hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.2.0-SNAPSHOT/share/hadoop/hdfs/*

and the HADOOP_OPTS

HADOOP_OPTS: -Djava.net.preferIPv4Stack=true -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dyarn.log.dir=/vagrant/hadoop-trunk/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/logs -Dyarn.log.file=hadoop.log -Dyarn.home.dir=/vagrant/hadoop-trunk/ -Dyarn.root.logger=INFO,console -Djava.library.path=/vagrant/hadoop-trunk/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/lib/native -Dhadoop.log.dir=/vagrant/hadoop-trunk/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/vagrant/hadoop-trunk/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT -Dhadoop.id.str=vagrant -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml

How can I go about fixing this error?

EDIT : Fixed

[#2 · Accepted solution]

Compile and install the packages as above and run all the commands from hadoop-dist/target/hadoop-3.2.0-SNAPSHOT directory and copy into etc/hadoop/conf all the config options.

Credits @stevel

[#3]

Even, i am facing a similar kind of error. My Hadoop cluster cam up seemingly well. But while trying to start 'sqoop2-server' i am getting the JNI error. Any leads to resolving this will be appreciated -

hadoop@master:/opt/hadoop/sqoop2/bin$ sqoop2-server start Setting conf dir: /opt/hadoop/sqoop2/conf Sqoop home directory: /opt/hadoop/sqoop2 Starting the Sqoop2 server... Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 64 at java.util.jar.JarFile.match(java.base@9-internal/JarFile.java:983) at java.util.jar.JarFile.checkForSpecialAttributes(java.base@9-internal/JarFile.java:1017) at java.util.jar.JarFile.isMultiRelease(java.base@9-internal/JarFile.java:399) at java.util.jar.JarFile.getEntry(java.base@9-internal/JarFile.java:524) at java.util.jar.JarFile.getJarEntry(java.base@9-internal/JarFile.java:480) at jdk.internal.util.jar.JarIndex.getJarIndex(java.base@9-internal/JarIndex.java:114) at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:640) at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:632) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(java.base@9-internal/URLClassPath.java:631) at jdk.internal.loader.URLClassPath$JarLoader.<init>(java.base@9-internal/URLClassPath.java:606) at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:386) at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:376) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:375) at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:352) at jdk.internal.loader.URLClassPath.getResource(java.base@9-internal/URLClassPath.java:218) at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:463) at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:460) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(java.base@9-internal/BuiltinClassLoader.java:459) at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(java.base@9-internal/BuiltinClassLoader.java:406) at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@9-internal/BuiltinClassLoader.java:364) at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@9-internal/ClassLoaders.java:184) at java.lang.ClassLoader.loadClass(java.base@9-internal/ClassLoader.java:419) at sun.launcher.LauncherHelper.loadMainClass(java.base@9-internal/LauncherHelper.java:585) at sun.launcher.LauncherHelper.checkAndLoadMain(java.base@9-internal/LauncherHelper.java:497)

Regards Pankaj

,

Even, i am facing a similar kind of error. My Hadoop cluster cam up seemingly well. But while trying to start 'sqoop2-server' i am getting the JNI error. Any leads to resolving this will be appreciated -

hadoop@master:/opt/hadoop/sqoop2/bin$ sqoop2-server start Setting conf dir: /opt/hadoop/sqoop2/conf Sqoop home directory: /opt/hadoop/sqoop2 Starting the Sqoop2 server... Error: A JNI error has occurred, please check your installation and try again Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 64 at java.util.jar.JarFile.match(java.base@9-internal/JarFile.java:983) at java.util.jar.JarFile.checkForSpecialAttributes(java.base@9-internal/JarFile.java:1017) at java.util.jar.JarFile.isMultiRelease(java.base@9-internal/JarFile.java:399) at java.util.jar.JarFile.getEntry(java.base@9-internal/JarFile.java:524) at java.util.jar.JarFile.getJarEntry(java.base@9-internal/JarFile.java:480) at jdk.internal.util.jar.JarIndex.getJarIndex(java.base@9-internal/JarIndex.java:114) at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:640) at jdk.internal.loader.URLClassPath$JarLoader$1.run(java.base@9-internal/URLClassPath.java:632) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.URLClassPath$JarLoader.ensureOpen(java.base@9-internal/URLClassPath.java:631) at jdk.internal.loader.URLClassPath$JarLoader.<init>(java.base@9-internal/URLClassPath.java:606) at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:386) at jdk.internal.loader.URLClassPath$3.run(java.base@9-internal/URLClassPath.java:376) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:375) at jdk.internal.loader.URLClassPath.getLoader(java.base@9-internal/URLClassPath.java:352) at jdk.internal.loader.URLClassPath.getResource(java.base@9-internal/URLClassPath.java:218) at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:463) at jdk.internal.loader.BuiltinClassLoader$3.run(java.base@9-internal/BuiltinClassLoader.java:460) at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method) at jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(java.base@9-internal/BuiltinClassLoader.java:459) at jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(java.base@9-internal/BuiltinClassLoader.java:406) at jdk.internal.loader.BuiltinClassLoader.loadClass(java.base@9-internal/BuiltinClassLoader.java:364) at jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(java.base@9-internal/ClassLoaders.java:184) at java.lang.ClassLoader.loadClass(java.base@9-internal/ClassLoader.java:419) at sun.launcher.LauncherHelper.loadMainClass(java.base@9-internal/LauncherHelper.java:585) at sun.launcher.LauncherHelper.checkAndLoadMain(java.base@9-internal/LauncherHelper.java:497)

Regards Pankaj

[#4]

you must copy hadoop-hdfs-client.jar to hadoop/hadoop-common-project/hadoop-common/target/hadoop-common-3.0.4-SNAPSHOT/share/hadoop/common dir, it works,i try it success.