Member since
09-28-2021
5
Posts
0
Kudos Received
0
Solutions
10-05-2021
04:15 AM
The problem was my classpath. export CLASSPATH=$(hadoop classpath --glob) wild cards not supported with Hadoop 3.0.0-cdh6.3.4
... View more
09-28-2021
01:27 AM
I have installed CDH on RHEL 7.9 .
The cloudera installation files are located in - /opt/cloudera/
java files are in /usr/java/jdk1.8.0_181-cloudera
I am trying to use apis in libhdfs.so from my application -
when I launch my application through command line I get the below error -
could not find method getRootCauseMessage from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String; could not find method getStackTrace from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String; FileSystem: loadFileSystems failed error: (unable to get root cause for java.lang.NoClassDefFoundError) (unable to get stack trace for java.lang.NoClassDefFoundError) # A fatal error has been detected by the Java Runtime Environment: # SIGSEGV (0xb) at pc=0x00007f21231904f1, pid=28247, tid=0x00007f1f5e7fc700 # JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops) # Problematic frame: # C [libhdfs.so+0xa4f1] # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # An error report file with more information is saved as: # /root/hs_err_pid28247.log # If you would like to submit a bug report, please visit: # http://bugreport.java.com/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug.
Form the error log -
Internal exceptions (10 events): Event: 0.057 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x00000005801663a8) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435] Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x00000005801670f0) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435] Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x0000000580167300) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435] Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x0000000580168030) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435] Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x0000000580168240) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435] Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x0000000580168f18) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435] Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x0000000580169128) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435] Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/lang/NoClassDefFoundError': org/apache/hadoop/fs/FileSystem> (0x00000005801695b0) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 199] Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/lang/NoSuchMethodError': getRootCauseMessage> (0x00000005801699f0) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jni.cpp, line 1581] Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/lang/NoSuchMethodError': getStackTrace> (0x0000000580169c40) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jni.cpp, line 1581]
VM Arguments: java_command: <unknown> java_class_path (initial): /etc/hadoop/conf:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-yarn/.//* Launcher Type: generic
Environment Variables: JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera/ CLASSPATH=/etc/hadoop/conf:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-yarn/.//* PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin LD_LIBRARY_PATH=/usr/java/jdk1.8.0_181-cloudera/jre/lib/amd64/server/libjvm.so SHELL=/bin/bash
I have set my env vars as -
[root@n138 ~]# echo $NHDFS_LIBJVM_PATH
/usr/java/jdk1.8.0_181-cloudera/jre/lib/amd64/server/libjvm.so
[root@n138 ~]# echo $NHDFS_LIBHDFS_PATH
/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib64/libhdfs.so
export CLASSPATH=$(hadoop classpath)
[root@n138 ~]# ldd /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib64/libhdfs.so linux-vdso.so.1 => (0x00007ffecb589000) libjvm.so => /lib64/libjvm.so (0x00007f27987c6000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f27985c2000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f27983a6000) libc.so.6 => /lib64/libc.so.6 (0x00007f2797fd8000) /lib64/ld-linux-x86-64.so.2 (0x00007f27999d9000) libm.so.6 => /lib64/libm.so.6 (0x00007f2797cd6000) [root@n138 ~]#
what is wrong with my setup ? why I am unable to to load libhdfs.so ?
... View more
Labels: