FileSystem: loadFileSystems failed error

sky1010 — Fri, 16 Sep 2022 14:43:42 GMT

I have installed CDH on RHEL 7.9 .

The cloudera installation files are located in - /opt/cloudera/

java files are in /usr/java/jdk1.8.0_181-cloudera

I am trying to use apis in libhdfs.so from my application -

when I launch my application through command line I get the below error -

could not find method getRootCauseMessage from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String;
could not find method getStackTrace from class (null) with signature (Ljava/lang/Throwable;)Ljava/lang/String;
FileSystem: loadFileSystems failed error:
(unable to get root cause for java.lang.NoClassDefFoundError)
(unable to get stack trace for java.lang.NoClassDefFoundError)
# A fatal error has been detected by the Java Runtime Environment:
# SIGSEGV (0xb) at pc=0x00007f21231904f1, pid=28247, tid=0x00007f1f5e7fc700
# JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C [libhdfs.so+0xa4f1]
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
# An error report file with more information is saved as:
# /root/hs_err_pid28247.log
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

Form the error log -

Internal exceptions (10 events):
Event: 0.057 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x00000005801663a8) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435]
Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x00000005801670f0) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435]
Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x0000000580167300) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435]
Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x0000000580168030) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435]
Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x0000000580168240) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435]
Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x0000000580168f18) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435]
Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/security/PrivilegedActionException'> (0x0000000580169128) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jvm.cpp, line 1435]
Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/lang/NoClassDefFoundError': org/apache/hadoop/fs/FileSystem> (0x00000005801695b0) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/classfile/systemDictionary.cpp, line 199]
Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/lang/NoSuchMethodError': getRootCauseMessage> (0x00000005801699f0) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jni.cpp, line 1581]
Event: 0.058 Thread 0x00007f211800a000 Exception <a 'java/lang/NoSuchMethodError': getStackTrace> (0x0000000580169c40) thrown at [/HUDSON/workspace/8-2-build-linux-amd64/jdk8u181/11358/hotspot/src/share/vm/prims/jni.cpp, line 1581]

VM Arguments:
java_command: <unknown>
java_class_path (initial): /etc/hadoop/conf:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-yarn/.//*
Launcher Type: generic

Environment Variables:
JAVA_HOME=/usr/java/jdk1.8.0_181-cloudera/
CLASSPATH=/etc/hadoop/conf:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib/hadoop/libexec/../../hadoop-yarn/.//*
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin
LD_LIBRARY_PATH=/usr/java/jdk1.8.0_181-cloudera/jre/lib/amd64/server/libjvm.so
SHELL=/bin/bash

I have set my env vars as -

[root@n138 ~]# echo $NHDFS_LIBJVM_PATH

/usr/java/jdk1.8.0_181-cloudera/jre/lib/amd64/server/libjvm.so

[root@n138 ~]# echo $NHDFS_LIBHDFS_PATH

/opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib64/libhdfs.so

export CLASSPATH=$(hadoop classpath)

[root@n138 ~]# ldd /opt/cloudera/parcels/CDH-6.3.4-1.cdh6.3.4.p0.6751098/lib64/libhdfs.so
linux-vdso.so.1 => (0x00007ffecb589000)
libjvm.so => /lib64/libjvm.so (0x00007f27987c6000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f27985c2000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f27983a6000)
libc.so.6 => /lib64/libc.so.6 (0x00007f2797fd8000)
/lib64/ld-linux-x86-64.so.2 (0x00007f27999d9000)
libm.so.6 => /lib64/libm.so.6 (0x00007f2797cd6000)
[root@n138 ~]#

what is wrong with my setup ? why I am unable to to load libhdfs.so ?

Re: FileSystem: loadFileSystems failed error

sky1010 — Tue, 05 Oct 2021 11:15:45 GMT

The problem was my classpath.

export CLASSPATH=$(hadoop classpath --glob)

wild cards not supported with Hadoop 3.0.0-cdh6.3.4

question Re: FileSystem: loadFileSystems failed error in Support Questions

FileSystem: loadFileSystems failed error

Re: FileSystem: loadFileSystems failed error