Support Questions

Find answers, ask questions, and share your expertise

libhdfs problems

avatar

So a few months ago I posted this question:

https://community.hortonworks.com/questions/222259/libhdfs-missing.html

But I realized I did a terrible job posting the question, and consequently, I didn't really get the help I was hoping for.

So attempt #2:

I am running HDP 3.0.1 on Ubuntu 16.04 nodes. We initially installed 3.0.0, and successfully upgraded later. The cluster itself is running fine and Ambari is a great administration tool. However...

libhdfs.so.0.0.0 does not exist in our cluster.

root@use1-mon-flink-1:~# dpkg -l | grep libhdfs
ii  libhdfs0                                       3.1.1.3.0.1.0-187                          all          libhdfs0 is a virtual package that brings libhdfs0-3-0-1-0-187 as a dependency.
ii  libhdfs0-3-0-1-0-187                           3.1.1.3.0.1.0-187                          amd64        Hadoop Filesystem Library
root@use1-mon-flink-1:~# ls -larth /usr/hdp/3.0.1.0-187/usr/lib/
total 8.0K
lrwxrwxrwx 1 root root   16 Sep 19 11:31 libhdfs.so -> libhdfs.so.0.0.0
drwxr-xr-x 4 root root 4.0K Oct 11 14:32 ..
drwxr-xr-x 2 root root 4.0K Oct 11 14:32 .
root@use1-mon-flink-1:~# find /usr -name "libhdfs.so.0.0.0"
root@use1-mon-flink-1:~# 
root@use1-mon-flink-1:~# apt search libhdfs
Sorting... Done
Full Text Search... Done
libhdfs0/unknown,now 3.1.1.3.0.1.0-187 all [installed]
  libhdfs0 is a virtual package that brings libhdfs0-3-0-1-0-187 as a dependency.

libhdfs0-3-0-1-0-187/unknown,now 3.1.1.3.0.1.0-187 amd64 [installed]
  Hadoop Filesystem Library

root@use1-mon-flink-1:~# apt install --reinstall libhdfs*
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'libhdfs0' for glob 'libhdfs*'
Note, selecting 'libhdfs0-3-0-1-0-187' for glob 'libhdfs*'
0 upgraded, 0 newly installed, 2 reinstalled, 0 to remove and 2 not upgraded.
Need to get 1002 B/2412 B of archives.
After this operation, 0 B of additional disk space will be used.
Get:1 http://public-repo-1.hortonworks.com/HDP/ubuntu16/3.x/updates/3.0.1.0 HDP/main amd64 libhdfs0 all 3.1.1.3.0.1.0-187 [1002 B]
Fetched 1002 B in 0s (6586 B/s)    
[master 90d1838] saving uncommitted changes in /etc prior to apt run
 3 files changed, 57 insertions(+), 2 deletions(-)
 rewrite oozie/3.0.1.0-187/0/oozie-site.jceks (64%)
(Reading database ... 132397 files and directories currently installed.)
Preparing to unpack .../libhdfs0_3.1.1.3.0.1.0-187_all.deb ...
Unpacking libhdfs0 (3.1.1.3.0.1.0-187) over (3.1.1.3.0.1.0-187) ...
Preparing to unpack .../libhdfs0-3-0-1-0-187_3.1.1.3.0.1.0-187_amd64.deb ...
Unpacking libhdfs0-3-0-1-0-187 (3.1.1.3.0.1.0-187) over (3.1.1.3.0.1.0-187) ...
Setting up libhdfs0-3-0-1-0-187 (3.1.1.3.0.1.0-187) ...
Setting up libhdfs0 (3.1.1.3.0.1.0-187) ...

root@use1-mon-flink-1:~# ls -larth /usr/hdp/3.0.1.0-187/usr/lib/
total 8.0K
lrwxrwxrwx 1 root root   16 Sep 19 11:31 libhdfs.so -> libhdfs.so.0.0.0
drwxr-xr-x 4 root root 4.0K Oct 11 14:32 ..
drwxr-xr-x 2 root root 4.0K Jan  7 15:00 .
root@use1-mon-flink-1:~# 

As you can see, even installing the Hortonworks package that should install libhdfs doesn't install the actual .so file. Just a dead symlink.

Has anyone else experienced this problem?

Note: I can also reproduce this problem on a fresh Ubuntu 16.04 VM. The initial install of libhdfs0 and libhdfs0-3-0-1-0-187 don't install an actual .so file.

3 REPLIES 3

avatar

So having now installed a 3.1.0 cluster on Ubuntu, I see this package still doesn't install an actual .so file for libhdfs.

root@use1-monstage-1:~# find /usr -name libhdfs*
/usr/share/doc/libhdfs0-3-1-0-0-78
/usr/lib/ams-hbase/lib/hadoop-native/libhdfs.so
/usr/lib/ams-hbase/lib/hadoop-native/libhdfs.a
/usr/hdp/3.1.0.0-78/hadoop/lib/native/libhdfs.a
/usr/hdp/3.1.0.0-78/usr/lib/libhdfs.so
root@use1-monstage-1:~# ls -larth /usr/hdp/3.1.0.0-78/usr/lib/libhdfs.so
lrwxrwxrwx 1 root root 16 Dec  6 13:58 /usr/hdp/3.1.0.0-78/usr/lib/libhdfs.so -> libhdfs.so.0.0.0
root@use1-monstage-1:~# ls -larth /usr/lib/ams-hbase/lib/hadoop-native/libhdfs.so
-rwxr-xr-x 1 root root 292K Dec  7 18:13 /usr/lib/ams-hbase/lib/hadoop-native/libhdfs.so

The ams-hbase package seems to have an actual libhdfs.so file, but trying to link against that doesn't work correctly.

Is there any chance we'll get libhdfs properly packaged for HDP?

avatar
Contributor

I have the same problem on HDP 3.1.4.0-315

What is the solution ?

avatar
Contributor