Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Super Guru

If you have erasure coded some directory and perform some operations on the directory you might have observed WARN messages like below

WARN erasurecode.ErasureCodeNative: Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)
WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable

This WARN messages are due to ISA Library not being present on the node.

Below are the steps to enable the library

1) Clone the isa-l github repository.

# git clone https://github.com/01org/isa-l.git

2) Go to the cloned directory

# cd isa-l

3) Install yasm if you do not have it already

# yum install -y yasm  ---> centOS
# apt-get install yasm  ----> ubuntu

4) Build the library

# make -f Makefile.unx

5) Copy the library files to lib directory

# cp bin/libisal.so bin/libisal.so.2 /lib64

6) Verify that isa-l library is enabled properly

# hadoop checknative
Expected output
18/10/12 10:20:03 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
18/10/12 10:20:03 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/hdp/3.0.0.0-1634/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false
snappy:  true /usr/hdp/3.0.0.0-1634/hadoop/lib/native/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   true /lib64/libisal.so.2     ------------->  Shows that ISA-L is loaded.

If step 6 uses /usr/lib64 directory instead of /lib64, you need copy the .so files in Step 5 to /usr/lib64 directory. Perform the steps on all datanode and namenode hosts or copy the .so files from the above node to /lib64 directories of all other nodes.

.

Hope this helps 🙂

4,368 Views
Comments

Tested in my cluster as well and was able to add ISA-L libs and remove the WARN messages.

hello I have the errors mentioned in the description and I am executing the procedure but I get the following errors in 

make -f Makefile.unx

[root@emkioqlnclo01 isa-l]# make -f Makefile.unx
---> Building erasure_code/gf_vect_mul_sse.asm x86_64
---> Building erasure_code/gf_vect_mul_avx.asm x86_64
---> Building erasure_code/gf_vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_2vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_3vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_4vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_5vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_6vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_2vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_3vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_4vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_5vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_6vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_2vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_3vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_4vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_5vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_6vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_vect_mad_sse.asm x86_64
---> Building erasure_code/gf_2vect_mad_sse.asm x86_64
---> Building erasure_code/gf_3vect_mad_sse.asm x86_64
---> Building erasure_code/gf_4vect_mad_sse.asm x86_64
---> Building erasure_code/gf_5vect_mad_sse.asm x86_64
---> Building erasure_code/gf_6vect_mad_sse.asm x86_64
---> Building erasure_code/gf_vect_mad_avx.asm x86_64
---> Building erasure_code/gf_2vect_mad_avx.asm x86_64
---> Building erasure_code/gf_3vect_mad_avx.asm x86_64
---> Building erasure_code/gf_4vect_mad_avx.asm x86_64
---> Building erasure_code/gf_5vect_mad_avx.asm x86_64
---> Building erasure_code/gf_6vect_mad_avx.asm x86_64
---> Building erasure_code/gf_vect_mad_avx2.asm x86_64
---> Building erasure_code/gf_2vect_mad_avx2.asm x86_64
---> Building erasure_code/gf_3vect_mad_avx2.asm x86_64
---> Building erasure_code/gf_4vect_mad_avx2.asm x86_64
---> Building erasure_code/gf_5vect_mad_avx2.asm x86_64
---> Building erasure_code/gf_6vect_mad_avx2.asm x86_64
---> Building erasure_code/ec_multibinary.asm x86_64
multibinary.asm:283: error: expression syntax error
multibinary.asm:359: error: expression syntax error
make: *** [bin/ec_multibinary.o] Error 1
 

webinar banner
Version history
Last update:
‎10-16-2018 03:27 AM
Updated by:
Contributors