Enable Intel's Intelligent Storage Acceleration Library for Erasure Coding

[#1]

If you have erasure coded some directory and perform some operations on the directory you might have observed WARN messages like below

WARN erasurecode.ErasureCodeNative: Loading ISA-L failed: Failed to load libisal.so.2 (libisal.so.2: cannot open shared object file: No such file or directory)
WARN erasurecode.ErasureCodeNative: ISA-L support is not available in your platform... using builtin-java codec where applicable

This WARN messages are due to ISA Library not being present on the node.

Below are the steps to enable the library

1) Clone the isa-l github repository.

# git clone https://github.com/01org/isa-l.git

2) Go to the cloned directory

# cd isa-l

3) Install yasm if you do not have it already

# yum install -y yasm  ---> centOS
# apt-get install yasm  ----> ubuntu

4) Build the library

# make -f Makefile.unx

5) Copy the library files to lib directory

# cp bin/libisal.so bin/libisal.so.2 /lib64

6) Verify that isa-l library is enabled properly

# hadoop checknative
Expected output
18/10/12 10:20:03 INFO bzip2.Bzip2Factory: Successfully loaded & initialized native-bzip2 library system-native
18/10/12 10:20:03 INFO zlib.ZlibFactory: Successfully loaded & initialized native-zlib library
Native library checking:
hadoop:  true /usr/hdp/3.0.0.0-1634/hadoop/lib/native/libhadoop.so.1.0.0
zlib:    true /lib64/libz.so.1
zstd  :  false
snappy:  true /usr/hdp/3.0.0.0-1634/hadoop/lib/native/libsnappy.so.1
lz4:     true revision:10301
bzip2:   true /lib64/libbz2.so.1
openssl: true /lib64/libcrypto.so
ISA-L:   true /lib64/libisal.so.2     ------------->  Shows that ISA-L is loaded.

If step 6 uses /usr/lib64 directory instead of /lib64, you need copy the .so files in Step 5 to /usr/lib64 directory. Perform the steps on all datanode and namenode hosts or copy the .so files from the above node to /lib64 directories of all other nodes.

.

Hope this helps 🙂

[#2]

Tested in my cluster as well and was able to add ISA-L libs and remove the WARN messages.

[#3]

hello I have the errors mentioned in the description and I am executing the procedure but I get the following errors in 

make -f Makefile.unx

[root@emkioqlnclo01 isa-l]# make -f Makefile.unx
---> Building erasure_code/gf_vect_mul_sse.asm x86_64
---> Building erasure_code/gf_vect_mul_avx.asm x86_64
---> Building erasure_code/gf_vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_2vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_3vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_4vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_5vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_6vect_dot_prod_sse.asm x86_64
---> Building erasure_code/gf_2vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_3vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_4vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_5vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_6vect_dot_prod_avx.asm x86_64
---> Building erasure_code/gf_2vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_3vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_4vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_5vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_6vect_dot_prod_avx2.asm x86_64
---> Building erasure_code/gf_vect_mad_sse.asm x86_64
---> Building erasure_code/gf_2vect_mad_sse.asm x86_64
---> Building erasure_code/gf_3vect_mad_sse.asm x86_64
---> Building erasure_code/gf_4vect_mad_sse.asm x86_64
---> Building erasure_code/gf_5vect_mad_sse.asm x86_64
---> Building erasure_code/gf_6vect_mad_sse.asm x86_64
---> Building erasure_code/gf_vect_mad_avx.asm x86_64
---> Building erasure_code/gf_2vect_mad_avx.asm x86_64
---> Building erasure_code/gf_3vect_mad_avx.asm x86_64
---> Building erasure_code/gf_4vect_mad_avx.asm x86_64
---> Building erasure_code/gf_5vect_mad_avx.asm x86_64
---> Building erasure_code/gf_6vect_mad_avx.asm x86_64
---> Building erasure_code/gf_vect_mad_avx2.asm x86_64
---> Building erasure_code/gf_2vect_mad_avx2.asm x86_64
---> Building erasure_code/gf_3vect_mad_avx2.asm x86_64
---> Building erasure_code/gf_4vect_mad_avx2.asm x86_64
---> Building erasure_code/gf_5vect_mad_avx2.asm x86_64
---> Building erasure_code/gf_6vect_mad_avx2.asm x86_64
---> Building erasure_code/ec_multibinary.asm x86_64
multibinary.asm:283: error: expression syntax error
multibinary.asm:359: error: expression syntax error
make: *** [bin/ec_multibinary.o] Error 1