Classpath for native libraries CM5 / CDH4.7 parcel -- for accumulo-site.xml

[#1]

So I've been playing around with Accumulo 1.5.1 on a CM5 / CDH4.7 parcel based install    (no issues on the RPM install) - and I'm unable to seem to get the native libraries on the classpath (should be straightforward, but it's not working).

I'm assuming this is a cloudera  parcel issue, since it works just fine with the RPM distribution.

My accumulo classpath looks like

<value>
  $ACCUMULO_HOME/lib/accumulo-server.jar,
  $ACCUMULO_HOME/lib/accumulo-core.jar,
  $ACCUMULO_HOME/lib/accumulo-start.jar,
  $ACCUMULO_HOME/lib/accumulo-fate.jar,
  $ACCUMULO_HOME/lib/accumulo-proxy.jar,
  $ACCUMULO_HOME/lib/[^.].*.jar,
  $ZOOKEEPER_HOME/zookeeper[^.].*.jar,
  $HADOOP_CONF_DIR,
  /usr/share/cmf/lib/cdh5/*.jar,
  $HADOOP_PREFIX/lib/hadoop/[^.].*.jar,
  $HADOOP_PREFIX/lib/hadoop/lib/[^.].*.jar,
  $HADOOP_PREFIX/lib/hadoop-hdfs/[^.].*.jar,
  $HADOOP_PREFIX/lib/hadoop-mapreduce/[^.].*.jar,
  $HADOOP_PREFIX/lib/hadoop-yarn/[^.].*.jar,
  $HADOOP_PREFIX/lib/hadoop/lib/native/*,$HADOOP_PREFIX/[^.].*.jar,
  $HADOOP_PREFIX/lib/[^.].*.jar,
</value>

(yep, I'll clean it up a bit later - first just trying to get everything on the classpath) 

Where:

$HADOOP_PREFIX=/opt/cloudera/parcels/CDH
$HADOOP_CONF_DIR=/etc/hadoop/conf
$ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH/lib/zookeeper

And I added:

export LD_LIBRARY_PATH=${HADOOP_PREFIX}/lib/hadoop/lib/native:${LD_LIBRARY_PATH}

Everything works fine, ran through the random walk test for 2 days - but I can't get the native libraries to load.  

I realize these are the types of things that would be run down when it's supported - but if someone could toss me a bone it would be much appreciated.

[#2 · Accepted solution]

In 1.5.x, the Accumulo scripts rely on HADOOP_PREFIX to load the hadoop native libraries. They ignore LD_LIBRARY_PATH entirely.

On a parcel install you should do something like the example we ship with 1.4.4-cdh4.5.0

accumulo-env.sh:

if [ -z "$HADOOP_HOME" ]
then
    test -z "$HADOOP_PREFIX" && export HADOOP_PREFIX=/opt/cloudera/parcels/CDH/lib/hadoop
    export HADOOP_HOME=$HADOOP_PREFIX
else
    HADOOP_PREFIX="$HADOOP_HOME"
fi

test -z "$HADOOP_CLIENT_HOME"    && export HADOOP_CLIENT_HOME=/opt/cloudera/parcels/CDH/lib/hadoop/client-0.20
test -z "$HADOOP_CONF_DIR"       && export HADOOP_CONF_DIR="$HADOOP_PREFIX/etc/hadoop"

test -z "$ZOOKEEPER_HOME"        && export ZOOKEEPER_HOME=/opt/cloudera/parcels/CDH/lib/zookeeper

accumulo-site.xml:

   <property>
      <name>general.classpaths</name>
      <value>
        $ACCUMULO_HOME/lib/[^.].*.jar,
        $ZOOKEEPER_HOME/zookeeper[^.].*-[0-9].*.jar,
        $HADOOP_CONF_DIR,
        $HADOOP_CLIENT_HOME/[^.].*-[0-9].*.jar,
        $HADOOP_PREFIX/lib/[^.].*.jar,
      </value>
      <description>Classpaths that accumulo checks for updates and class files.
      When using the Security Manager, please remove the ".../target/classes/" values.
      </description>
    </property>

In particular, note that HADOOP_PREFIX points at the lib dir for Hadoop within the CDH parcel, rather than the top level CDH dir.

That last line in the classpath for HADOOP_PREFIX/lib is just to get around ACCUMULO-2301 / ACCUMULO-2786. So you could further limit it to just be the jars needed to satisfy the jetty dependency for the Monitor.

If you're doing this as a new install, you're better off just running 1.6.0. It updated how native library loading was done to more closely match the expectations of Hadoop (and just general native code loading), so it goes much smoother with parcels.

[#3]

Thanks, that works - and make sense.   In retrospect should have thought about looking at what you all shipped with 1.4.x.

Roger on 1.6.0 - I have that up and running (without issue) - I was trying to generate a puppet config for 1.5.x so I could validate against that as well - and obvioulsy went off on a tangent with the parcel scheme and the prefix directory that, I think, sent me down a spiral of doom.

Anyway, much appreciated!