Can not start Nodemanager in Kerberos secure mode

[#1]

I have a 1-node kerberized Hadoop cluster on Centos 6.6. Starting Nodemanager fails with this error

container-executor: /lib64/libc.so.6: version `GLIBC_2.14' not found

Ok, I did a little bit research and found that on RHEL 6 glibc 2.12 is the highest supported version. However it should be possible to run glibc 2.14 along with 2.12 according to this stackexchange post. But still I get the same error. It seams that the binary `container-executor` does not use the env variable `LD_LIBRARY_PATH` or the installation of glibc 2.14 failed. How can I verify the installation? How can I start nodemanager in kerberos secured mode?

[#2]

@Andreas Knapp

- Any attempt of upgrading the "glibc" version might affect many other components installed at the OS level as well as the hadoop services. Hence it is not usually advised to upgrade the "glibc" package downloaded from other places apart from the one that is shipped with the OS.

- The combination of later version of glibc with the existing packages have not been tested, although most of the programs may be able work because of backward compatibility.

- The HDP components are also tested/certified on these RHEL/Centos releases which contains the "glibc" versions shipped along with the distribution.

- So if you really want to use the higher version of "glibc" then you should try using the RHEL7 versions that includes higher version of glibc.

.

[#3]

@Jay SenSharma, actually I don't really want to upgrade glibc, I just want to run nodemanager in a Kerberos secured mode on CentOS 6.6. Upgrading glibc was just an assumption from myself to solve this issue.

I assume there should be another solution to run Nodemanager with kerberos enabled on CentOS 6.6?

For the sake of completeness here is my yarn-site.xml and container-executor.cfg.

This is my yarn-site.xml

<configuration>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
      <name>yarn.application.classpath</name>
      <value>/usr/local/hadoop/etc/hadoop, /usr/local/hadoop/share/hadoop/common/*, /usr/local/hadoop/share/hadoop/common/lib/*, /usr/local/hadoop/share/hadoop/hdfs/*, /usr/local/hadoop/share/hadoop/hdfs/lib/*, /usr/local/hadoop/share/hadoop/mapreduce/*, /usr/local/hadoop/share/hadoop/mapreduce/lib/*, /usr/local/hadoop/share/hadoop/yarn/*, /usr/local/hadoop/share/hadoop/yarn/lib/*</value>
    </property>
    <property>
    <description>
      Number of seconds after an application finishes before the nodemanager's
      DeletionService will delete the application's localized file directory
      and log directory.


      To diagnose Yarn application problems, set this property's value large
      enough (for example, to 600 = 10 minutes) to permit examination of these
      directories. After changing the property's value, you must restart the
      nodemanager in order for it to have an effect.


      The roots of Yarn applications' work directories is configurable with
      the yarn.nodemanager.local-dirs property (see below), and the roots
      of the Yarn applications' log directories is configurable with the
      yarn.nodemanager.log-dirs property (see also below).
    </description>
    <name>yarn.nodemanager.delete.debug-delay-sec</name>
    <value>600</value>
  </property>
  <property>
     <name>yarn.resourcemanager.principal</name>
     <value>rm/HOSTNAME@EXAMPLE.COM</value>
  </property>
  <property>
       <name>yarn.resourcemanager.keytab</name>
       <value>/etc/security/keytabs/rm.service.keytab</value>
  </property>
  <property>
       <name>yarn.nodemanager.principal</name>
       <value>nm/HOSTNAME@EXAMPLE.COM</value>
  </property>
  <property>
       <name>yarn.nodemanager.keytab</name>
       <value>/etc/security/keytabs/nm.service.keytab</value>
  </property>
  <property>
       <name>yarn.nodemanager.container-executor.class</name>
       <value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
  </property>
  <property>
       <name>yarn.nodemanager.linux-container-executor.path</name>
       <value>/usr/local/hadoop/bin/container-executor</value>
  </property>
  <property>
       <name>yarn.nodemanager.linux-container-executor.group</name>
       <value>root</value>
  </property>
  <property>
       <name>yarn.timeline-service.principal</name>
       <value>yarn/HOSTNAME@EXAMPLE.COM</value>
  </property>
  <property>
       <name>yarn.timeline-service.keytab</name>
       <value>/etc/security/keytabs/yarn.service.keytab</value>
  </property>
  <property>
       <name>yarn.resourcemanager.webapp.delegation-token-auth-filter.enabled</name>
       <value>true</value>
  </property>
  <property>
       <name>yarn.timeline-service.http-authentication.type</name>
       <value>kerberos</value>
  </property>
  <property>
       <name>yarn.timeline-service.http-authentication.kerberos.principal</name>
       <value>HTTP/HOSTNAME@EXAMPLE.COM</value>
  </property>
  <property>
       <name>yarn.timeline-service.http-authentication.kerberos.keytab</name>
       <value>/etc/security/keytabs/yarn.service.keytab</value>
  </property>
</configuration>

The container-executor.cfg looks like this

yarn.nodemanager.local-dirs=/usr/local/hadoop/nodemanager
yarn.nodemanager.linux-container-executor.group=root
yarn.nodemanager.log-dirs=/usr/local/hadoop/nodemanager
banned.users=bin	
min.user.id=1000

Am I missing something?

Could it be that I have a wrong container-executor binary? the binary seems to be compiled for RHEL7 but I should have one for RHEL6. Can I download such a binary from somewhere manually?

[#4 · Accepted solution]

I finally solved it by myself by re-building the container-executor binary from the source code manually.

This post explains how to do that.

Everything can also be seen in the Dockerfile of my project of a single node kerberized hadoop cluster using docker on Github and Dockerhub.