Reply
New Contributor
Posts: 4
Registered: ‎05-14-2015

Yarn container configuration not accessible

My cluster is setup with org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor as the container executor. This is a kerberos enabled cluster integrated with Active Directory (AD). I have a yarn application that tries to read the site.xml files from HADOOP_CONF_DIR that is made available to the container.

For instance, this folder looks like /var/run/cloudera-scm-agent/process/36079-yarn-NODEMANAGER

My application is not able to read this container and get an access denied exception.

 

at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.7.0_67]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.7.0_67]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.7.0_67]
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:426) ~[?:1.7.0_67]
at java.nio.file.Files.newDirectoryStream(Files.java:481) ~[?:1.7.0_67]

 

I'm running the application as a AD user.

 

Any idea what might be the issue here?

Cloudera Employee
Posts: 314
Registered: ‎01-16-2014

Re: Yarn container configuration not accessible

Please always provide the CDH version and if you use it the CM version.

I can see that you use CM based on that path. Can you tell me if you enabled kerberos through the wizard or manually?

 

If you ran a simple Pi exampe job does it work or does that fail also?

 

For this failure can you provide a full stack trace. There is information missing and I would like to see where the exception is thrown from.

 

Wilfred

New Contributor
Posts: 4
Registered: ‎05-14-2015

Re: Yarn container configuration not accessible

Thank you for the response.

 

Please find my answers inline.

 

Please always provide the CDH version and if you use it the CM version.

This cluster is running CDH 5.4 with the latest CM that comes with that version

 

I can see that you use CM based on that path. Can you tell me if you enabled kerberos through the wizard or manually?

Kerberos was enabled through the wizard and has AD integration.

 

If you ran a simple Pi exampe job does it work or does that fail also?

Simple Pi example works fine without any issue.

 

For this failure can you provide a full stack trace. There is information missing and I would like to see where the exception is thrown from.

 

This is the stacktrace from the yarn application.

—”hëµ◊∂9flA@í∫·P VERSIONAPPLICATION_ACLAPPLICATION_OWNERdhdpsnplgcbtch.,container_e2383_1430952168731_1853_02_000001®ZXstderr1656SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hdfs14/yarn/nm/usercache/dhdpsnplgcbtch/appcache/application_1430952168731_1853/container_e2383_1430952168731_1853_02_000001/tmp/snaplogic/run/lib/yam/log4j-slf4j-impl-2.0-rc1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hdfs14/yarn/nm/usercache/dhdpsnplgcbtch/appcache/application_1430952168731_1853/container_e2383_1430952168731_1853_02_000001/tmp/snaplogic/run/lib/yam/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.Log4jLoggerFactory]
2015-05-07T15:19:53,396 E main YarnUtil.java [ ] Unable to read yarn configuration files java.nio.file.AccessDeniedException: /var/run/cloudera-scm-agent/process/36077-yarn-NODEMANAGER
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84) ~[?:1.7.0_67]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) ~[?:1.7.0_67]
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) ~[?:1.7.0_67]
at sun.nio.fs.UnixFileSystemProvider.newDirectoryStream(UnixFileSystemProvider.java:426) ~[?:1.7.0_67]
at java.nio.file.Files.newDirectoryStream(Files.java:481) ~[?:1.7.0_67]
at com.snaplogic.util.YarnUtil.getYarnConfiguration(YarnUtil.java:123) ~[jcommon-4.0.jar:?]
at com.snaplogic.cc.YplexManager.main(YplexManager.java:147) ~[jcc-4.0.jar:?]
at com.snaplogic.common.ServerTrampoline.main(ServerTrampoline.java:289) [jcc-4.0-mrc177-snapreduce.war:?]

 

Cloudera Employee
Posts: 314
Registered: ‎01-16-2014

Re: Yarn container configuration not accessible

Based on the fact that the Pi example works, YARN works and it is something that is being done in the application code.

 

In a secured cluster the container does not run as the same user as the Node Manager. Normally the node manager runs as the user YARN and the container as the user whom started the application. The configuration directory that you are trying to access is the configuration directory of the service not for the container. The container gets the configuration passed in. You should not be accessing the service configuration as it most likely differs from the container configuration.

There are a lot of settings that are not relevant for a service in the configuration and thus are not set or set to the hadoop default. If they exist they normally are ignored and read from the configuration that was created by the application on submission. The application should never accesses those files.

 

Wilfred

New Contributor
Posts: 4
Registered: ‎05-14-2015

Re: Yarn container configuration not accessible

If YARN_CONF_DIR is not the config directory for the application, where else would the application get the hadoop cluster configuration like fs.defaultFS, mapreduce job tracker address, etc?
Posts: 1,896
Kudos: 433
Solutions: 303
Registered: ‎07-31-2013

Re: Yarn container configuration not accessible

In the MR2 YARN application, the client writes a XML file carrying the
cluster configuration and sends it as a localised file. This pattern
needs to be followed for all applications that rely on cluster
configuration - they must be sent by the client.

Announcements