CM separates server-side configs and client-side configs. The specific property of "yarn.log-aggregation-enable" is only used by NodeManagers as a toggle. Clients do not use the above property, so they aren't found in the regular /etc/hadoop/conf/*.xml path configs (these are gateway configs, or client-configs). When your Java class runs from within a YARN container, it also inherits the parent environment (parent being the NodeManager here). Thereby, it sees the same paths the NM service does, which would explain your output of true when that happens. YARN, speaking on its role as an application framework, provides no notion of 'application configuration', and expects custom applications to roll their own solution. For example, the MR2 app uses the concept of a 'job.xml' written and sent from clients that is then utilised as the single source of configuration truth (rather than sourced from the environment from the random NM the AM/Tasks may run on). Does this help resolve the confusion? ... View more

You (or your used software) appear to be using appends on files that are being modified in parallel by other concurrent jobs/workflows/etc.. HDFS uses a single-writer model for its files, so observing this error is normal if your software does not have logic handling it and waiting for a proper writer lease to perform its work. Without audit logs of the filenames involved, there's little more we can tell. We also advise against using appends unless you absolutely require it for your use-cases. ... View more

The last specified -Xmx takes precedence, normally, in Oracle/Sun JREs. What evidence in the failure logs is suggesting that the mapper starts only with 200m heap instead? ... View more