I have been trying to figure the cause of missing logs on the jobhistory server for completed mapred job yarn containers. The problem only seems to arise when a single nodemanager is running many containers (12+); when running fewer containers this problem doesn't come up. The nodemanager has 13 data disks and that's where the yarn container local/log data get's stored. Log aggregation indicates it completes successfully and the aggregated log file looks correct (i was able to read it using LogAggregationIndexedFileController) but in the jobhistory server i see this message when trying to look at a completed map tasks logs: 2018-07-13 00:48:34,244 WARN webapp.View (IndexedFileAggregatedLogsBlock.java:render(139)) - Can not load log meta from the log file:hdfs://hadoopnn1.net:8020/app-logs/rana/logs-ifile/application_1530921118753_0010/hadoopdn4.net_45454 Current log aggregation settings are: "yarn.log-aggregation.file-formats" : "IndexedFormat,TFile", "yarn.nodemanager.log-aggregation.debug-enabled" : "false", "yarn.log-aggregation.retain-seconds" : "2592000", "yarn.nodemanager.log-aggregation.num-log-files-per-app" : "336", "yarn.log-aggregation.file-controller.IndexedFormat.class" : "org.apache.hadoop.yarn.logaggregation.filecontroller.ifile.LogAggregationIndexedFileController", "yarn.log-aggregation-enable" : "true", "yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds" : "3600", "yarn.log-aggregation.file-controller.TFile.class" : "org.apache.hadoop.yarn.logaggregation.filecontroller.tfile.LogAggregationTFileController", "yarn.nodemanager.log-aggregation.compression-type" : "gz", Has anyone seen this before? ... View more

I am trying to create a generic policy for access to hbase tables. I want to give all users the ability to create tables TABLE_* but only owners access to read and write from it. For the create i would give {USER} permission and for read write i want to give {OWNER} permission and when the table is created i set the OWNER in the table meta info (METADATA => {'OWNER' => 'rana'}) as specifed in http://hbase.apache.org/0.94/book/hbase.accesscontrol.configuration.html. So far I still get DENIED permission as seen in ranger audit for the scannerOpen action for the same user set as the table owner. Is this not supported? ... View more