Member since
04-12-2016
7
Posts
3
Kudos Received
0
Solutions
05-28-2016
01:17 AM
3 Kudos
If this problem happens a lot,I mean you always need know the mapping from file operations (create, delete, rename etc) to upper level applications, I think you can suggest users use caller context feature, which was released to HDP 2.2 and up. The feature introduces a new setting hadoop.caller.context.enabled . When set to additional fields are written into namenode audit log records to help identify the job or query that introduced each NameNode operation. This feature is enabled by default starting with this release of HDP. New Behavior: This feature brings a new key-value pair at the end of each audit log record. The newly added key at is callerContext , value context:signature . The overall format would be callerContext=context:signature. If the signature is null or empty, the value will be context only, in the format of callerContext=context . If the hadoop.caller.context.enabled config key is false, the key-value pair will not be showing. The audit log format is not changed in this case. It is also possible to limit the maximum length of context and signature. Consider the hadoop.caller.context.max.size config key (default 128 bytes) and hadoop.caller.context.signature.max.size (default 40 bytes) config key respectively. There is a chance that the new information in the audit log may break existing scripts/automation that was being used to analyze the audit log. In this case the scripts may need to be fixed. We do not recommend disabling this feature as it can be a useful troubleshooting aid. Please refer to release notes.
... View more