Reply
Explorer
Posts: 7
Registered: ‎08-24-2017

cloudera-scm-agent corrupt JSON error

[ Edited ]

We are using Cloudera 5.10.1 enterprise.  We have Cloudera Navigator setup and generally it's working and audit events are getting to the Cloudera Navigator Audit Server and are published to Kafka.

 

For HUE and HIVE audit log files (ie. /var/log/hue/audit and /var/log/hive/audit) cloudera-scm-agent on that node tries to consume the audit log files but gets an error for some of the entries and stops processor the audit files.

 

I see an error like this...

 

cloudera-scm-agent.log:... Audit-Plugin navigator_logs ERROR

  Stopping processing of hue-HUE_SERVER-... because of corrupt navigator log line:

{"username":"user1",..."operation":"USER_LOGOUT",{"username":"user1",...}

 

Essentially it looks like 2 JSON events are being combined and the 1st event isn't completely written out (ie. there is no '}' ) before the next event starts (ie. '{').

 

I tried deleted the /var/log/hue/audit files and restarting the service but it just occurs again.  Any ideas?

 

Also I looked at the timestamps on the audit events causing the issues and they are old.  So I'm unclear why deleting the /var/log/hue/audit files and restarting caused the audit events to be regenerated with the old timestamps.

 

Maybe to work around this issue there is a cloudera-scm-agent options to skip those corrupt lines and continue?

 

Richard

 

Announcements
New solutions