Reply
Explorer
Posts: 17
Registered: ‎08-04-2017

Impala Daemon Audit Pipeline Test failed

Hi,

 

Impala Daemon health check failed due Audit pipeline test failure. Four out of six nodes failed with same error.

 

[13/Sep/2018 16:14:53 +0000] 13226 Monitor-HostMonitor throttling_logger ERROR (30 skipped) Could not find local file system for /var/run/cloudera-scm-agent/process
[13/Sep/2018 16:15:51 +0000] 13226 Audit-Plugin throttling_logger ERROR (313 skipped) Error occurred when sending entry to audit server

 

I'm able to see user queries and other audit information in /var/log/impalad/audit 

 

Users are experiencing performance issues using Impala. Please advice.

 

Thanks,

Meher

Explorer
Posts: 17
Registered: ‎08-04-2017

Re: Impala Daemon Audit Pipeline Test failed

Found this error message in Navigator audit server log

 

2018-09-14 10:49:51,987 WARN org.hibernate.engine.jdbc.spi.SqlExceptionHelper [NavigatorServer-2655]: SQL Error: 1366, SQLState: HY000
2018-09-14 10:49:51,987 ERROR org.hibernate.engine.jdbc.spi.SqlExceptionHelper [NavigatorServer-2655]: Incorrect string value: '\xF0\x9F\x87\xBA\xF0\x9F...' for column 'OPERATION_TEXT' at row 1
2018-09-14 10:49:52,018 WARN com.cloudera.navigator.NavigatorIPCImpl [NavigatorServer-2655]: Error persisting events for service impala

 

Is this a known defect or bug in Impala where user query is unable to insert into operation_text column of audit database table?

 

 

 

Cloudera Employee
Posts: 437
Registered: ‎07-29-2015

Re: Impala Daemon Audit Pipeline Test failed

I'd suggest asking in the Cloudera Manager forum - it looks like an issue with some of the CM/Navigator infrastructure rather than anything in Impala itself. I wouldn't expect something like this to affect performance of Impala queries - impala just writes out the logs, everything beyond that is handled by CM/Navigator

Highlighted
Explorer
Posts: 17
Registered: ‎08-04-2017

Re: Impala Daemon Audit Pipeline Test failed

Thank you Tim. I will post it in Cloudera Navigator group as the below errors showed up in Navigator server logs:

 

mysql database characterset is currently set to utf8. It used to have latin1 before. Changed characterset but still the issue persists.

 

Caused by: java.sql.BatchUpdateException: Incorrect string value: '\xF0\x9F\x87\xBA\xF0\x9F...' for column 'OPERATION_TEXT' at row 1
Caused by: java.sql.BatchUpdateException: Incorrect string value: '\xF0\x9F\x92\x98' ...' for column 'OPERATION_TEXT' at row 1
Caused by: java.sql.BatchUpdateException: Incorrect string value: '\xF0\x9F\x93\x9D, ...' for column 'OPERATION_TEXT' at row 1