question Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service) in Archives of Support Questions (Read Only)

Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

EduardBangga — Mon, 29 Jan 2018 07:15:29 GMT

Hi All,

I have some issue with Navigator Audit Server because the Cloudera Agent can't send audit to audit server

This is the log from cloudera agent

[29/Jan/2018 13:37:49 +0000] 127451 Audit-Plugin throttling_logger ERROR    (2 skipped) Unable to send message to audit server for hdfs service.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/audit/navigator_sender.py", line 98, in _send_avro_data
    response = self._requestor.request(ipc_request_name, dict(request=request))
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 482, in transceive
    self.write_framed_message(request)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 501, in write_framed_message
    self.conn.request(req_method, self.req_resource, req_body, req_headers)
  File "/usr/lib64/python2.7/httplib.py", line 1017, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1045, in _send_request
    self.putrequest(method, url, **skips)
  File "/usr/lib64/python2.7/httplib.py", line 906, in putrequest
    raise CannotSendRequest()
CannotSendRequest
[29/Jan/2018 13:37:49 +0000] 127451 Audit-Plugin throttling_logger INFO     (2 skipped) Unable to send data to nav server. Will try again.

Can someone help me to identify what is this error and how to solve this?

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

EduardBangga — Tue, 30 Jan 2018 09:19:22 GMT

Updates:

I enabled Cloudera Navigator TLS/SSL, the error in Cloudera Manager gone. But it seems Navigator Audit Server still have some problem with sending message saying "Connection Refused"

2018-01-30 16:04:10,178 WARN org.apache.cxf.phase.PhaseInterceptorChain [CMPollThread]: Interceptor for {http://api.cloudera.com/}ApiRootResourceExternal has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message.
	at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
	at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:607)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:543)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206)
	at com.sun.proxy.$Proxy34.readClusters(Unknown Source)
	at com.cloudera.nav.cm.CmApiClient.getClusters(CmApiClient.java:118)
	at com.cloudera.navigator.NavigatorScmProxy.getConfigs(NavigatorScmProxy.java:136)
	at com.cloudera.navigator.NavigatorScmProxy.updateAuditEventFilters(NavigatorScmProxy.java:220)
	at com.cloudera.navigator.NavigatorScmProxy.access$100(NavigatorScmProxy.java:38)
	at com.cloudera.navigator.NavigatorScmProxy$1.run(NavigatorScmProxy.java:106)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: ConnectException invoking https://n01.bigdata.bri.co.id:7183/api/v18/clusters: Connection refused (Connection refused)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1338)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1322)
	at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
	at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:622)
	at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
	... 17 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
	at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
	at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:260)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1517)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1490)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1309)
	... 20 more

Need some help to solve this

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

EduardBangga — Tue, 30 Jan 2018 10:18:01 GMT

Here is another log from cloudera-scm-server.log

2018-01-30 17:08:30,058 ERROR HistoryEvent Collector 1 Thread 3:com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector: Failed to retrieve navigator records.
org.apache.http.conn.HttpHostConnectException: Connection to https://ipaddress:7186 refused
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.request(AbstractNavigatorHistoryEventCollector.java:145)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.getFromOffset(AbstractNavigatorHistoryEventCollector.java:228)
	at com.cloudera.server.web.cmf.history.HistoryEventWrappingIterator.<init>(HistoryEventWrappingIterator.java:39)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.call(AbstractNavigatorHistoryEventCollector.java:237)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.call(AbstractNavigatorHistoryEventCollector.java:71)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
	at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:414)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
	... 15 more
2018-01-30 17:08:33,650 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
	at com.sun.proxy.$Proxy141.writeMetrics(Unknown Source)
	at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:325)
	at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:312)
	at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:146)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
	... 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at sun.net.www.http.HttpClient.New(HttpClient.java:339)
	at sun.net.www.http.HttpClient.New(HttpClient.java:357)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
	at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
	at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
	at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
	at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
	at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
	at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
	... 11 more

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

access — Tue, 15 May 2018 00:53:55 GMT

Any Solution link for this issue.

Thanks

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

EduardBangga — Tue, 15 May 2018 03:14:18 GMT

Hi,

I have found the solution for this, this issue happened because there are not enough threads for Navigator Audit. I solve this issue by increasing threads from default 50 to 150

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Bob_Shrestha — Tue, 12 Jun 2018 15:23:52 GMT

how do you increase the threads for the Navigator Audit Server

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

rohitpath — Mon, 08 Oct 2018 17:14:03 GMT

Where did you increase the number of threads ?

Safety valve ? Please let me know.

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

guoting — Thu, 16 May 2019 03:17:55 GMT

How do?

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

EduardBangga — Thu, 16 May 2019 07:02:15 GMT

Hi, you can solve this issue by adding this line to navigator audit server safety valve

navigator.server.max_threads=150

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

EduardBangga — Thu, 16 May 2019 07:02:41 GMT

Hi, you can solve this issue by adding this line to navigator audit server safety valve

navigator.server.max_threads=150