Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Solved Go to solution

Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Hi All,

 

I have some issue with Navigator Audit Server because the Cloudera Agent can't send audit to audit serverCapture.PNG

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

This is the log from cloudera agent

[29/Jan/2018 13:37:49 +0000] 127451 Audit-Plugin throttling_logger ERROR    (2 skipped) Unable to send message to audit server for hdfs service.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/audit/navigator_sender.py", line 98, in _send_avro_data
    response = self._requestor.request(ipc_request_name, dict(request=request))
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 482, in transceive
    self.write_framed_message(request)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 501, in write_framed_message
    self.conn.request(req_method, self.req_resource, req_body, req_headers)
  File "/usr/lib64/python2.7/httplib.py", line 1017, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1045, in _send_request
    self.putrequest(method, url, **skips)
  File "/usr/lib64/python2.7/httplib.py", line 906, in putrequest
    raise CannotSendRequest()
CannotSendRequest
[29/Jan/2018 13:37:49 +0000] 127451 Audit-Plugin throttling_logger INFO     (2 skipped) Unable to send data to nav server. Will try again.

Can someone help me to identify what is this error and how to solve this?

1 ACCEPTED SOLUTION

Accepted Solutions

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Hi,

 

I have found the solution for this, this issue happened because there are not enough threads for Navigator Audit. I solve this issue by increasing threads from default 50 to 150

9 REPLIES 9

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Updates:

I enabled Cloudera Navigator TLS/SSL, the error in Cloudera Manager gone. But it seems Navigator Audit Server still have some problem with sending message saying "Connection Refused"

 

2018-01-30 16:04:10,178 WARN org.apache.cxf.phase.PhaseInterceptorChain [CMPollThread]: Interceptor for {http://api.cloudera.com/}ApiRootResourceExternal has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message.
	at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
	at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:607)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:543)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206)
	at com.sun.proxy.$Proxy34.readClusters(Unknown Source)
	at com.cloudera.nav.cm.CmApiClient.getClusters(CmApiClient.java:118)
	at com.cloudera.navigator.NavigatorScmProxy.getConfigs(NavigatorScmProxy.java:136)
	at com.cloudera.navigator.NavigatorScmProxy.updateAuditEventFilters(NavigatorScmProxy.java:220)
	at com.cloudera.navigator.NavigatorScmProxy.access$100(NavigatorScmProxy.java:38)
	at com.cloudera.navigator.NavigatorScmProxy$1.run(NavigatorScmProxy.java:106)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: ConnectException invoking https://n01.bigdata.bri.co.id:7183/api/v18/clusters: Connection refused (Connection refused)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1338)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1322)
	at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
	at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:622)
	at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
	... 17 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
	at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
	at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:260)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1517)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1490)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1309)
	... 20 more

 

Need some help to solve this

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Here is another log from cloudera-scm-server.log

 

2018-01-30 17:08:30,058 ERROR HistoryEvent Collector 1 Thread 3:com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector: Failed to retrieve navigator records.
org.apache.http.conn.HttpHostConnectException: Connection to https://ipaddress:7186 refused
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.request(AbstractNavigatorHistoryEventCollector.java:145)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.getFromOffset(AbstractNavigatorHistoryEventCollector.java:228)
	at com.cloudera.server.web.cmf.history.HistoryEventWrappingIterator.<init>(HistoryEventWrappingIterator.java:39)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.call(AbstractNavigatorHistoryEventCollector.java:237)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.call(AbstractNavigatorHistoryEventCollector.java:71)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
	at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:414)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
	... 15 more
2018-01-30 17:08:33,650 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
	at com.sun.proxy.$Proxy141.writeMetrics(Unknown Source)
	at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:325)
	at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:312)
	at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:146)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
	... 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at sun.net.www.http.HttpClient.New(HttpClient.java:339)
	at sun.net.www.http.HttpClient.New(HttpClient.java:357)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
	at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
	at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
	at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
	at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
	at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
	at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
	... 11 more

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

New Contributor

HI 

 

Any Solution link for this issue.

 

Thanks 

 

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Hi,

 

I have found the solution for this, this issue happened because there are not enough threads for Navigator Audit. I solve this issue by increasing threads from default 50 to 150

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

New Contributor

how do you increase the threads for the Navigator Audit Server

Highlighted

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

New Contributor

Where did you increase the number of threads ? 

 

Safety valve ? Please let me know.

 

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

New Contributor

How  do?

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Hi, you can solve this issue by adding this line to navigator audit server safety valve

navigator.server.max_threads=150

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Hi, you can solve this issue by adding this line to navigator audit server safety valve

navigator.server.max_threads=150