Reply
Explorer
Posts: 13
Registered: ‎11-23-2017
Accepted Solution

Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Hi All,

 

I have some issue with Navigator Audit Server because the Cloudera Agent can't send audit to audit serverCapture.PNG

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

This is the log from cloudera agent

[29/Jan/2018 13:37:49 +0000] 127451 Audit-Plugin throttling_logger ERROR    (2 skipped) Unable to send message to audit server for hdfs service.
Traceback (most recent call last):
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/cmf-5.13.1-py2.7.egg/cmf/audit/navigator_sender.py", line 98, in _send_avro_data
    response = self._requestor.request(ipc_request_name, dict(request=request))
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 141, in request
    return self.issue_request(call_request, message_name, request_datum)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 254, in issue_request
    call_response = self.transceiver.transceive(call_request)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 482, in transceive
    self.write_framed_message(request)
  File "/usr/lib64/cmf/agent/build/env/lib/python2.7/site-packages/avro-1.6.3-py2.7.egg/avro/ipc.py", line 501, in write_framed_message
    self.conn.request(req_method, self.req_resource, req_body, req_headers)
  File "/usr/lib64/python2.7/httplib.py", line 1017, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib64/python2.7/httplib.py", line 1045, in _send_request
    self.putrequest(method, url, **skips)
  File "/usr/lib64/python2.7/httplib.py", line 906, in putrequest
    raise CannotSendRequest()
CannotSendRequest
[29/Jan/2018 13:37:49 +0000] 127451 Audit-Plugin throttling_logger INFO     (2 skipped) Unable to send data to nav server. Will try again.

Can someone help me to identify what is this error and how to solve this?

Explorer
Posts: 13
Registered: ‎11-23-2017

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

[ Edited ]

Updates:

I enabled Cloudera Navigator TLS/SSL, the error in Cloudera Manager gone. But it seems Navigator Audit Server still have some problem with sending message saying "Connection Refused"

 

2018-01-30 16:04:10,178 WARN org.apache.cxf.phase.PhaseInterceptorChain [CMPollThread]: Interceptor for {http://api.cloudera.com/}ApiRootResourceExternal has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault: Could not send Message.
	at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:64)
	at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:271)
	at org.apache.cxf.jaxrs.client.AbstractClient.doRunInterceptorChain(AbstractClient.java:607)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.doChainedInvocation(ClientProxyImpl.java:543)
	at org.apache.cxf.jaxrs.client.ClientProxyImpl.invoke(ClientProxyImpl.java:206)
	at com.sun.proxy.$Proxy34.readClusters(Unknown Source)
	at com.cloudera.nav.cm.CmApiClient.getClusters(CmApiClient.java:118)
	at com.cloudera.navigator.NavigatorScmProxy.getConfigs(NavigatorScmProxy.java:136)
	at com.cloudera.navigator.NavigatorScmProxy.updateAuditEventFilters(NavigatorScmProxy.java:220)
	at com.cloudera.navigator.NavigatorScmProxy.access$100(NavigatorScmProxy.java:38)
	at com.cloudera.navigator.NavigatorScmProxy$1.run(NavigatorScmProxy.java:106)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: ConnectException invoking https://n01.bigdata.bri.co.id:7183/api/v18/clusters: Connection refused (Connection refused)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.mapException(HTTPConduit.java:1338)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1322)
	at org.apache.cxf.transport.AbstractConduit.close(AbstractConduit.java:56)
	at org.apache.cxf.transport.http.HTTPConduit.close(HTTPConduit.java:622)
	at org.apache.cxf.interceptor.MessageSenderInterceptor$MessageSenderEndingInterceptor.handleMessage(MessageSenderInterceptor.java:62)
	... 17 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264)
	at sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:367)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:191)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
	at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:177)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564)
	at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
	at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
	at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:347)
	at org.apache.cxf.transport.http.URLConnectionHTTPConduit$URLConnectionWrappedOutputStream.getResponseCode(URLConnectionHTTPConduit.java:260)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponseInternal(HTTPConduit.java:1517)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.handleResponse(HTTPConduit.java:1490)
	at org.apache.cxf.transport.http.HTTPConduit$WrappedOutputStream.close(HTTPConduit.java:1309)
	... 20 more

 

Need some help to solve this

Explorer
Posts: 13
Registered: ‎11-23-2017

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Here is another log from cloudera-scm-server.log

 

2018-01-30 17:08:30,058 ERROR HistoryEvent Collector 1 Thread 3:com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector: Failed to retrieve navigator records.
org.apache.http.conn.HttpHostConnectException: Connection to https://ipaddress:7186 refused
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:190)
	at org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:294)
	at org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:643)
	at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:479)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805)
	at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:784)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.request(AbstractNavigatorHistoryEventCollector.java:145)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.getFromOffset(AbstractNavigatorHistoryEventCollector.java:228)
	at com.cloudera.server.web.cmf.history.HistoryEventWrappingIterator.<init>(HistoryEventWrappingIterator.java:39)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.call(AbstractNavigatorHistoryEventCollector.java:237)
	at com.cloudera.server.web.cmf.history.navigator.AbstractNavigatorHistoryEventCollector.call(AbstractNavigatorHistoryEventCollector.java:71)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:673)
	at org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:414)
	at org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:180)
	... 15 more
2018-01-30 17:08:33,650 INFO CMMetricsForwarder-0:com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder: Failed to send metrics.
java.lang.reflect.UndeclaredThrowableException
	at com.sun.proxy.$Proxy141.writeMetrics(Unknown Source)
	at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendWithAvro(ClouderaManagerMetricsForwarder.java:325)
	at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.sendMetrics(ClouderaManagerMetricsForwarder.java:312)
	at com.cloudera.server.cmf.components.ClouderaManagerMetricsForwarder.run(ClouderaManagerMetricsForwarder.java:146)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.avro.AvroRemoteException: java.net.ConnectException: Connection refused (Connection refused)
	at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88)
	... 11 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at sun.net.NetworkClient.doConnect(NetworkClient.java:175)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:463)
	at sun.net.www.http.HttpClient.openServer(HttpClient.java:558)
	at sun.net.www.http.HttpClient.<init>(HttpClient.java:242)
	at sun.net.www.http.HttpClient.New(HttpClient.java:339)
	at sun.net.www.http.HttpClient.New(HttpClient.java:357)
	at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156)
	at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050)
	at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1334)
	at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1309)
	at org.apache.avro.ipc.HttpTransceiver.writeBuffers(HttpTransceiver.java:71)
	at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:58)
	at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72)
	at org.apache.avro.ipc.Requestor.request(Requestor.java:147)
	at org.apache.avro.ipc.Requestor.request(Requestor.java:101)
	at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72)
	... 11 more
New Contributor
Posts: 1
Registered: ‎02-27-2018

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

HI 

 

Any Solution link for this issue.

 

Thanks 

 

Explorer
Posts: 13
Registered: ‎11-23-2017

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

Hi,

 

I have found the solution for this, this issue happened because there are not enough threads for Navigator Audit. I solve this issue by increasing threads from default 50 to 150

New Contributor
Posts: 1
Registered: ‎06-12-2018

Re: Navigator Audit Pipeline Health Check (Unable to send message to audit server for hdfs service)

how do you increase the threads for the Navigator Audit Server

Announcements
New solutions