TaskTracker status unknown

[#1]

CM 4.8 + CDH 4.4 + Centos6

======================

the TaskTracker running fine, but cm status unknown, the image below (cloudera can not upload image successful. -_-! 😞

http://dl2.iteye.com/upload/attachment/0093/6958/4ccc98f9-8df4-31ee-b31d-335e3601674e.jpg

cloudera manager server side log:

=========================

2014-02-10 11:18:53,340 ERROR [1370430967@scm-web-92:cmf.WebController@322] Exception occurred when checking the service health of mapreduce1
2014-02-10 11:19:47,588 ERROR [791289084@scm-web-95:cmf.WebController@322] Exception occurred when checking the service health of mapreduce1
2014-02-10 11:20:47,911 ERROR [791289084@scm-web-95:cmf.WebController@322] Exception occurred when checking the service health of mapreduce1
2014-02-10 11:20:56,194 ERROR [1882873310@scm-web-96:cmf.WebController@350] Exception occurred when checking the role health of mapreduce1-TASKTRACKER-23e87dc83e16f6b18c05945b5abb1213
2014-02-10 11:21:10,772 ERROR [1882873310@scm-web-96:cmf.WebController@350] Exception occurred when checking the role health of mgmt1-SERVICEMONITOR-c37b52ac4062c41f94a67f14be61b846
2014-02-10 11:21:59,882 ERROR [1534966381@scm-web-85:cmf.WebController@322] Exception occurred when checking the service health of mapreduce1
2014-02-10 11:22:04,457 ERROR [1526556624@scm-web-98:cmf.WebController@350] Exception occurred when checking the role health of mapreduce1-TASKTRACKER-23e87dc83e16f6b18c05945b5abb1213
2014-02-10 11:22:51,445 ERROR [1882873310@scm-web-96:cmf.WebController@350] Exception occurred when checking the role health of mapreduce1-TASKTRACKER-23e87dc83e16f6b18c05945b5abb1213

agent side log (no ERROR log):

=======================

[10/Feb/2014 11:16:08 +0000] 112885 MainThread agent        INFO     No command line vars
[10/Feb/2014 11:16:08 +0000] 112885 MainThread agent        INFO     Found database jar: /usr/share/java/mysql-connector-java.jar
[10/Feb/2014 11:16:08 +0000] 112885 MainThread agent        INFO     Missing database jar: /usr/share/java/oracle-connector-java.jar (normal, if you're not using this database type)
[10/Feb/2014 11:16:08 +0000] 112885 MainThread agent        INFO     Found database jar: /usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar
[10/Feb/2014 11:16:08 +0000] 112885 MainThread agent        INFO     Agent starting as pid 112885 user root(0) group root(0).
[10/Feb/2014 11:16:08 +0000] 112885 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent
[10/Feb/2014 11:16:10 +0000] 112885 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups
[10/Feb/2014 11:16:10 +0000] 112885 MainThread cgroups      INFO     Reusing /var/run/cloudera-scm-agent/cgroups/memory
[10/Feb/2014 11:16:10 +0000] 112885 MainThread cgroups      INFO     Reusing /var/run/cloudera-scm-agent/cgroups/cpu
[10/Feb/2014 11:16:10 +0000] 112885 MainThread cgroups      INFO     Reusing /var/run/cloudera-scm-agent/cgroups/cpuacct
[10/Feb/2014 11:16:10 +0000] 112885 MainThread cgroups      INFO     Reusing /var/run/cloudera-scm-agent/cgroups/blkio
[10/Feb/2014 11:16:10 +0000] 112885 MainThread agent        INFO     Found cgroups capabilities: {'has_memory': True, 'default_memory_limit_in_bytes': 9223372036854775807, 'default_memory_soft_limit_in_bytes': 9223372036854775807, 'writable_cgroup_dot_procs': True, 'default_cpu_rt_runtime_us': 950000, 'has_cpu': True, 'default_blkio_weight': 1000, 'default_cpu_shares': 1024, 'has_cpuacct': True, 'has_blkio': True}
[10/Feb/2014 11:16:10 +0000] 112885 MainThread agent        INFO     Setting up supervisord event monitor.
[10/Feb/2014 11:16:10 +0000] 112885 MainThread filesystem_map INFO     Monitored nodev filesystem types: ['nfs', 'nfs4', 'tmpfs']
[10/Feb/2014 11:16:10 +0000] 112885 MainThread filesystem_map INFO     Using timeout of 2.000000
[10/Feb/2014 11:16:10 +0000] 112885 MainThread filesystem_map INFO     Using join timeout of 0.100000
[10/Feb/2014 11:16:10 +0000] 112885 MainThread filesystem_map INFO     Using tolerance of 60.000000
[10/Feb/2014 11:16:10 +0000] 112885 MainThread agent        INFO     Using metrics_url_timeout_seconds of 30.000000
[10/Feb/2014 11:16:10 +0000] 112885 MainThread agent        INFO     Using task_metrics_timeout_seconds of 5.000000
[10/Feb/2014 11:16:10 +0000] 112885 MainThread agent        INFO     Using max_collection_wait_seconds of 10.000000
[10/Feb/2014 11:16:10 +0000] 112885 MainThread metrics      INFO     Importing tasktracker metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/tasktracker/schema.json
[10/Feb/2014 11:16:10 +0000] 112885 MainThread dns_names    INFO     Using timeout of 2.000000
[10/Feb/2014 11:16:10 +0000] 112885 MainThread __init__     INFO     Importing metric schema from file /usr/lib64/cmf/agent/src/cmf/monitor/schema.json
[10/Feb/2014 11:16:11 +0000] 112885 MainThread agent        INFO     Supervised processes will add the following to their environment (in addition to the supervisor's env): {'CDH_OOZIE_HOME': '/usr/lib/oozie', 'CDH_MR2_HOME': '/usr/lib/hadoop-mapreduce', 'CDH_ZOOKEEPER_HOME': '/usr/lib/zookeeper', 'CDH_HADOOP_BIN': '/usr/bin/hadoop', 'MGMT_HOME': '/usr/share/cmf', 'CDH_IMPALA_HOME': '/usr/lib/impala', 'CLOUDERA_MYSQL_CONNECTOR_JAR': '/usr/share/java/mysql-connector-java.jar', 'CDH_YARN_HOME': '/usr/lib/hadoop-yarn', 'CMF_PACKAGE_DIR': '/usr/lib64/cmf/service', 'CDH_HDFS_HOME': '/usr/lib/hadoop-hdfs', 'CDH_SOLR_HOME': '/usr/lib/solr', 'CDH_PIG_HOME': '/usr/lib/pig', 'CDH_SQOOP2_HOME': '/usr/lib/sqoop2', 'CDH_HUE_PLUGINS_HOME': '/usr/lib/hadoop', 'CM_STATUS_CODES': u'STATUS_NONE HDFS_DFS_DIR_NOT_EMPTY JOBTRACKER_IN_STANDBY_MODE', 'CDH_MR1_HOME': '/usr/lib/hadoop-0.20-mapreduce', 'CLOUDERA_ORACLE_CONNECTOR_JAR': '/usr/share/java/oracle-connector-java.jar', 'CDH_HUE_HOME': '/usr/share/hue', 'CDH_HIVE_HOME': '/usr/lib/hive', 'CDH_HTTPFS_HOME': '/usr/lib/hadoop-httpfs', 'CDH_HADOOP_HOME': '/usr/lib/hadoop', 'JSVC_HOME': '/usr/libexec/bigtop-utils', 'HIVE_DEFAULT_XML': '/etc/hive/conf.dist/hive-default.xml', 'CDH_FLUME_HOME': '/usr/lib/flume-ng', 'CLOUDERA_POSTGRESQL_JDBC_JAR': '/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar', 'CDH_HBASE_INDEXER_HOME': '/usr/lib/hbase-solr', 'TOMCAT_HOME': '/usr/lib/bigtop-tomcat', 'CDH_HBASE_HOME': '/usr/lib/hbase', 'CDH_HCAT_HOME': '/usr/lib/hcatalog'}
[10/Feb/2014 11:16:11 +0000] 112885 MainThread agent        INFO     To override these variables, use /etc/cloudera-scm-agent/config.ini. Environment variables for CDH locations are not used when CDH is installed from parcels.
[10/Feb/2014 11:16:11 +0000] 112885 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
[10/Feb/2014 11:16:11 +0000] 112885 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor
[10/Feb/2014 11:16:11 +0000] 112885 MainThread agent        INFO     Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
[10/Feb/2014 11:16:11 +0000] 112885 MainThread agent        INFO     Connecting to previous supervisor: agent-105652-1391993776.
[10/Feb/2014 11:16:11 +0000] 112885 MainThread agent        INFO     Pre-existing process: 365-mapreduce-TASKTRACKER
[10/Feb/2014 11:16:11 +0000] 112885 MainThread agent        INFO     Pre-existing process: 363-hdfs-DATANODE
[10/Feb/2014 11:16:11 +0000] 112885 MainThread _cplogging   INFO     [10/Feb/2014:11:16:11] ENGINE Bus STARTING
[10/Feb/2014 11:16:11 +0000] 112885 MainThread _cplogging   INFO     [10/Feb/2014:11:16:11] ENGINE Started monitor thread '_TimeoutMonitor'.
[10/Feb/2014 11:16:12 +0000] 112885 MainThread _cplogging   INFO     [10/Feb/2014:11:16:12] ENGINE Serving on dn3.hadoop.com:9000
[10/Feb/2014 11:16:12 +0000] 112885 MainThread _cplogging   INFO     [10/Feb/2014:11:16:12] ENGINE Bus STARTED
[10/Feb/2014 11:16:12 +0000] 112885 MainThread __init__     INFO     New monitor: (<cmf.monitor.host.HostMonitor object at 0x289de50>,)
[10/Feb/2014 11:16:12 +0000] 112885 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: (<cmf.monitor.host.HostMonitor object at 0x289de50>,)
[10/Feb/2014 11:16:12 +0000] 112885 MainThread agent        WARNING  Setting default socket timeout to 30!
[10/Feb/2014 11:16:12 +0000] 112885 MainThread parcel       INFO     Agent creates users/groups and applies file permissions: True
[10/Feb/2014 11:16:12 +0000] 112885 MainThread downloader   INFO     Downloader path: /opt/cloudera/parcel-cache
[10/Feb/2014 11:16:12 +0000] 112885 MainThread parcel_cache INFO     Using /opt/cloudera/parcel-cache for parcel cache
[10/Feb/2014 11:16:12 +0000] 112885 MainThread firehoses    INFO     Reporting interval updated: 5.0 -> 60
[10/Feb/2014 11:16:12 +0000] 112885 MainThread agent        INFO     Process with same id has changed: 363-hdfs-DATANODE.
[10/Feb/2014 11:16:12 +0000] 112885 MainThread agent        INFO     Refreshing files for process 363-hdfs-DATANODE: []
[10/Feb/2014 11:16:12 +0000] 112885 MainThread parcel       INFO     prepare_environment begin: {}, [u'cdh'], [u'cdh-plugin', u'hdfs-plugin']
[10/Feb/2014 11:16:12 +0000] 112885 MainThread parcel       INFO     No parcels activated for use
[10/Feb/2014 11:16:12 +0000] 112885 MainThread util         INFO     Extracted 0 files and 0 dirs to /var/run/cloudera-scm-agent/process/363-hdfs-DATANODE.
[10/Feb/2014 11:16:12 +0000] 112885 MainThread abstract_monitor INFO     Refreshing DataNodeMonitor for None
[10/Feb/2014 11:16:12 +0000] 112885 MainThread __init__     INFO     New monitor: (<cmf.monitor.datanode.DataNodeMonitor object at 0x377fd50>,)
[10/Feb/2014 11:16:12 +0000] 112885 MainThread agent        INFO     Process with same id has changed: 365-mapreduce-TASKTRACKER.
[10/Feb/2014 11:16:12 +0000] 112885 MainThread agent        INFO     Refreshing files for process 365-mapreduce-TASKTRACKER: []
[10/Feb/2014 11:16:12 +0000] 112885 MainThread parcel       INFO     prepare_environment begin: {}, [u'cdh'], [u'cdh-plugin', u'mapreduce-plugin']
[10/Feb/2014 11:16:12 +0000] 112885 MainThread parcel       INFO     No parcels activated for use
[10/Feb/2014 11:16:12 +0000] 112885 MainThread util         INFO     Extracted 0 files and 0 dirs to /var/run/cloudera-scm-agent/process/365-mapreduce-TASKTRACKER.
[10/Feb/2014 11:16:12 +0000] 112885 MainThread abstract_monitor INFO     Refreshing TaskTrackerMonitor for None
[10/Feb/2014 11:16:12 +0000] 112885 MainThread __init__     INFO     New monitor: (<cmf.monitor.tasktracker.TaskTrackerMonitor object at 0x377f890>,)
[10/Feb/2014 11:16:12 +0000] 112885 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: (<cmf.monitor.datanode.DataNodeMonitor object at 0x377fd50>,)
[10/Feb/2014 11:16:12 +0000] 112885 MonitorDaemon-Scheduler __init__     INFO     Monitor ready to report: (<cmf.monitor.tasktracker.TaskTrackerMonitor object at 0x377f890>,)
[10/Feb/2014 11:16:12 +0000] 112885 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the ACTIVITYMONITOR.
[10/Feb/2014 11:16:12 +0000] 112885 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the SERVICEMONITOR.
[10/Feb/2014 11:16:12 +0000] 112885 MonitorDaemon-Reporter firehoses    INFO     Creating a connection to the HOSTMONITOR.
[10/Feb/2014 11:17:12 +0000] 112885 Monitor-HostMonitor network_interfaces WARNING  Interface 'eth2' reported unknown speed '4294967295'
[10/Feb/2014 11:17:12 +0000] 112885 Monitor-HostMonitor network_interfaces WARNING  Interface 'eth2' reported unknown duplex mode '255'
[10/Feb/2014 11:17:12 +0000] 112885 Monitor-HostMonitor network_interfaces WARNING  Interface 'eth3' reported unknown speed '4294967295'
[10/Feb/2014 11:17:12 +0000] 112885 Monitor-HostMonitor network_interfaces WARNING  Interface 'eth3' reported unknown duplex mode '255'
[10/Feb/2014 11:17:12 +0000] 112885 Monitor-HostMonitor network_interfaces INFO     NIC iface virbr0 doesn't support ETHTOOL (95)
[10/Feb/2014 11:22:21 +0000] 112885 CP Server Thread-4 _cplogging   INFO     172.16.11.25 - - [10/Feb/2014:11:22:21] "GET /download_log?path=%2Fvar%2Flog%2Fhadoop-0.20-mapreduce%2Fhadoop-cmf-mapreduce1-TASKTRACKER-dn3.hadoop.com.log.out&onlyTail=true&compress=false HTTP/1.1" 200 1993 "" "Java/1.6.0_35"

thx

--heipark

[#2]

server side more detail:

=========================

2014-02-10 12:41:40,238 ERROR [1901825644@scm-web-7:cmf.WebController@215] Exception occurred when checking the role health of mapreduce1-TASKTRACKER-23e87dc83e16f6b18c05945b5abb1213 org.apache.avro.AvroRemoteException: java.net.SocketTimeoutException: Read timed out at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:88) at $Proxy59.getAvroHealthReport(Unknown Source) at com.cloudera.cmf.protocol.firehose.nozzle.TimeoutNozzleIPC.getAvroHealthReport(TimeoutNozzleIPC.java:123) at com.cloudera.cmon.NozzleIPCWrapper.getHealthReport(NozzleIPCWrapper.java:584) at com.cloudera.server.web.cmf.HealthCheckController.safeGetHealthReport(HealthCheckController.java:212) at com.cloudera.server.web.cmf.HealthCheckController.instanceStatusHealthCheckJSON(HealthCheckController.java:101) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:436) at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:424) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:669) at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:574) at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) at javax.servlet.http.HttpServlet.service(HttpServlet.java:820) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221) at org.mortbay.servlet.UserAgentFilter.doFilter(UserAgentFilter.java:78) at org.mortbay.servlet.GzipFilter.doFilter(GzipFilter.java:131) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at com.jamonapi.http.JAMonServletFilter.doFilter(JAMonServletFilter.java:48) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at com.cloudera.enterprise.JavaMelodyFacade$MonitoringFilter.doFilter(JavaMelodyFacade.java:109) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:311) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:101) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:323) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:173) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.StatisticsHandler.handle(StatisticsHandler.java:53) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) Caused by: java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:218) at java.io.BufferedInputStream.read1(BufferedInputStream.java:258) at java.io.BufferedInputStream.read(BufferedInputStream.java:317) at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:690) at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:633) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) at org.apache.avro.ipc.HttpTransceiver.readBuffers(HttpTransceiver.java:54) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:59) at org.apache.avro.ipc.Transceiver.transceive(Transceiver.java:72) at org.apache.avro.ipc.Requestor.request(Requestor.java:147) at org.apache.avro.ipc.Requestor.request(Requestor.java:101) at org.apache.avro.ipc.specific.SpecificRequestor.invoke(SpecificRequestor.java:72) ... 74 more

[#3 · Accepted solution]

problem resovled 

when i restart the agent which running on the cloudera manager host.

It's a little strange, but it works.