Member since
11-27-2020
9
Posts
0
Kudos Received
0
Solutions
12-17-2020
02:30 AM
we are facing two issues in this prod server DNS test fail and NTP server connection time out. [17/Dec/2020 00:41:47 +0000] 9333 Monitor-HostMonitor throttling_logger ERROR Timeout with args ['ntpq', '-np']
[17/Dec/2020 00:41:47 +0000] 9333 Monitor-HostMonitor throttling_logger ERROR Failed to collect NTP metrics
[17/Dec/2020 00:42:08 +0000] 9333 MainThread agent ERROR Failed to configure inotify. Parcel repository will not auto-refresh.
[17/Dec/2020 01:31:26 +0000] 9333 Monitor-HostMonitor throttling_logger ERROR Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/0
[17/Dec/2020 01:31:54 +0000] 9333 MainThread agent ERROR Failed to configure inotify. Parcel repository will not auto-refresh.
[17/Dec/2020 08:18:56 +0000] 9333 MonitorDaemon-Reporter throttling_logger ERROR Error sending messages to firehose: mgmt-SERVICEMONITOR-b9bbe3508c15c97839a21fc44a6226b5
[17/Dec/2020 10:07:14 +0000] 9333 MainThread agent ERROR Failed to configure inotify. Parcel repository will not auto-refresh.
[17/Dec/2020 11:15:43 +0000] 9333 MainThread agent ERROR Failed to configure inotify. Parcel repository will not auto-refresh.
[17/Dec/2020 11:31:05 +0000] 9333 DnsResolutionMonitor throttling_logger ERROR Timeout with args ['/usr/java/jdk1.8.0_251-amd64/bin/java', '-classpath', '/opt/cloudera/cm/lib/agent-6.3.0.jar', 'com.cloudera.cmon.agent.DnsTest']
[17/Dec/2020 11:31:05 +0000] 9333 DnsResolutionMonitor throttling_logger ERROR Failed to run DnsTest.
[17/Dec/2020 11:31:18 +0000] 9333 MainThread agent ERROR Failed to configure inotify. Parcel repository will not auto-refresh.
[17/Dec/2020 12:13:44 +0000] 9333 MainThread agent ERROR Failed to configure inotify. Parcel repository will not auto-refresh.
[ PROD [17/Dec/2020 11:07:11 +0000] 9333 MainThread agent WARNING Long HB processing time: 16.7383139133
[17/Dec/2020 11:07:23 +0000] 9333 Monitor-HostMonitor filesystem_map WARNING Failed to join worker process collecting filesystem usage. All nodev filesystems will have unknown usage until the worker process is no longer active. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/0
[17/Dec/2020 11:15:29 +0000] 9333 MainThread agent WARNING Supervisor failed (pid 97042). Restarting agent.
[17/Dec/2020 11:15:43 +0000] 9333 MainThread agent ERROR Failed to configure inotify. Parcel repository will not auto-refresh.
[17/Dec/2020 11:15:43 +0000] 9333 MainThread throttling_logger WARNING Failed parsing alternatives line: libnssckbi.so.x86_64 string index out of range link currently points to /usr/lib64/pkcs11/p11-kit-trust.so
[17/Dec/2020 11:15:48 +0000] 9333 MainThread agent WARNING Long HB processing time: 5.60892701149
[17/Dec/2020 11:30:53 +0000] 9333 Monitor-HostMonitor filesystem_map WARNING Failed to join worker process collecting filesystem usage. All nodev filesystems will have unknown usage until the worker process is no longer active. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/0
[17/Dec/2020 11:30:54 +0000] 9333 MainThread agent WARNING Long HB processing time: 33.9636788368
[17/Dec/2020 11:30:54 +0000] 9333 MainThread agent WARNING Delayed HB: 19s since last
[17/Dec/2020 11:31:05 +0000] 9333 DnsResolutionMonitor throttling_logger ERROR Timeout with args ['/usr/java/jdk1.8.0_251-amd64/bin/java', '-classpath', '/opt/cloudera/cm/lib/agent-6.3.0.jar', 'com.cloudera.cmon.agent.DnsTest']
[17/Dec/2020 11:31:05 +0000] 9333 DnsResolutionMonitor throttling_logger ERROR Failed to run DnsTest.
[17/Dec/2020 11:31:09 +0000] 9333 MainThread agent WARNING Supervisor failed (pid 97042). Restarting agent.
[17/Dec/2020 11:31:18 +0000] 9333 MainThread agent ERROR Failed to configure inotify. Parcel repository will not auto-refresh.
[17/Dec/2020 11:31:18 +0000] 9333 MainThread throttling_logger WARNING Failed parsing alternatives line: libnssckbi.so.x86_64 string index out of range link currently points to /usr/lib64/pkcs11/p11-kit-trust.so
[17/Dec/2020 11:31:23 +0000] 9333 MainThread agent WARNING Long HB processing time: 5.59937500954
[17/Dec/2020 12:07:16 +0000] 9333 MainThread agent WARNING Long HB processing time: 18.2336220741
[17/Dec/2020 12:07:16 +0000] 9333 MainThread agent WARNING Delayed HB: 3s since last
[17/Dec/2020 12:07:21 +0000] 9333 Monitor-HostMonitor filesystem_map WARNING Failed to join worker process collecting filesystem usage. All nodev filesystems will have unknown usage until the worker process is no longer active. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/0
[17/Dec/2020 12:13:34 +0000] 9333 MainThread agent WARNING Supervisor failed (pid 97042). Restarting agent.
[17/Dec/2020 12:13:44 +0000] 9333 MainThread agent ERROR Failed to configure inotify. Parcel repository will not auto-refresh.
[17/Dec/2020 12:13:44 +0000] 9333 MainThread throttling_logger WARNING Failed parsing alternatives line: libnssckbi.so.x86_64 string index out of range link currently points to /usr/lib64/pkcs11/p11-kit-trust.so
[17/Dec/2020 12:13:50 +0000] 9333 MainThread agent WARNING Long HB processing time: 5.56325793266
[ PROD root@ [17/Dec/2020 11:30:53 +0000] 9333 Monitor-HostMonitor filesystem_map WARNING Failed to join worker process collecting filesystem usage. All nodev filesystems will have unknown usage until the worker process is no longer active. Current nodev filesystems: /dev/shm,/run,/sys/fs/cgroup,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/0
[17/Dec/2020 11:30:54 +0000] 9333 MainThread agent WARNING Long HB processing time: 33.9636788368
[17/Dec/2020 11:30:54 +0000] 9333 MainThread agent WARNING Delayed HB: 19s since last
[17/Dec/2020 11:31:05 +0000] 9333 DnsResolutionMonitor throttling_logger ERROR Timeout with args ['/usr/java/jdk1.8.0_251-amd64/bin/java', '-classpath', '/opt/cloudera/cm/lib/agent-6.3.0.jar', 'com.cloudera.cmon.agent.DnsTest']
None
[17/Dec/2020 11:31:05 +0000] 9333 DnsResolutionMonitor throttling_logger ERROR Failed to run DnsTest.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/dns_names.py", line 87, in collect_dns_metrics
self._subprocess_with_timeout(args, self._poll_timeout)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/monitor/host/dns_names.py", line 59, in _subprocess_with_timeout
return subprocess_with_timeout(args, timeout)
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/subprocess_timeout.py", line 95, in subprocess_with_timeout
raise Exception("timeout with args %s" % args)
Exception: timeout with args ['/usr/java/jdk1.8.0_251-amd64/bin/java', '-classpath', '/opt/cloudera/cm/lib/agent-6.3.0.jar', 'com.cloudera.cmon.agent.DnsTest']
[17/Dec/2020 11:31:09 +0000] 9333 MainThread agent WARNING Supervisor failed (pid 97042). Restarting agent.
[17/Dec/2020 11:31:11 +0000] 9333 MainThread agent INFO ================================================================================
[17/Dec/2020 11:31:11 +0000] 9333 MainThread agent INFO SCM Agent Version: 6.3.0
[17/Dec/2020 11:31:11 +0000] 9333 MainThread agent INFO Agent Protocol Version: 4
[17/Dec/2020 11:31:11 +0000] 9333 MainThread __init__ INFO Agent UUID file was last modified at 2020-06-22 17:15:03.518251
[17/Dec/2020 11:31:11 +0000] 9333 MainThread agent INFO Using Host ID: 2b537ad6-388e-4e32-bea2-7584f509d4df
[17/Dec/2020 11:31:11 +0000] 9333 MainThread agent INFO Using directory: /run/cloudera-scm-agent
[17/Dec/2020 11:31:11 +0000] 9333 MainThread agent INFO Using supervisor binary path: /opt/cloudera/cm-agent/bin/../bin/supervisord
[17/Dec/2020 11:31:11 +0000] 9333 MainThread agent INFO Agent Logging Level: INFO
[17/Dec/2020 11:31:11 +0000] 9333 MainThread agent INFO Agent config:
[17/Dec/2020 11:31:11 +0000] 9333 MainThread agent INFO Security.use_tls = 0
[17/Dec/2020 11:31:11 +0000] 9333 MainThread agent INFO Security.max_cert_depth = 9
[17/Dec/2020 11:3 [17/Dec/2020 10:07:14 +0000] 9333 MainThread agent ERROR Failed to configure inotify. Parcel repository will not auto-refresh.
Traceback (most recent call last):
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/agent.py", line 1007, in _init_after_first_heartbeat_response
self.inotify = self.repo.configure_inotify()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/cmf/parcel.py", line 408, in configure_inotify
wm = pyinotify.WatchManager()
File "/opt/cloudera/cm-agent/lib/python2.7/site-packages/pyinotify.py", line 1783, in __init__
raise OSError(err % self._inotify_wrapper.str_errno())
OSError: Cannot initialize new instance of inotify, Errno=Too many open files (EMFILE)
[17/Dec/2020 10:07:14 +0000] 9333 MainThread downloader INFO Downloader path: /opt/cloudera/parcel-cache
[17/Dec/2020 10:07:14 +0000] 9333 MainThread parcel_cache INFO Using /opt/cloudera/parcel-cache for parcel cache
[17/Dec/2020 10:07:14 +0000] 9333 MainThread throttling_logger WARNING Failed parsing alternatives line: libnssckbi.so.x86_64 string index out of range link currently points to /usr/lib64/pkcs11/p11-kit-trust.so
[
... View more
Labels:
- Labels:
-
Cloudera Manager
12-11-2020
08:16 PM
My cluster Java Heap memory values kindly please check this values. Raj@77 Your Service Monitor is running out of Java Heap hence the issue:
where can i found exact location of service monitor using java heap memory value. is there any specific requirement for service monitor Java Heap value. Is there any issue. /etc/default/cloudera-scm-server
Java Options.
#
# Default value sets Java maximum heap size to 2GB, and Java maximum permanent
# generation size to 256MB.
#
export CMF_JAVA_OPTS="-Xmx2G -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp" Heap size: https://docs.cloudera.com/cloudera-manager/7.2.1/managing-clusters/topics/cm-configuring-memory-allo... Link is not working
... View more
12-11-2020
06:46 AM
SERVICE_MONITOR_AGGREGATION_RUN_DURATION
The health test result for SERVICE_MONITOR_AGGREGATION_RUN_DURATION has become bad: The last metrics aggregation run duration is 33.9 second(s). Critical threshold: 30 second(s).
Service Monitor File /var/log/cloudera-scm-firehose/mgmt-cmf-mgmt-SERVICEMONITOR.log.out
8:35:51.569 PM WARN Groups Potential performance problem: getGroups(user=hue) took 8024 milliseconds.
8:36:21.270 PM WARN JvmPauseMonitor Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 18430ms: no GCs detected.
8:36:21.558 PM WARN EnterpriseService com.cloudera.cmf.PollingScmProxy: run duration exceeded desired period. Duration: 19230 ms. Desired period: 1000 ms.
8:36:22.253 PM INFO AggregatingTimeSeriesStore Run took PT33.916S which is over the slow run threshold of PT30S. 15690 metrics written for 28 entities. PT27.952S write time over 2 writes. Longest writes: PT27.876S,PT0.076S.
8:37:02.720 PM INFO LDBPartitionManager Updating partition=LDBPartitionMetadataWrapper{tableName=stream, partitionName=stream_2020-12-10T17:48:01.580Z, startTime=2020-12-10T17:48:01.580Z, endTime=null, version=2, state=OPEN}. Setting endTime=2020-12-10T18:38:02.709Z Metrics Aggregation Run Duration Thresholds
critical:30000.0, warning:10000.0
present we are using default values in my cluster. any changes required for this values. else any other issue is there in my cluster.
... View more
Labels:
- Labels:
-
Cloudera Manager
12-11-2020
06:16 AM
This is for new cluster. Installations all are done Present it was in production. This was happen Name node. some times in day (10 to 20) min it was not SYNC with NFS server. we are getting all running services facing clock offset Alerts.
... View more
12-11-2020
06:08 AM
Thanks for your response. we are configuring 60000. present it is ok
... View more
12-08-2020
05:14 AM
cloudera-scm-agent.log Monitor-HostMonitor throttling_logger ERROR [08/Dec/2020 12:28:50 +0000] 76583 Monitor-HostMonitor throttling_logger ERROR (2 skipped) Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /run,/dev/shm,/sys/fs/cgroup,/run/user/0,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/1074
[08/Dec/2020 12:35:33 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.02 mean:0.04 max:0.12 LIFE_MAX:0.09
[08/Dec/2020 12:45:33 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.02 mean:0.03 max:0.08 LIFE_MAX:0.09
[08/Dec/2020 12:55:34 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.03 mean:0.06 max:0.35 LIFE_MAX:0.09
[08/Dec/2020 12:55:49 +0000] 76583 MainThread throttling_logger INFO (14 skipped) Identified java component java8 with full version java version "1.8.0_251" Java(TM) SE Runtime Environment (build 1.8.0_251-b08) Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode) for requested version .
[08/Dec/2020 13:05:35 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.03 mean:0.04 max:0.09 LIFE_MAX:0.09
[08/Dec/2020 13:13:50 +0000] 76583 Monitor-HostMonitor throttling_logger ERROR (1 skipped) Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /run,/dev/shm,/sys/fs/cgroup,/run/user/0,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/1074
[08/Dec/2020 13:15:35 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.02 mean:0.04 max:0.08 LIFE_MAX:0.09
[08/Dec/2020 13:25:36 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.02 mean:0.04 max:0.11 LIFE_MAX:0.09
[08/Dec/2020 13:26:06 +0000] 76583 MainThread throttling_logger INFO (14 skipped) Identified java component java8 with full version java version "1.8.0_251" Java(TM) SE Runtime Environment (build 1.8.0_251-b08) Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode) for requested version .
[08/Dec/2020 13:35:36 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.02 mean:0.04 max:0.14 LIFE_MAX:0.09
[08/Dec/2020 13:45:37 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.02 mean:0.04 max:0.06 LIFE_MAX:0.09
[08/Dec/2020 13:55:37 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.02 mean:0.04 max:0.06 LIFE_MAX:0.09
[08/Dec/2020 13:56:22 +0000] 76583 MainThread throttling_logger INFO (14 skipped) Identified java component java8 with full version java version "1.8.0_251" Java(TM) SE Runtime Environment (build 1.8.0_251-b08) Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode) for requested version .
[08/Dec/2020 14:05:38 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.02 mean:0.04 max:0.16 LIFE_MAX:0.09
[08/Dec/2020 14:12:50 +0000] 76583 Monitor-HostMonitor throttling_logger ERROR Timed out waiting for worker process collecting filesystem usage to complete. This may occur if the host has an NFS or other remote filesystem that is not responding to requests in a timely fashion. Current nodev filesystems: /run,/dev/shm,/sys/fs/cgroup,/run/user/0,/run/cloudera-scm-agent/process,/run/cloudera-scm-agent/process,/run/user/1074
[08/Dec/2020 14:15:38 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.03 mean:0.05 max:0.10 LIFE_MAX:0.09
[08/Dec/2020 14:25:39 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.03 mean:0.05 max:0.07 LIFE_MAX:0.09
[08/Dec/2020 14:26:24 +0000] 76583 MainThread throttling_logger INFO (14 skipped) Identified java component java8 with full version java version "1.8.0_251" Java(TM) SE Runtime Environment (build 1.8.0_251-b08) Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode) for requested version .
[08/Dec/2020 14:35:40 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.02 mean:0.04 max:0.09 LIFE_MAX:0.09
[08/Dec/2020 14:45:40 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.02 mean:0.04 max:0.06 LIFE_MAX:0.09
[08/Dec/2020 14:55:41 +0000] 76583 MainThread heartbeat_tracker INFO HB stats (seconds): num:40 LIFE_MIN:0.02 min:0.03 mean:0.04 max:0.07 LIFE_MAX:0.09
[08/Dec/2020 14:56:26 +0000] 76583 MainThread throttling_logger INFO (14 skipped) Identified java component java8 with full version java version "1.8.0_251" Java(TM) SE Runtime Environment (build 1.8.0_251-b08) Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode) for requested version .
[
... View more
Labels:
- Labels:
-
Cloudera Manager
11-27-2020
11:06 PM
Like this we are getting some times HDFS canery good and some times HDFS Canary Bad
HDFS Canary Good
2 Still Concerning
Nov 27 12:15:53 PM
HDFS Canary Bad
Nov 27 12:15:08 PM
DataNode Health Concerning
Nov 27 11:58:47 AM
DataNode Health Bad
Nov 27 11:58:12 AM
DataNode Health Concerning
Nov 27 10:07:15 AM
DataNode Health Bad
Nov 27 10:07:00 AM
DataNode Health Concerning
Nov 27 9:29:35 AM
DataNode Health Bad
Nov 27 9:29:20 AM
DataNode Health Concerning
Nov 27 8:45:31 AM
DataNode Health Bad
Nov 27 8:45:06 AM
DataNode Health Concerning
Nov 26 10:03 PM
HDFS Canary Good
2 Still Bad
Nov 26 10:02:23 PM
DataNode Health Bad
Nov 26 10:02:18 PM
HDFS Canary Bad
Nov 26 10:01:42 PM
HDFS Canary Good
2 Still Concerning
Nov 26 8:01:53 PM
HDFS Canary Bad
Nov 26 8:01:03 PM
HDFS Canary Good
2 Still Concerning
Nov 26 6:16:18 PM
HDFS Canary Bad
Nov 26 6:15:38 PM
DataNode Health Concerning
Nov 26 4:45:01 PM
DataNode Health Bad We are finding this logs in service Monitor 12:06:35.706 PM INFO LDBPartitionManager
Expiring partition LDBPartitionMetadataWrapper{tableName=stream, partitionName=stream_2020-11-24T10:05:30.100Z, startTime=2020-11-24T10:05:30.100Z, endTime=2020-11-24T10:55:30.100Z, version=2, state=CLOSED}
12:06:35.706 PM INFO LDBPartitionMetadataStore
Setting partition state=DELETING for partition LDBPartitionMetadataWrapper{tableName=stream, partitionName=stream_2020-11-24T10:05:30.100Z, startTime=2020-11-24T10:05:30.100Z, endTime=2020-11-24T10:55:30.100Z, version=2, state=CLOSED}
12:06:35.717 PM INFO LDBPartitionManager
Couldn't close partition because it was already closed by another thread
12:06:35.718 PM INFO LDBPartitionMetadataStore
Deleting partition LDBPartitionMetadataWrapper{tableName=stream, partitionName=stream_2020-11-24T10:05:30.100Z, startTime=2020-11-24T10:05:30.100Z, endTime=2020-11-24T10:55:30.100Z, version=2, state=CLOSED}
12:06:39.374 PM INFO LDBTimeSeriesRollupManager
Running the LDBTimeSeriesRollupManager at 2020-11-27T10:06:39.374Z, forMigratedData=false
12:11:39.374 PM INFO LDBTimeSeriesRollupManager
Running the LDBTimeSeriesRollupManager at 2020-11-27T10:11:39.374Z, forMigratedData=false
12:11:39.375 PM INFO LDBTimeSeriesRollupManager
Starting rollup from raw to rollup=TEN_MINUTELY for rollupTimestamp=2020-11-27T10:10:00.000Z
12:11:41.505 PM INFO LDBTimeSeriesRollupManager
Finished rollup: duration=PT2.130S, numStreamsChecked=54046, numStreamsRolledUp=18786
12:13:40.962 PM INFO LDBResourceManager
Closed: 0 partitions 12:14:57.535 PM INFO DataStreamer
Exception in createBlockOutputStream blk_1086073148_12332434
java.net.SocketTimeoutException: 13000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.27:47442 remote=/172.27.12:9866]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:537)
at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1762)
at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
12:14:57.536 PM WARN DataStreamer
Abandoning BP-1768670017-172.-1592847899660:blk_1086073148_12332434 12:14:57.536 PM WARN DataStreamer
Abandoning BP-1768670017- -1592847899660:blk_1086073148_12332434
12:14:57.543 PM WARN DataStreamer
Excluding datanode DatanodeInfoWithStorage[172.27.129.28:9866,DS-211016d1-2920-4748-ba83-46a493759fe3,DISK]
12:15:05.558 PM INFO DataStreamer
Exception in createBlockOutputStream blk_1086073149_12332435
java.net.SocketTimeoutException: 8000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/172.27.129.30:56202 remote=/172.27.129.29:9866]
at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:164)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at java.io.FilterInputStream.read(FilterInputStream.java:83)
at org.apache.hadoop.hdfs.protocolPB.PBHelperClient.vintPrefixed(PBHelperClient.java:537)
at org.apache.hadoop.hdfs.DataStreamer.createBlockOutputStream(DataStreamer.java:1762)
at org.apache.hadoop.hdfs.DataStreamer.nextBlockOutputStream(DataStreamer.java:1679)
at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:716)
12:15:05.559 PM WARN DataStreamer
Abandoning BP-1768670017-172.27.0-1592847899660:blk_1086073149_12332435
12:15:05.568 PM WARN DataStreamer
Excluding datanode DatanodeInfoWithStorage[172.27.:9866,DS-5696ff0f-56d5-4dab-b0c3-5fbdde410da4,DISK]
12:15:05.573 PM WARN DataStreamer this are my cluster values. we thinking this values are issue
dfs.socket.timeout : 3000
dfs.datanode.socket.write.timeout :3000
we are found internet this values like this. this is the issue are any other
dfs.socket.timeout : 60000
dfs.datanode.socket.write.timeout :480000
... View more
Labels:
- Labels:
-
Cloudera Manager