Community Articles
Find and share helpful community-sourced technical articles
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.
Labels (1)
Not applicable

Device Behavior Analytics allows administrators to detect straggler nodes/disks which reduce the cluster performance. Finding and fixing these nodes can improve the overall cluster throughput. This feature will be available in HDP-2.6.1 and later. There are two parts to Device Behavior Analytics – Slow Datanode Detection and Slow Disk Detection.

SLOW DATANODE (PEER) DETECTION

Datanode will collect latency statistics about their peers during the normal operation of the datanode write pipeline. These latency stats are used to detect outliers among the peers. Slow peer detection is not performed unless the datanode has statistics of at least 10 peers. Namenode maintains the list of slow peers and administrators can read it via JMX. Datanodes also expose the average write latency of their peers through Datanode JMX.

SLOW DISK DETECTION

Each datanode will collect I/O statistics from all its disks. We can configure the percentage of file I/O events to be sampled to limit the performance impact of I/O profiling. Slow disk detection is not performed unless the datanode has at least 5 disks. Slow disk information is available via Datanode JMX. Namenode also exposes the slowest disks in the cluster via Namenode JMX.

ENABLING DEVICE BEHAVIOR ANALYTICS

To enable Device Behavior Analytics, the following configurations must be set.

  • dfs.datanode.peer.stats.enabled

Set to true to enable Slow Datanode Detection.

  • dfs.datanode.fileio.profiling.sampling.percentage

Set to a value between 1 and 100 to enable Slow Disk detection. This setting controls the fraction of file IO events which will be sampled for profiling disk statistics. Setting this value to 100 will sample all disk IO. Sampling a large fraction of disk IO events might have a small performance impact.

  • dfs.datanode.outliers.report.interval

This setting allows you to control how frequently datanodes will report their peer latencies to the Namenode via heartbeat and the frequency of disk outlier detection by the datanode. The default value for this setting is 30 minutes.

These settings should be added to hdfs-site.xml. If it is an Ambari installed cluster, then the settings can be added via custom hdfs-site.xml.

SAMPLE JMX OUTPUTS

  • Sample Namenode JMX output reporting slow nodes

  • Sample Datanode JMX output showing average write latency of peers

  • Sample Datanode JMX output showing slow disks

The Datanode JMX output above reports disk3 and disk4 as outliers for the Datanode. The JMX also reports the latencies and number of operations per volume in another metric. The sample JMX output below for DataNodeVolume information for disk3 shows high average latencies for metadata and write IO operations.

  • Sample Namenode JMX output reporting slow disks and their latencies

Please follow the blog post link here for a detailed explanation about Device Behavior Analytics.

629 Views
Don't have an account?
Coming from Hortonworks? Activate your account here
Version history
Revision #:
1 of 1
Last update:
‎06-22-2017 12:09 AM
Updated by:
 
Contributors
Top Kudoed Authors