Created on 10-22-201505:33 PM - edited 09-16-202201:32 AM
Various open-source tools and applications are available to do performance, scalability and reliability testing for various popular HDP components. Here is a list of some of the popular ones:
Measure the I/O performance of HDFS in your cluster. Source code for the tool can be found here.
Applies load on the namenode by performing continuous read, write, rename and delete operations on small files. Source code for this tool can be found here.
Synthetic Load Generator
The synthetic load generator (SLG) is a tool for testing NameNode behavior under different client loads. The user can generate different mixes of read, write, and list requests by specifying the probabilities of read and write. The user controls the intensity of the load by adjusting parameters for the number of worker threads and the delay between operations. More information on the tool can be found here.
Measure performance by measuring time to sort 1TB of data. The test runs in three steps. First one is TeraGen to generate the dataset, second one is the TeraSort to sort the generated data and third one is TeraValidate to verify the sort order is correct. You can change to use a different data size.
Runs a job multiple times and takes average of all runs. Source code for the tool can be found here.
GridMix submits a mix of synthetic jobs, modeling a profile mined from production loads. More information on the tool can be found here.
Performance evaluation of HBase under pre-defined workloads. More information can be found here.
HBase Performance Evaluation
Script used for evaluating HBase performance and scalability. Runs a HBase
client that steps through one of a set of hardcoded tests or 'experiments'
(e.g. a random reads test, a random writes test, etc.). More information can be found here.
A command-line utility that reads, writes, and verifies data. Unlike
PerformanceEvaluation, this tool validates the data written, and supports
simultaneously writing and reading the same set of keys. Source for the tool can be found here.
A utility to injects faults in a running cluster. More information can be found here.
TPC Benchmarks (TPCDS & TPCH)
TPCDS and TPCH are analytic benchmarks that model generally applicable aspects of decision support system. Automated scripts to run TPC benchmarks at scale including the converted queries can be found here.
PigMix is a set of queries used to test pig performance. More information can be found here.