Really sorry for the late reply. This somehow fell off my radar. To answer your question, for the exchange node 6, you can sum  the TotalBytesSent for each DatastreamSender for the total  bytes shuffled between all instances of F00 to all instances of exchange node 6.   Impala currently doesn't consider network bandwidth to decide whether it will compress row batches or not. It's true that if the network bandwidth is plentiful, we may save some CPU time by skipping the compression/decompression step. For multi-rack deployment, the network bandwidth is likely bound by the throughput of top-of-rack switch so compression usually helps in this case. Please let me know if I may have missed other questions. ... View more

Thanks for your response. I used a statically linked Impala this time. create-test-configuration.sh give me following output:   philipp@philipp:/media/philipp/f5de6362-e4f-4f0e-ab1a-4b13bedebfc2/Impala$ bin/create-test-configuration.sh The minikdc is not running. Creating node-3 at /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/cluster/cdh5/node-3 node-3 will use ports DATANODE_PORT=31000, DATANODE_HTTP_PORT=31010, DATANODE_IPC_PORT=31020, DATANODE_HTTPS_PORT=31030, NODEMANAGER_PORT=31100, NODEMANAGER_LOCALIZER_PORT=31120, NODEMANAGER_WEBUI_PORT=31140, KUDU_TS_RPC_PORT=31200, and KUDU_TS_WEBUI_PORT=31300 Creating node-2 at /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/cluster/cdh5/node-2 node-2 will use ports DATANODE_PORT=31001, DATANODE_HTTP_PORT=31011, DATANODE_IPC_PORT=31021, DATANODE_HTTPS_PORT=31031, NODEMANAGER_PORT=31101, NODEMANAGER_LOCALIZER_PORT=31121, NODEMANAGER_WEBUI_PORT=31141, KUDU_TS_RPC_PORT=31201, and KUDU_TS_WEBUI_PORT=31301 Creating node-1 at /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/cluster/cdh5/node-1 node-1 will use ports DATANODE_PORT=31002, DATANODE_HTTP_PORT=31012, DATANODE_IPC_PORT=31022, DATANODE_HTTPS_PORT=31032, NODEMANAGER_PORT=31102, NODEMANAGER_LOCALIZER_PORT=31122, NODEMANAGER_WEBUI_PORT=31142, KUDU_TS_RPC_PORT=31202, and KUDU_TS_WEBUI_PORT=31302 Config dir: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources Current user: philipp Metastore DB: hive_impala /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala Linking core-site.xml from local cluster Linking hdfs-site.xml from local cluster Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/hive-site.xml Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/log4j.properties Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/hive-log4j.properties Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/hbase-site.xml Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/authz-policy.ini Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/sentry-site.xml /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala Completed config generation Searching for auxiliary tests, workloads, and datasets (if any exist). No auxiliary tests found at: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/../Impala-auxiliary-tests/testdata/workloads No auxiliary tests found at: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/../Impala-auxiliary-tests/testdata/datasets No auxiliary tests found at: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/../Impala-auxiliary-tests/tests Then, start-impala-cluster.py results in:    Starting State Store logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/statestored.INFO Starting Catalog Service logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/catalogd.INFO Starting Impala Daemon logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/impalad.INFO Starting Impala Daemon logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/impalad_node1.INFO Starting Impala Daemon logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/impalad_node2.INFO MainThread: Found 3 impalad/1 statestored/1 catalogd process(es) MainThread: Getting num_known_live_backends from philipp:25000 MainThread: Debug webpage not yet available. MainThread: Debug webpage not yet available. MainThread: Debug webpage not yet available. MainThread: Debug webpage not yet available. MainThread: Debug webpage not yet available. MainThread: Debug webpage not yet available. MainThread: Debug webpage not yet available. ... /testdata/run-all.sh:   Killing running services... Starting cluster services... Stopping kudu Stopping kms Stopping yarn Stopping hdfs Starting hdfs (Web UI - http://localhost:5070) Namenode started Starting yarn (Web UI - http://localhost:8088) Starting kms (Web UI - http://localhost:9600) Waiting for ntpd to synchronize... OK! Starting kudu (Web UI - http://localhost:8051) hdfs-datanode is not running on node-1 hdfs-datanode is not running on node-2 hdfs-datanode is not running on node-3 Error in /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/bin/run-mini-dfs.sh at line 40: $IMPALA_HOME/testdata/cluster/admin start_cluster Error in testdata/bin/run-all.sh at line 42: tee ${IMPALA_CLUSTER_LOGS_DIR}/run-mini-dfs.log     Unfortunately there's still the error with start_cluster.  ... View more