Mini cluster setup CDH5.13.1

[#1]

Hi! I'm trying to setup a mini cluster with CDH5.13.1 for some experiments but unfortunately I always receive this error when running "run-all.sh":

impala@impala:/Impala$ testdata/bin/run-all.sh
Killing running services...
Starting cluster services...
Stopping kudu
Stopping kms
Stopping yarn
Stopping hdfs
Starting hdfs (Web UI - http://localhost:5070)
Failed to start hdfs-datanode. The end of the log (/Impala/testdata/cluster/cdh5/node-1/var/log/hdfs-datanode.out) is:
Failed to start hdfs-datanode. The end of the log (/Impala/testdata/cluster/cdh5/node-3/var/log/hdfs-datanode.out) is:
Failed to start hdfs-datanode. The end of the log (/Impala/testdata/cluster/cdh5/node-2/var/log/hdfs-datanode.out) is:
Namenode started
Error in /Impala/testdata/bin/run-mini-dfs.sh at line 40: $IMPALA_HOME/testdata/cluster/admin start_cluster
Error in testdata/bin/run-all.sh at line 42: tee ${IMPALA_CLUSTER_LOGS_DIR}/run-mini-dfs.log

As you can see, all node logs are empty. run-mini-dfs.log provides following information:

Error in /Impala/testdata/bin/run-mini-dfs.sh at line 40: $IMPALA_HOME/testdata/cluster/admin start_cluster

I don't know what's wrong with the start_cluster function. 

Further information:
Build: ./buildall.sh -notests -so -start_minicluster
Ubuntu 14.04

I hope you can help me to fix the problem.

[#2]

Does anyone have an idea what the problem might be? I'm stuck at this point 😕 

[#3]

I get the same error with Ubuntu 16.04 and the newest Impala version.

[#4]

Not having seen detailed logs, I'm just throwing out two quick ideas:

1. The minicluster looks quite broken; I wonder if it is a configuration problem. I would first try running bin/create-test-configuration.sh to fix the configuration:

  $ # After sourcing bin/impala-config.sh:

  $ bin/create-test-configuration.sh

then try starting the minicluster.

2. Generally I would suggest avoiding the "-so" switch for buildall.sh: there are multiple outstanding Jiras about the shared-object build not working properly. Building statically linked Impala binaries gives you less artifacts to copy, although the build takes longer.

Let me know if this was any help,

  - Laszlo

[#5]

Thanks for your response. I used a statically linked Impala this time. create-test-configuration.sh give me following output:

philipp@philipp:/media/philipp/f5de6362-e4f-4f0e-ab1a-4b13bedebfc2/Impala$ bin/create-test-configuration.sh
The minikdc is not running.
Creating node-3 at /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/cluster/cdh5/node-3
node-3 will use ports DATANODE_PORT=31000, DATANODE_HTTP_PORT=31010, DATANODE_IPC_PORT=31020, DATANODE_HTTPS_PORT=31030, NODEMANAGER_PORT=31100, NODEMANAGER_LOCALIZER_PORT=31120, NODEMANAGER_WEBUI_PORT=31140, KUDU_TS_RPC_PORT=31200, and KUDU_TS_WEBUI_PORT=31300
Creating node-2 at /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/cluster/cdh5/node-2
node-2 will use ports DATANODE_PORT=31001, DATANODE_HTTP_PORT=31011, DATANODE_IPC_PORT=31021, DATANODE_HTTPS_PORT=31031, NODEMANAGER_PORT=31101, NODEMANAGER_LOCALIZER_PORT=31121, NODEMANAGER_WEBUI_PORT=31141, KUDU_TS_RPC_PORT=31201, and KUDU_TS_WEBUI_PORT=31301
Creating node-1 at /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/cluster/cdh5/node-1
node-1 will use ports DATANODE_PORT=31002, DATANODE_HTTP_PORT=31012, DATANODE_IPC_PORT=31022, DATANODE_HTTPS_PORT=31032, NODEMANAGER_PORT=31102, NODEMANAGER_LOCALIZER_PORT=31122, NODEMANAGER_WEBUI_PORT=31142, KUDU_TS_RPC_PORT=31202, and KUDU_TS_WEBUI_PORT=31302
Config dir: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources
Current user: philipp
Metastore DB: hive_impala
/media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala
Linking core-site.xml from local cluster
Linking hdfs-site.xml from local cluster
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/hive-site.xml
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/log4j.properties
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/hive-log4j.properties
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/hbase-site.xml
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/authz-policy.ini
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/sentry-site.xml
/media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala
Completed config generation
Searching for auxiliary tests, workloads, and datasets (if any exist).
No auxiliary tests found at: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/../Impala-auxiliary-tests/testdata/workloads
No auxiliary tests found at: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/../Impala-auxiliary-tests/testdata/datasets
No auxiliary tests found at: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/../Impala-auxiliary-tests/tests

Then, start-impala-cluster.py results in: 

Starting State Store logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/statestored.INFO
Starting Catalog Service logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/catalogd.INFO
Starting Impala Daemon logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/impalad.INFO
Starting Impala Daemon logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/impalad_node1.INFO
Starting Impala Daemon logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/impalad_node2.INFO
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting num_known_live_backends from philipp:25000
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
...

/testdata/run-all.sh:

Killing running services...
Starting cluster services...
Stopping kudu
Stopping kms
Stopping yarn
Stopping hdfs
Starting hdfs (Web UI - http://localhost:5070)
Namenode started
Starting yarn (Web UI - http://localhost:8088)
Starting kms (Web UI - http://localhost:9600)
Waiting for ntpd to synchronize... OK!
Starting kudu (Web UI - http://localhost:8051)

hdfs-datanode is not running on node-1 
hdfs-datanode is not running on node-2 
hdfs-datanode is not running on node-3
Error in /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/bin/run-mini-dfs.sh at line 40: $IMPALA_HOME/testdata/cluster/admin start_cluster
Error in testdata/bin/run-all.sh at line 42: tee ${IMPALA_CLUSTER_LOGS_DIR}/run-mini-dfs.log

Unfortunately there's still the error with start_cluster.