Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Mini cluster setup CDH5.13.1

Mini cluster setup CDH5.13.1

Explorer

Hi! I'm trying to setup a mini cluster with CDH5.13.1 for some experiments but unfortunately I always receive this error when running "run-all.sh":

impala@impala:/Impala$ testdata/bin/run-all.sh
Killing running services...
Starting cluster services...
Stopping kudu
Stopping kms
Stopping yarn
Stopping hdfs
Starting hdfs (Web UI - http://localhost:5070)
Failed to start hdfs-datanode. The end of the log (/Impala/testdata/cluster/cdh5/node-1/var/log/hdfs-datanode.out) is:
Failed to start hdfs-datanode. The end of the log (/Impala/testdata/cluster/cdh5/node-3/var/log/hdfs-datanode.out) is:
Failed to start hdfs-datanode. The end of the log (/Impala/testdata/cluster/cdh5/node-2/var/log/hdfs-datanode.out) is:
Namenode started
Error in /Impala/testdata/bin/run-mini-dfs.sh at line 40: $IMPALA_HOME/testdata/cluster/admin start_cluster
Error in testdata/bin/run-all.sh at line 42: tee ${IMPALA_CLUSTER_LOGS_DIR}/run-mini-dfs.log

As you can see, all node logs are empty. run-mini-dfs.log provides following information:

Error in /Impala/testdata/bin/run-mini-dfs.sh at line 40: $IMPALA_HOME/testdata/cluster/admin start_cluster

I don't know what's wrong with the start_cluster function. 

 

Further information:
Build: ./buildall.sh -notests -so -start_minicluster
Ubuntu 14.04

I hope you can help me to fix the problem.

4 REPLIES 4

Re: Mini cluster setup CDH5.13.1

Explorer

Does anyone have an idea what the problem might be? I'm stuck at this point :/ 

Re: Mini cluster setup CDH5.13.1

Explorer

I get the same error with Ubuntu 16.04 and the newest Impala version.

Re: Mini cluster setup CDH5.13.1

Cloudera Employee

Not having seen detailed logs, I'm just throwing out two quick ideas:

1. The minicluster looks quite broken; I wonder if it is a configuration problem. I would first try running bin/create-test-configuration.sh to fix the configuration:

 

  $ # After sourcing bin/impala-config.sh:

  $ bin/create-test-configuration.sh

 

then try starting the minicluster.

 

2. Generally I would suggest avoiding the "-so" switch for buildall.sh: there are multiple outstanding Jiras about the shared-object build not working properly. Building statically linked Impala binaries gives you less artifacts to copy, although the build takes longer.

 

Let me know if this was any help,

 

  - Laszlo

Re: Mini cluster setup CDH5.13.1

Explorer

Thanks for your response. I used a statically linked Impala this time. create-test-configuration.sh give me following output:

 

philipp@philipp:/media/philipp/f5de6362-e4f-4f0e-ab1a-4b13bedebfc2/Impala$ bin/create-test-configuration.sh
The minikdc is not running.
Creating node-3 at /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/cluster/cdh5/node-3
node-3 will use ports DATANODE_PORT=31000, DATANODE_HTTP_PORT=31010, DATANODE_IPC_PORT=31020, DATANODE_HTTPS_PORT=31030, NODEMANAGER_PORT=31100, NODEMANAGER_LOCALIZER_PORT=31120, NODEMANAGER_WEBUI_PORT=31140, KUDU_TS_RPC_PORT=31200, and KUDU_TS_WEBUI_PORT=31300
Creating node-2 at /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/cluster/cdh5/node-2
node-2 will use ports DATANODE_PORT=31001, DATANODE_HTTP_PORT=31011, DATANODE_IPC_PORT=31021, DATANODE_HTTPS_PORT=31031, NODEMANAGER_PORT=31101, NODEMANAGER_LOCALIZER_PORT=31121, NODEMANAGER_WEBUI_PORT=31141, KUDU_TS_RPC_PORT=31201, and KUDU_TS_WEBUI_PORT=31301
Creating node-1 at /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/cluster/cdh5/node-1
node-1 will use ports DATANODE_PORT=31002, DATANODE_HTTP_PORT=31012, DATANODE_IPC_PORT=31022, DATANODE_HTTPS_PORT=31032, NODEMANAGER_PORT=31102, NODEMANAGER_LOCALIZER_PORT=31122, NODEMANAGER_WEBUI_PORT=31142, KUDU_TS_RPC_PORT=31202, and KUDU_TS_WEBUI_PORT=31302
Config dir: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources
Current user: philipp
Metastore DB: hive_impala
/media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala
Linking core-site.xml from local cluster
Linking hdfs-site.xml from local cluster
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/hive-site.xml
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/log4j.properties
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/hive-log4j.properties
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/hbase-site.xml
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/authz-policy.ini
Generated /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/fe/src/test/resources/sentry-site.xml
/media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala
Completed config generation
Searching for auxiliary tests, workloads, and datasets (if any exist).
No auxiliary tests found at: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/../Impala-auxiliary-tests/testdata/workloads
No auxiliary tests found at: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/../Impala-auxiliary-tests/testdata/datasets
No auxiliary tests found at: /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/../Impala-auxiliary-tests/tests

Then, start-impala-cluster.py results in: 

 

Starting State Store logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/statestored.INFO
Starting Catalog Service logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/catalogd.INFO
Starting Impala Daemon logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/impalad.INFO
Starting Impala Daemon logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/impalad_node1.INFO
Starting Impala Daemon logging to /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/logs/cluster/impalad_node2.INFO
MainThread: Found 3 impalad/1 statestored/1 catalogd process(es)
MainThread: Getting num_known_live_backends from philipp:25000
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
MainThread: Debug webpage not yet available.
...

/testdata/run-all.sh:

 

Killing running services...
Starting cluster services...
Stopping kudu
Stopping kms
Stopping yarn
Stopping hdfs
Starting hdfs (Web UI - http://localhost:5070)
Namenode started
Starting yarn (Web UI - http://localhost:8088)
Starting kms (Web UI - http://localhost:9600)
Waiting for ntpd to synchronize... OK!
Starting kudu (Web UI - http://localhost:8051)

hdfs-datanode is not running on node-1 
hdfs-datanode is not running on node-2 
hdfs-datanode is not running on node-3
Error in /media/philipp/f5de6362-e43f-4f0e-ab1a-4b13bedebfc2/Impala/testdata/bin/run-mini-dfs.sh at line 40: $IMPALA_HOME/testdata/cluster/admin start_cluster
Error in testdata/bin/run-all.sh at line 42: tee ${IMPALA_CLUSTER_LOGS_DIR}/run-mini-dfs.log

 

 

Unfortunately there's still the error with start_cluster.