Impala refuses to start in CDH 5.0.0

[#1]

Hi,

I just upgraded our cluster from the CDH 5 beta to the newly available version 5.0.0-1, using parcels.

When restarting the cluster after upgrading, the Impala daemons fail to start. The error message in the log is as follows:

E0403 09:25:58.360676 59211 impala-server.cc:208] ERROR: block location tracking is not properly enabled because
  - dfs.client.file-block-storage-locations.timeout.millis is too low. It should be at least 10 seconds.
E0403 09:25:58.361166 59211 impala-server.cc:210] Aborting Impala Server startup due to improper configuration

When searching for a configuration called 'dfs.client.file-block-storage-locations.timeout.millis' nothing shows up. What I can find is instead the setting 'dfs.client.file-block-storage-locations.timeout', which is set to 3000 seconds.

I am guessing that this configuration setting has changed name at some point in time, and that my software versions are somehow out of sync. Not really sure which one I should be looking at though, so any suggestion on how to resolve are appreciated.

Our current package versions are:

Bigtop-Tomcat (CDH 5 only)	CDH5	0.7.0+cdh5.0.0+0
Cloudera Manager Agent	Not applicable	5.0.0-beta-2
Cloudera Manager Management Daemons	Not applicable	5.0.0-beta-2
Crunch (CDH 5 only)	CDH5	0.9.0+cdh5.0.0+21
Flume NG	CDH5	1.4.0+cdh5.0.0+111
Hadoop	CDH5	2.3.0+cdh5.0.0+548
MapReduce 1	CDH5	2.3.0+cdh5.0.0+548
HDFS	CDH5	2.3.0+cdh5.0.0+548
HttpFS	CDH5	2.3.0+cdh5.0.0+548
MapReduce 2	CDH5	2.3.0+cdh5.0.0+548
YARN	CDH5	2.3.0+cdh5.0.0+548
HBase	CDH5	0.96.1.1+cdh5.0.0+60
Lily HBase Indexer	CDH5	1.3+cdh5.0.0+38
Hive	CDH5	0.12.0+cdh5.0.0+308
HCatalog	CDH5	0.12.0+cdh5.0.0+308
Hue	CDH5	3.5.0+cdh5.0.0+365
Impala	CDH5	1.3.0+cdh5.0.0+0

Regards

\Knut Nordin

[#2 · Accepted solution]

Hi Knut,

You need to upgrade CM to the GA release of 5.0.0 as well. CM 5.0.0 has renamed this property and changed the default to 10 seconds so Impala wil start correctly.

There's a few manual steps, when you do this upgrade, but it seems like you've already figured them out since you got the CDH 5 upgrade working.

Also be sure to hard_restart your CM agents when you do the CM upgrade, as described in the CM 4 to 5 upgrade guide (which is sort of relevant when doing beta 2 to GA upgrade):

http://www.cloudera.com/content/cloudera-content/cloudera-docs/CM5/latest/Cloudera-Manager-Administr...

Thanks,

Darren

[#3]

Thanks!

Upgrading CM helped. For some reason I did not get any new packages when I first tried running yum upgrade for CM, so I figured the beta2 one was still supposed to be used. Not sure what I did there really, but at least things worked better this time.

\Knut