Reply
New Contributor
Posts: 3
Registered: ‎07-28-2016

Kudu on old CPU

Hi all,

 

I'm trying to install Kudu on a small physical cluster (4 hosts with RHEL6.8) for evaluation purposes only.

The cluster is based on CDH5.12 and is managed by Cloudera Express 5.12.

 

Unfortunately Kudu processes fail to start with the following message:

 

F0718 11:16:20.229274 22398 init.cc:81] Check failed: _s.ok() Bad status: Not implemented: The CPU on this system (Intel(R) Xeon(R) CPU           X7460  @ 2.66GHz^@) does not support the SSE4.2 instruction set which is required for running Kudu. If you are running inside a VM, you may need to enable SSE4.2 pass-through.

 

Actually above mentioned processor support SSE up to 4.1 but lacks 4.2.

 

Any suggestion to work around the issue?

 

Thanks in advance

--

Domenico

Posts: 1,903
Kudos: 435
Solutions: 305
Registered: ‎07-31-2013

Re: Kudu on old CPU

Currently, Apache Kudu has a hard dependency on SSE 4.2 and SSE3 as noted here: https://kudu.apache.org/docs/quickstart.html#_failing_with_lack_of_sse4_2_support_when_running_insid..., and there's no way to circumvent that requirement.

IIUC, SSE4.2 instructions seem to be used in Kudu for computing CRC32 checksums and also in performing varint encoding.
New Contributor
Posts: 3
Registered: ‎07-28-2016

Re: Kudu on old CPU

After further investigation I came to the following Jira issue:

 

https://issues.apache.org/jira/browse/KUDU-1210

 

Apparently I'm the first (and only?) complainer :-)

 

Thank you for the quick answer.

 

I wonder if I could compile Kudu from source and make some "hack" to get rid of the dependency and still have a pretty much working sw...

Any clue?

New Contributor
Posts: 3
Registered: ‎07-28-2016

Re: Kudu on old CPU

That might perhaps help somebody else.

I downloaded the Cloudera Kudu tarball from:
http://archive.cloudera.com/kudu/kudu/5/kudu-1.4.0-cdh5.12.0.tar.gz

and followed the installation guide from Apache Kudu project:
https://kudu.apache.org/docs/installation.html#build_from_source

The first part of the build procedure (after required packages installation) is triggered by:

build-support/enable_devtoolset.sh thirdparty/build-if-necessary.sh

It downloads and builds a bunch of dependencies, including https://github.com/rurban/crcutil.
As you can see this library implements CRC checksum using SSE4.2 CRC32 instruction (the one missing on my CPU).
Luckily you can set CRCUTIL_USE_MM_CRC32=0 to simulate the instruction in software.
All I need was to edit kudu-1.4.0-cdh5.12.0/thirdparty/src/crcutil-440ba7babeff77ffad992df3a10c767f184e946e/autogen.sh
I'm almost sure there is a better way to configure the compiler but it worked and that's enough.
Now you need to run again the build of dependencies after the tweak to get the patched library.

Before running the build of Kudu you need to modify the source file:
kudu-1.4.0-cdh5.12.0/src/kudu/util/init.cc

to neutralize the check on CPU capabilities. I simply changed the original:

if (!cpu.has_sse42()) {
    return BadCPUStatus(cpu, "SSE4.2");

into

if (!cpu.has_sse41()) {
    return BadCPUStatus(cpu, "SSE4.1");

but YMMV.

Then you can finaly run the build of Kudu.
It takes a while to complete and at the end of the day you can find all you need in build/latest/bin and build/latest/lib/exported.

And now the tricky part.
I overwrited some files of the Kudu Parcel installed on the cluster with the ones just built.
Specifically:

/opt/cloudera/parcels/KUDU/lib/kudu/bin-release/kudu
/opt/cloudera/parcels/KUDU/lib/kudu/sbin-release/kudu-master
/opt/cloudera/parcels/KUDU/lib/kudu/sbin-release/kudu-tserver
/opt/cloudera/parcels/KUDU/lib64/libkudu_client.so.0.1.0

Furthermore I had to overwrite the kudu shared library linked by impalad:

/opt/cloudera/parcels/CDH/lib/impala/lib/libkudu_client.so.0

Finally from Cloudera Express I started Kudu (this time worked like a charm) and restarted Impala.

Of course is pure hacking.
Kudu relies on the missing CPU instruction to achieve good performance.
Not to mention the mess on CDH and Kudu installation.
But as I said is a PoC cluster, so who cares?
I have a running Kudu to play with.
Mission accomplished!