HDP 2.3.2 and HBase 0.98 Compatibility

[#1]

We are currently in the process of upgrading to HDP 2.3.2 (not finalized) and I've noticed something in the HDP 2.3.2 documentation that is concerning. Here https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/install... it states that to use the HBase Sink with HDP 2.3.2 you need to have HBase 0.98 running but this is not what comes with the platform. My question is two parts 1) Does this documentation indicate that the HBase Sink is not compatible with HBase version 1.1.2 that comes with HDP 2.3.2? (or is it saying HBase must be at least version 0.98?) 2) When installing HDP 2.3.2 can you opt to not upgrade HBase and leave it at the current version?

[#2]

I would assume that the the documentation is wrong as we do significant cross-component testing for each release (and I see we have some tests for Flume writing to HBase).

"At least HBase 0.98" is probably the correct phrasing.

[#3]

@Josh Elser So if I am receiving the following error then I can assume that I incorrectly compiled my code against the wrong version of HBase?

18 Aug 2016 19:02:24,873 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hbase.HBaseSink.process:356)  - Failed to commit transaction.Transaction rolled back.
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V
	at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:379)
	at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:374)
	at org.apache.flume.sink.hbase.HBaseSink.runPrivileged(HBaseSink.java:427)
	at org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:374)
	at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:344)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)

[#4]

I would assume so. I see this method does exist in HDP-2.3.2.

[#5]

Josh,

org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V does not exists in 2.3.2 
org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)[Lorg.apache.hadoop.hbase.client.Put exists.
That looks like incompatibility issue.

[#6]

Oh my. Thanks for correcting me, Vlad! I am totally wrong and missed the return type (just looked at the method arguments).

[#7]

If you are compiling Flume yourself.. you probably got something wrong there. You should be using the Flume bundled in HDP instead. It is tested to work along with the other components bundled with HDP (HBase, Hive, Hdfs etc.) ... so you wont run into issues like this.

[#8]

This was the hint in the right direction for me solving my issue. My intentions were to use Flume as compiled on the server but I had a custom environment configuration overriding the location of the Flume install to use for running the agent and was able to locate the file causing this issue after much digging.

[#9]

For #2, this is currently not possible.

You

need to upgrade the whole stack.

[#10]

Here is the code in HBaseSink.java:

if (r instanceof Put) {

((Put) r).setWriteToWAL(enableWal);

Here is code in Put.java of HDP 2.3:

public Put setWriteToWAL(boolean write) {

return (Put) super.setWriteToWAL(write);

In pom.xml of Flume in HDP 2.3:

<hbaseversion>0.94.2</hbaseversion>

Planning to raise an internal JIRA for this incompatibility.