Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

HDP 2.3.2 and HBase 0.98 Compatibility

avatar

We are currently in the process of upgrading to HDP 2.3.2 (not finalized) and I've noticed something in the HDP 2.3.2 documentation that is concerning. Here https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/install... it states that to use the HBase Sink with HDP 2.3.2 you need to have HBase 0.98 running but this is not what comes with the platform. My question is two parts 1) Does this documentation indicate that the HBase Sink is not compatible with HBase version 1.1.2 that comes with HDP 2.3.2? (or is it saying HBase must be at least version 0.98?) 2) When installing HDP 2.3.2 can you opt to not upgrade HBase and leave it at the current version?

1 ACCEPTED SOLUTION

avatar
Explorer

The doc should really not say anything about the HBase version there... as all HDP components are tested work with each other within a release. However, in this case "0.98 and above" would have been a better way to phrase it.

View solution in original post

12 REPLIES 12

avatar
Super Guru

I would assume that the the documentation is wrong as we do significant cross-component testing for each release (and I see we have some tests for Flume writing to HBase).

"At least HBase 0.98" is probably the correct phrasing.

avatar

@Josh Elser So if I am receiving the following error then I can assume that I incorrectly compiled my code against the wrong version of HBase?

18 Aug 2016 19:02:24,873 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hbase.HBaseSink.process:356)  - Failed to commit transaction.Transaction rolled back.
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V
	at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:379)
	at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:374)
	at org.apache.flume.sink.hbase.HBaseSink.runPrivileged(HBaseSink.java:427)
	at org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:374)
	at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:344)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)

avatar
Super Guru

I would assume so. I see this method does exist in HDP-2.3.2.

avatar
Rising Star
Josh,

org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V does not exists in 2.3.2 
org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)[Lorg.apache.hadoop.hbase.client.Put exists.
That looks like incompatibility issue.

avatar
Super Guru

Oh my. Thanks for correcting me, Vlad! I am totally wrong and missed the return type (just looked at the method arguments).

avatar
Explorer

If you are compiling Flume yourself.. you probably got something wrong there. You should be using the Flume bundled in HDP instead. It is tested to work along with the other components bundled with HDP (HBase, Hive, Hdfs etc.) ... so you wont run into issues like this.

avatar

This was the hint in the right direction for me solving my issue. My intentions were to use Flume as compiled on the server but I had a custom environment configuration overriding the location of the Flume install to use for running the agent and was able to locate the file causing this issue after much digging.

avatar
Master Collaborator

For #2, this is currently not possible.

You

need to upgrade the whole stack.

avatar
Master Collaborator

Here is the code in HBaseSink.java:

if (r instanceof Put) {

((Put) r).setWriteToWAL(enableWal);

Here is code in Put.java of HDP 2.3:

public Put setWriteToWAL(boolean write) {

return (Put) super.setWriteToWAL(write);

In pom.xml of Flume in HDP 2.3:

<hbaseversion>0.94.2</hbaseversion>

Planning to raise an internal JIRA for this incompatibility.