Support Questions

Find answers, ask questions, and share your expertise

HDP 2.3.2 and HBase 0.98 Compatibility

New Contributor

We are currently in the process of upgrading to HDP 2.3.2 (not finalized) and I've noticed something in the HDP 2.3.2 documentation that is concerning. Here https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/install... it states that to use the HBase Sink with HDP 2.3.2 you need to have HBase 0.98 running but this is not what comes with the platform. My question is two parts 1) Does this documentation indicate that the HBase Sink is not compatible with HBase version 1.1.2 that comes with HDP 2.3.2? (or is it saying HBase must be at least version 0.98?) 2) When installing HDP 2.3.2 can you opt to not upgrade HBase and leave it at the current version?

1 ACCEPTED SOLUTION

Explorer

The doc should really not say anything about the HBase version there... as all HDP components are tested work with each other within a release. However, in this case "0.98 and above" would have been a better way to phrase it.

View solution in original post

12 REPLIES 12

I would assume that the the documentation is wrong as we do significant cross-component testing for each release (and I see we have some tests for Flume writing to HBase).

"At least HBase 0.98" is probably the correct phrasing.

New Contributor

@Josh Elser So if I am receiving the following error then I can assume that I incorrectly compiled my code against the wrong version of HBase?

18 Aug 2016 19:02:24,873 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hbase.HBaseSink.process:356)  - Failed to commit transaction.Transaction rolled back.
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V
	at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:379)
	at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:374)
	at org.apache.flume.sink.hbase.HBaseSink.runPrivileged(HBaseSink.java:427)
	at org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:374)
	at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:344)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)

I would assume so. I see this method does exist in HDP-2.3.2.

Contributor
Josh,

org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V does not exists in 2.3.2 
org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)[Lorg.apache.hadoop.hbase.client.Put exists.
That looks like incompatibility issue.

Oh my. Thanks for correcting me, Vlad! I am totally wrong and missed the return type (just looked at the method arguments).

Explorer

If you are compiling Flume yourself.. you probably got something wrong there. You should be using the Flume bundled in HDP instead. It is tested to work along with the other components bundled with HDP (HBase, Hive, Hdfs etc.) ... so you wont run into issues like this.

New Contributor

This was the hint in the right direction for me solving my issue. My intentions were to use Flume as compiled on the server but I had a custom environment configuration overriding the location of the Flume install to use for running the agent and was able to locate the file causing this issue after much digging.

Super Collaborator

For #2, this is currently not possible.

You

need to upgrade the whole stack.

Super Collaborator

Here is the code in HBaseSink.java:

if (r instanceof Put) {

((Put) r).setWriteToWAL(enableWal);

Here is code in Put.java of HDP 2.3:

public Put setWriteToWAL(boolean write) {

return (Put) super.setWriteToWAL(write);

In pom.xml of Flume in HDP 2.3:

<hbaseversion>0.94.2</hbaseversion>

Planning to raise an internal JIRA for this incompatibility.

New Contributor

@Ted Yu So to use the version of Flume supplied with HDP 2.3 I need to be using HBase 0.94. Alternatively I could potentially use Flume 1.6 with the 1.1.2 version of HBase supplied with HDP 2.3?

Super Collaborator

I have sent an email to Roshan who is more familiar with Flume component.

Explorer

The doc should really not say anything about the HBase version there... as all HDP components are tested work with each other within a release. However, in this case "0.98 and above" would have been a better way to phrase it.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.