question Re: HDP 2.3.2 and HBase 0.98 Compatibility in Support Questions

HDP 2.3.2 and HBase 0.98 Compatibility

sarah_etzler — Mon, 22 Aug 2016 20:16:01 GMT

We are currently in the process of upgrading to HDP 2.3.2 (not finalized) and I've noticed something in the HDP 2.3.2 documentation that is concerning. Here https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/installing_flume.html it states that to use the HBase Sink with HDP 2.3.2 you need to have HBase 0.98 running but this is not what comes with the platform. My question is two parts 1) Does this documentation indicate that the HBase Sink is not compatible with HBase version 1.1.2 that comes with HDP 2.3.2? (or is it saying HBase must be at least version 0.98?) 2) When installing HDP 2.3.2 can you opt to not upgrade HBase and leave it at the current version?

Re: HDP 2.3.2 and HBase 0.98 Compatibility

elserj — Mon, 22 Aug 2016 21:34:41 GMT

I would assume that the the documentation is wrong as we do significant cross-component testing for each release (and I see we have some tests for Flume writing to HBase).

"At least HBase 0.98" is probably the correct phrasing.

Re: HDP 2.3.2 and HBase 0.98 Compatibility

sarah_etzler — Mon, 22 Aug 2016 21:42:32 GMT

@Josh Elser So if I am receiving the following error then I can assume that I incorrectly compiled my code against the wrong version of HBase?

18 Aug 2016 19:02:24,873 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hbase.HBaseSink.process:356)  - Failed to commit transaction.Transaction rolled back.
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V
	at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:379)
	at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:374)
	at org.apache.flume.sink.hbase.HBaseSink.runPrivileged(HBaseSink.java:427)
	at org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:374)
	at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:344)
	at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
	at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
	at java.lang.Thread.run(Thread.java:745)

Re: HDP 2.3.2 and HBase 0.98 Compatibility

elserj — Mon, 22 Aug 2016 21:52:51 GMT

I would assume so. I see this method does exist in HDP-2.3.2.

Re: HDP 2.3.2 and HBase 0.98 Compatibility

tyu — Mon, 22 Aug 2016 21:54:26 GMT

For #2, this is currently not possible.

You

need to upgrade the whole stack.

Re: HDP 2.3.2 and HBase 0.98 Compatibility

tyu — Mon, 22 Aug 2016 21:59:30 GMT

Here is the code in HBaseSink.java:

if (r instanceof Put) {

((Put) r).setWriteToWAL(enableWal);

Here is code in Put.java of HDP 2.3:

public Put setWriteToWAL(boolean write) {

return (Put) super.setWriteToWAL(write);

In pom.xml of Flume in HDP 2.3:

Planning to raise an internal JIRA for this incompatibility.

Re: HDP 2.3.2 and HBase 0.98 Compatibility

sarah_etzler — Mon, 22 Aug 2016 22:13:53 GMT

@Ted Yu So to use the version of Flume supplied with HDP 2.3 I need to be using HBase 0.94. Alternatively I could potentially use Flume 1.6 with the 1.1.2 version of HBase supplied with HDP 2.3?

Re: HDP 2.3.2 and HBase 0.98 Compatibility

tyu — Mon, 22 Aug 2016 22:48:14 GMT

I have sent an email to Roshan who is more familiar with Flume component.

Re: HDP 2.3.2 and HBase 0.98 Compatibility

roshan1 — Wed, 24 Aug 2016 04:53:38 GMT

If you are compiling Flume yourself.. you probably got something wrong there. You should be using the Flume bundled in HDP instead. It is tested to work along with the other components bundled with HDP (HBase, Hive, Hdfs etc.) ... so you wont run into issues like this.

Re: HDP 2.3.2 and HBase 0.98 Compatibility

roshan1 — Wed, 24 Aug 2016 04:56:12 GMT

The doc should really not say anything about the HBase version there... as all HDP components are tested work with each other within a release. However, in this case "0.98 and above" would have been a better way to phrase it.

Re: HDP 2.3.2 and HBase 0.98 Compatibility

vrodionov — Thu, 25 Aug 2016 23:34:59 GMT

Josh,

org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V does not exists in 2.3.2 
org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)[Lorg.apache.hadoop.hbase.client.Put exists.
That looks like incompatibility issue.

Re: HDP 2.3.2 and HBase 0.98 Compatibility

elserj — Fri, 26 Aug 2016 00:20:31 GMT

Oh my. Thanks for correcting me, Vlad! I am totally wrong and missed the return type (just looked at the method arguments).

Re: HDP 2.3.2 and HBase 0.98 Compatibility

sarah_etzler — Fri, 26 Aug 2016 21:48:05 GMT

This was the hint in the right direction for me solving my issue. My intentions were to use Flume as compiled on the server but I had a custom environment configuration overriding the location of the Flume install to use for running the agent and was able to locate the file causing this issue after much digging.