Member since
08-19-2016
5
Posts
0
Kudos Received
0
Solutions
08-26-2016
02:48 PM
This was the hint in the right direction for me solving my issue. My intentions were to use Flume as compiled on the server but I had a custom environment configuration overriding the location of the Flume install to use for running the agent and was able to locate the file causing this issue after much digging.
... View more
08-22-2016
03:13 PM
@Ted Yu So to use the version of Flume supplied with HDP 2.3 I need to be using HBase 0.94. Alternatively I could potentially use Flume 1.6 with the 1.1.2 version of HBase supplied with HDP 2.3?
... View more
08-22-2016
02:42 PM
@Josh Elser So if I am receiving the following error then I can assume that I incorrectly compiled my code against the wrong version of HBase? 18 Aug 2016 19:02:24,873 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hbase.HBaseSink.process:356) - Failed to commit transaction.Transaction rolled back.
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V
at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:379)
at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:374)
at org.apache.flume.sink.hbase.HBaseSink.runPrivileged(HBaseSink.java:427)
at org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:374)
at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:344)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
... View more
08-22-2016
01:16 PM
We are currently in the process of upgrading to HDP 2.3.2 (not finalized) and I've noticed something in the HDP 2.3.2 documentation that is concerning. Here https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_installing_manually_book/content/installing_flume.html it states that to use the HBase Sink with HDP 2.3.2 you need to have HBase 0.98 running but this is not what comes with the platform.
My question is two parts
1) Does this documentation indicate that the HBase Sink is not compatible with HBase version 1.1.2 that comes with HDP 2.3.2? (or is it saying HBase must be at least version 0.98?)
2) When installing HDP 2.3.2 can you opt to not upgrade HBase and leave it at the current version?
... View more
Labels:
- Labels:
-
Apache Flume
-
Apache HBase
08-19-2016
01:05 AM
We recently upgraded from HDP 2.1.4 to HDP 2.3.2 and I have some Flume agents that use HBaseSink to publish logs to HBase. After the upgrade I've been receiving the error found below. I compiled my customer event serializer for my events against HBase version 1.1.2.2.3.2.0-2950 and Hadoop version 2.7.1.2.3.2.0-2950 but something still isn't right. Is there something special I need to do to get this working?
HDP 2.3.2 contains HBase 1.1.2 + Hadoop 2.7.1 + Flume 1.5.2 18 Aug 2016 19:02:24,873 ERROR [SinkRunner-PollingRunner-DefaultSinkProcessor] (org.apache.flume.sink.hbase.HBaseSink.process:356) - Failed to commit transaction.Transaction rolled back.
java.lang.NoSuchMethodError: org.apache.hadoop.hbase.client.Put.setWriteToWAL(Z)V
at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:379)
at org.apache.flume.sink.hbase.HBaseSink$3.run(HBaseSink.java:374)
at org.apache.flume.sink.hbase.HBaseSink.runPrivileged(HBaseSink.java:427)
at org.apache.flume.sink.hbase.HBaseSink.putEventsAndCommit(HBaseSink.java:374)
at org.apache.flume.sink.hbase.HBaseSink.process(HBaseSink.java:344)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:745)
... View more
Labels:
- Labels:
-
Apache Flume
-
Apache HBase