question getting error while submitting flume with hive sink in Support Questions

getting error while submitting flume with hive sink

rishit606 — Sat, 17 Dec 2016 15:15:21 GMT

Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.streaming.RecordWriter

my flume-conf file is as below:

agent1.channels = ch1 agent1.sinks = HIVE agent1.sources = sql-source # agent1.sources = sql-source agent1.sources.sql-source.type = org.keedio.flume.source.SQLSource agent1.sources.sql-source.channels = ch1 # # # URL to connect to database (currently only mysql is supported) #agent1.sources.sql-source.connection.url = jdbc:oracle:thin:@<ip>:<SID> agent1.sources.sql-source.hibernate.connection.url = jdbc:oracle:thin:@<IP>:<SID> # # # Database connection properties agent1.sources.sql-source.hibernate.connection.user = mis_user agent1.sources.sql-source.hibernate.connection.password = abcd_1234 agent1.sources.sql-source.table = ods_user.bb_rm_data #agent1.sources.sql-source.database = ods_user # agent1.sources.sql-source.columns.to.select = * agent1.sources.sql-source.max.rows = 100 # # Increment column properties # # agent1.sources.sql-source.incremental.column.name = id # # Increment value is from you want to start taking data from tables (0 will import entire table) # agent1.sources.sql-source.incremental.value = 0 #agent1.sources.sql-source.hibernate.dialect = org.keedio.flume.source.SQLServerCustomDialect # Query delay, each configured milisecond the query will be sent agent1.sources.sql-source.run.query.delay=5000000 # # Status file is used to save last readed row agent1.sources.sql-source.status.file.path = /home/dev_Rht agent1.sources.sql-source.status.file.name = sql-source.status # #channel properties agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 2000000 # agent1.sinks.HIVE.type = hive agent1.sinks.HIVE.channel = ch1 agent1.sinks.HIVE.hive.metastore = thrift://localhost:9083 agent1.sinks.HIVE.hive.database = default agent1.sinks.HIVE.hive.table = fi_bb6 agent1.sinks.HIVE.useLocalTimeStamp = false agent1.sinks.HIVE.round = true agent1.sinks.HIVE.roundValue = 10 agent1.sinks.HIVE.roundUnit = minute agent1.sinks.HIVE.serializer = DELIMITED agent1.sinks.HIVE.serializer.delimiter = "," agent1.sinks.HIVE.serializer.serdeSeparator = ',' agent1.sinks.HIVE.serializer.fieldnames = col1,col2,col3,col4,col5,col6,col7,col8,col9,col10,col11,col12,col13,col14,col15,col16,col17,col18,col19,col20,col21,col22,col23,col24,col25,col26,col27,col28,col29,col30

Re: getting error while submitting flume with hive sink

rishit606 — Sat, 17 Dec 2016 15:18:34 GMT

adding conf file for readable purposeflume-hive-hortonworks.txt

pls reply as i am stuck.

Re: getting error while submitting flume with hive sink

aervits — Sun, 18 Dec 2016 03:03:43 GMT

Make sure you have Hive and Hcatalog clients installed on the Flume node

Re: getting error while submitting flume with hive sink

rishit606 — Tue, 20 Dec 2016 18:02:50 GMT

hi,

it seems that all the jars need to be present in flume/lib directory.

it works for me with this, but this doesnt seem to be ideal case.

any suggestion?

thanks,

Rishit Shah

Re: getting error while submitting flume with hive sink

aervits — Wed, 21 Dec 2016 03:46:07 GMT

@Rishit shah can you follow my suggestion and install hive and hcat client tools on the flume nodes? It will dynamically link the jars to proper locations. Just need you to confirm.

Re: getting error while submitting flume with hive sink

rishit606 — Fri, 23 Dec 2016 13:54:08 GMT

hi @Artem Ervits - they are already installed. the flume-env.sh set up was not proper. i added paths of lib and it got solved.

But, now i get error of memory capacity full. i had tried to increase channel capacity and transaction capacity as well but same error.

any advice?

thanks.

Rishit Shah

Re: getting error while submitting flume with hive sink

aervits — Mon, 26 Dec 2016 23:12:25 GMT

capacity issues indicate you may be sending too much data or your destination cannot keep up with that much data. That's one of the bigger reasons Apache Nifi is a superior tool as it has backpressure built in as well as visual guides that you may have some kind of capacity issues. Try to throttle down how much data you're sending to the memory channel or see if you can expire stale data. Oh yeah, in Nifi you can expire data too.

Re: getting error while submitting flume with hive sink

cstanca — Tue, 27 Dec 2016 04:10:50 GMT

@Rishit shah

It seems that you found the response to your own question. I believe that the new question should be separate from the current question. We want to prevent open-ended questions. Could you open a different question and notify @Artem Ervits? This way, he can place his response and if that helps, please vote and accept his response or any response that makes sense to you on the new question. That question is worth it a full discussion and it is of larger interest.