getting error while submitting flume with hive sink

[#1]

Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.streaming.RecordWriter

my flume-conf file is as below:

agent1.channels = ch1 agent1.sinks = HIVE agent1.sources = sql-source # agent1.sources = sql-source agent1.sources.sql-source.type = org.keedio.flume.source.SQLSource agent1.sources.sql-source.channels = ch1 # # # URL to connect to database (currently only mysql is supported) #agent1.sources.sql-source.connection.url = jdbc:oracle:thin:@<ip>:<SID> agent1.sources.sql-source.hibernate.connection.url = jdbc:oracle:thin:@<IP>:<SID> # # # Database connection properties agent1.sources.sql-source.hibernate.connection.user = mis_user agent1.sources.sql-source.hibernate.connection.password = abcd_1234 agent1.sources.sql-source.table = ods_user.bb_rm_data #agent1.sources.sql-source.database = ods_user # agent1.sources.sql-source.columns.to.select = * agent1.sources.sql-source.max.rows = 100 # # Increment column properties # # agent1.sources.sql-source.incremental.column.name = id # # Increment value is from you want to start taking data from tables (0 will import entire table) # agent1.sources.sql-source.incremental.value = 0 #agent1.sources.sql-source.hibernate.dialect = org.keedio.flume.source.SQLServerCustomDialect # Query delay, each configured milisecond the query will be sent agent1.sources.sql-source.run.query.delay=5000000 # # Status file is used to save last readed row agent1.sources.sql-source.status.file.path = /home/dev_Rht agent1.sources.sql-source.status.file.name = sql-source.status # #channel properties agent1.channels.ch1.type = memory agent1.channels.ch1.capacity = 2000000 # agent1.sinks.HIVE.type = hive agent1.sinks.HIVE.channel = ch1 agent1.sinks.HIVE.hive.metastore = thrift://localhost:9083 agent1.sinks.HIVE.hive.database = default agent1.sinks.HIVE.hive.table = fi_bb6 agent1.sinks.HIVE.useLocalTimeStamp = false agent1.sinks.HIVE.round = true agent1.sinks.HIVE.roundValue = 10 agent1.sinks.HIVE.roundUnit = minute agent1.sinks.HIVE.serializer = DELIMITED agent1.sinks.HIVE.serializer.delimiter = "," agent1.sinks.HIVE.serializer.serdeSeparator = ',' agent1.sinks.HIVE.serializer.fieldnames = col1,col2,col3,col4,col5,col6,col7,col8,col9,col10,col11,col12,col13,col14,col15,col16,col17,col18,col19,col20,col21,col22,col23,col24,col25,col26,col27,col28,col29,col30

[#2]

adding conf file for readable purposeflume-hive-hortonworks.txt

pls reply as i am stuck.

[#3]

Make sure you have Hive and Hcatalog clients installed on the Flume node

[#4]

hi,

it seems that all the jars need to be present in flume/lib directory.

it works for me with this, but this doesnt seem to be ideal case.

any suggestion?

thanks,

Rishit Shah

[#5]

@Rishit shah can you follow my suggestion and install hive and hcat client tools on the flume nodes? It will dynamically link the jars to proper locations. Just need you to confirm.

[#6 · Accepted solution]

hi @Artem Ervits - they are already installed. the flume-env.sh set up was not proper. i added paths of lib and it got solved.

But, now i get error of memory capacity full. i had tried to increase channel capacity and transaction capacity as well but same error.

any advice?

thanks.

Rishit Shah

[#7]

capacity issues indicate you may be sending too much data or your destination cannot keep up with that much data. That's one of the bigger reasons Apache Nifi is a superior tool as it has backpressure built in as well as visual guides that you may have some kind of capacity issues. Try to throttle down how much data you're sending to the memory channel or see if you can expire stale data. Oh yeah, in Nifi you can expire data too.

[#8]

@Rishit shah

It seems that you found the response to your own question. I believe that the new question should be separate from the current question. We want to prevent open-ended questions. Could you open a different question and notify @Artem Ervits? This way, he can place his response and if that helps, please vote and accept his response or any response that makes sense to you on the new question. That question is worth it a full discussion and it is of larger interest.