Problem with hive sink

IvanLeung — Fri, 16 Sep 2022 10:44:55 GMT

2016-10-20 13:54:03,301 INFO org.apache.flume.sink.hive.HiveSink: sink1: Creating Writer to Hive end point : {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
2016-10-20 13:54:04,283 INFO hive.metastore: Trying to connect to metastore with URI thrift://10.0.201.34:9083
2016-10-20 13:54:04,315 INFO hive.metastore: Opened a connection to metastore, current connections: 1
2016-10-20 13:54:04,502 INFO hive.metastore: Connected to metastore.
2016-10-20 13:54:04,619 INFO hive.metastore: Trying to connect to metastore with URI thrift://10.0.201.34:9083
2016-10-20 13:54:04,619 INFO hive.metastore: Opened a connection to metastore, current connections: 2
2016-10-20 13:54:04,628 INFO hive.metastore: Connected to metastore.
2016-10-20 13:54:04,765 WARN org.apache.flume.sink.hive.HiveSink: sink1 : Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296)
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.hive.hcatalog.streaming.StreamingException: Cannot stream to table that has not been bucketed : {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.<init>(AbstractRecordWriter.java:69)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:115)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:92)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:73)
at org.apache.flume.sink.hive.HiveDelimitedTextSerializer.createRecordWriter(HiveDelimitedTextSerializer.java:65)
at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:89)
... 6 more
2016-10-20 13:54:04,774 ERROR org.apache.flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:268)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296)
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254)
... 3 more
Caused by: org.apache.hive.hcatalog.streaming.StreamingException: Cannot stream to table that has not been bucketed : {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.<init>(AbstractRecordWriter.java:69)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:115)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:92)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:73)
at org.apache.flume.sink.hive.HiveDelimitedTextSerializer.createRecordWriter(HiveDelimitedTextSerializer.java:65)
at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:89)
... 6 more

Re: Problem with hive sink

csguna — Tue, 25 Oct 2016 05:39:00 GMT

You have to bucket the hive table but not sorted .

Streaming to a unpartionated table is currently not supported.

In your case please check your table schema

table - m_tel_record

question Problem with hive sink in Archives of Support Questions (Read Only)

Problem with hive sink

Re: Problem with hive sink