Archives of Support Questions (Read Only)

This is an archived board for historical reference. Information and links may no longer be available or relevant
Announcements
This board is archived and read-only for historical reference. To ask a new question, please post a new topic on the appropriate active board.

Problem with hive sink

avatar
Explorer

2016-10-20 13:54:03,301 INFO org.apache.flume.sink.hive.HiveSink: sink1: Creating Writer to Hive end point : {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
2016-10-20 13:54:04,283 INFO hive.metastore: Trying to connect to metastore with URI thrift://10.0.201.34:9083
2016-10-20 13:54:04,315 INFO hive.metastore: Opened a connection to metastore, current connections: 1
2016-10-20 13:54:04,502 INFO hive.metastore: Connected to metastore.
2016-10-20 13:54:04,619 INFO hive.metastore: Trying to connect to metastore with URI thrift://10.0.201.34:9083
2016-10-20 13:54:04,619 INFO hive.metastore: Opened a connection to metastore, current connections: 2
2016-10-20 13:54:04,628 INFO hive.metastore: Connected to metastore.
2016-10-20 13:54:04,765 WARN org.apache.flume.sink.hive.HiveSink: sink1 : Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296)
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.hive.hcatalog.streaming.StreamingException: Cannot stream to table that has not been bucketed : {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.<init>(AbstractRecordWriter.java:69)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:115)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:92)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:73)
at org.apache.flume.sink.hive.HiveDelimitedTextSerializer.createRecordWriter(HiveDelimitedTextSerializer.java:65)
at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:89)
... 6 more
2016-10-20 13:54:04,774 ERROR org.apache.flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:268)
at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68)
at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147)
at java.lang.Thread.run(Thread.java:722)
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344)
at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296)
at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254)
... 3 more
Caused by: org.apache.hive.hcatalog.streaming.StreamingException: Cannot stream to table that has not been bucketed : {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] }
at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.<init>(AbstractRecordWriter.java:69)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:115)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:92)
at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:73)
at org.apache.flume.sink.hive.HiveDelimitedTextSerializer.createRecordWriter(HiveDelimitedTextSerializer.java:65)
at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:89)
... 6 more
1 ACCEPTED SOLUTION

avatar
Champion

You have to bucket the hive table but not sorted .

Streaming to a unpartionated  table is currently not supported.

In your case please check your table schema 

table - m_tel_record

 

 

 

View solution in original post

1 REPLY 1

avatar
Champion

You have to bucket the hive table but not sorted .

Streaming to a unpartionated  table is currently not supported.

In your case please check your table schema 

table - m_tel_record