Options
- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Solved
Go to solution
Problem with hive sink
Labels:
- Labels:
-
Apache Flume
-
Apache HCatalog
-
Apache Hive
Explorer
Created on ‎10-19-2016 11:03 PM - edited ‎09-16-2022 03:44 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2016-10-20 13:54:03,301 INFO org.apache.flume.sink.hive.HiveSink: sink1: Creating Writer to Hive end point : {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] } 2016-10-20 13:54:04,283 INFO hive.metastore: Trying to connect to metastore with URI thrift://10.0.201.34:9083 2016-10-20 13:54:04,315 INFO hive.metastore: Opened a connection to metastore, current connections: 1 2016-10-20 13:54:04,502 INFO hive.metastore: Connected to metastore. 2016-10-20 13:54:04,619 INFO hive.metastore: Trying to connect to metastore with URI thrift://10.0.201.34:9083 2016-10-20 13:54:04,619 INFO hive.metastore: Opened a connection to metastore, current connections: 2 2016-10-20 13:54:04,628 INFO hive.metastore: Connected to metastore. 2016-10-20 13:54:04,765 WARN org.apache.flume.sink.hive.HiveSink: sink1 : Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] } org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] } at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99) at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344) at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296) at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.hive.hcatalog.streaming.StreamingException: Cannot stream to table that has not been bucketed : {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] } at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.<init>(AbstractRecordWriter.java:69) at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:115) at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:92) at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:73) at org.apache.flume.sink.hive.HiveDelimitedTextSerializer.createRecordWriter(HiveDelimitedTextSerializer.java:65) at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:89) ... 6 more 2016-10-20 13:54:04,774 ERROR org.apache.flume.SinkRunner: Unable to deliver event. Exception follows. org.apache.flume.EventDeliveryException: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] } at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:268) at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:68) at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:147) at java.lang.Thread.run(Thread.java:722) Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] } at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99) at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:344) at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:296) at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:254) ... 3 more Caused by: org.apache.hive.hcatalog.streaming.StreamingException: Cannot stream to table that has not been bucketed : {metaStoreUri='thrift://10.0.201.34:9083,thrift://10.0.201.35:9083,thrift://10.0.201.36:9083', database='d_ec_telephone', table='m_tel_record', partitionVals=[] } at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.<init>(AbstractRecordWriter.java:69) at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:115) at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:92) at org.apache.hive.hcatalog.streaming.DelimitedInputWriter.<init>(DelimitedInputWriter.java:73) at org.apache.flume.sink.hive.HiveDelimitedTextSerializer.createRecordWriter(HiveDelimitedTextSerializer.java:65) at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:89) ... 6 more
1 ACCEPTED SOLUTION
Champion
Created ‎10-24-2016 10:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You have to bucket the hive table but not sorted .
Streaming to a unpartionated table is currently not supported.
In your case please check your table schema
table - m_tel_record
1 REPLY 1
Champion
Created ‎10-24-2016 10:39 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You have to bucket the hive table but not sorted .
Streaming to a unpartionated table is currently not supported.
In your case please check your table schema
table - m_tel_record
