Reply
New Contributor
Posts: 2
Registered: ‎07-20-2017

Flume Hive Sink use mysql error

Hi,I tried to use Flume1.7.0 and Hive2.1.1 to save data For Hadoop to mapreduce. But I had errors when I  send data from java app (started by shell),the error msg is below:

17/07/20 15:34:13 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
17/07/20 15:34:13 INFO instrumentation.MonitoredCounterGroup: Component type: SOURCE, name: r1 started
17/07/20 15:34:13 INFO source.AvroSource: Avro source r1 started.
17/07/20 15:34:59 INFO ipc.NettyServer: [id: 0x41dadf6d, /192.168.100.223:54324 => /192.168.100.223:4141] OPEN
17/07/20 15:34:59 INFO ipc.NettyServer: [id: 0x41dadf6d, /192.168.100.223:54324 => /192.168.100.223:4141] BOUND: /192.168.100.223:4141
17/07/20 15:34:59 INFO ipc.NettyServer: [id: 0x41dadf6d, /192.168.100.223:54324 => /192.168.100.223:4141] CONNECTED: /192.168.100.223:54324
17/07/20 15:34:59 INFO hive.HiveSink: k1: Creating Writer to Hive end point : {metaStoreUri='thrift://192.168.100.223:9083', database='default', table='traffic_a', partitionVals=[20170720] }
17/07/20 15:35:00 INFO ipc.NettyServer: [id: 0x41dadf6d, /192.168.100.223:54324 :> /192.168.100.223:4141] DISCONNECTED
17/07/20 15:35:00 INFO ipc.NettyServer: [id: 0x41dadf6d, /192.168.100.223:54324 :> /192.168.100.223:4141] UNBOUND
17/07/20 15:35:00 INFO ipc.NettyServer: [id: 0x41dadf6d, /192.168.100.223:54324 :> /192.168.100.223:4141] CLOSED
17/07/20 15:35:00 INFO ipc.NettyServer: Connection to /192.168.100.223:54324 disconnected.
17/07/20 15:35:00 INFO conf.HiveConf: Found configuration file null
17/07/20 15:35:01 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/07/20 15:35:01 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.100.223:9083
17/07/20 15:35:01 INFO hive.metastore: Opened a connection to metastore, current connections: 1
17/07/20 15:35:02 INFO hive.metastore: Connected to metastore.
17/07/20 15:35:02 INFO hive.metastore: Trying to connect to metastore with URI thrift://192.168.100.223:9083
17/07/20 15:35:02 INFO hive.metastore: Opened a connection to metastore, current connections: 2
17/07/20 15:35:02 INFO hive.metastore: Connected to metastore.
17/07/20 15:35:04 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/31b54131-2886-4f94-862d-371a950882fe
17/07/20 15:35:04 INFO session.SessionState: Created local directory: /tmp/root/31b54131-2886-4f94-862d-371a950882fe
17/07/20 15:35:04 INFO session.SessionState: Created HDFS directory: /tmp/hive/root/31b54131-2886-4f94-862d-371a950882fe/_tmp_space.db
17/07/20 15:35:04 INFO ql.Driver: Compiling command(queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8): use default
17/07/20 15:35:06 INFO ql.Driver: Semantic Analysis Completed
17/07/20 15:35:06 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
17/07/20 15:35:06 INFO ql.Driver: Completed compiling command(queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8); Time taken: 1.764 seconds
17/07/20 15:35:06 INFO lockmgr.DbTxnManager: Setting lock request transaction to txnid:0 for queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8
17/07/20 15:35:06 INFO ql.Driver: Executing command(queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8): use default
17/07/20 15:35:06 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
17/07/20 15:35:06 INFO ql.Driver: Completed executing command(queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8); Time taken: 0.033 seconds
OK
17/07/20 15:35:06 INFO ql.Driver: OK
17/07/20 15:35:06 INFO lockmgr.DbLockManager: releaseLocks: []
17/07/20 15:35:06 INFO ql.Driver: Compiling command(queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8): alter table traffic_a add if not exists partition  ( dt='20170720' )
17/07/20 15:35:07 INFO ql.Driver: Semantic Analysis Completed
17/07/20 15:35:07 INFO ql.Driver: Returning Hive schema: Schema(fieldSchemas:null, properties:null)
17/07/20 15:35:07 INFO ql.Driver: Completed compiling command(queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8); Time taken: 0.412 seconds
17/07/20 15:35:07 INFO lockmgr.DbTxnManager: Setting lock request transaction to txnid:0 for queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8
17/07/20 15:35:07 INFO lockmgr.DbLockManager: Requesting: queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8 LockRequest(component:[LockComponent(type:SHARED_READ, level:TABLE, dbname:default, tablename:traffic_a, operationType:NO_TXN, isAcid:true)], txnid:0, user:root, hostname:master, agentInfo:root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8)
17/07/20 15:35:07 INFO lockmgr.DbLockManager: Response to queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8 LockResponse(lockid:6, state:ACQUIRED)
17/07/20 15:35:07 INFO lockmgr.DbTxnManager: Started heartbeat with delay/interval = 150000/150000 MILLISECONDS for query: root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8
17/07/20 15:35:07 INFO ql.Driver: Executing command(queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8): alter table traffic_a add if not exists partition  ( dt='20170720' )
17/07/20 15:35:07 INFO ql.Driver: Starting task [Stage-0:DDL] in serial mode
17/07/20 15:35:07 INFO ql.Driver: Completed executing command(queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8); Time taken: 0.407 seconds
OK
17/07/20 15:35:07 INFO ql.Driver: OK
17/07/20 15:35:07 INFO lockmgr.DbTxnManager: Stopped heartbeat for query: root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8
17/07/20 15:35:07 INFO lockmgr.DbLockManager: releaseLocks: [lockid:6 queryId=root_20170720153504_b96f2d29-9360-4354-9238-fccf85a57fe8 txnid:0]
17/07/20 15:35:07 INFO session.SessionState: Deleted directory: /tmp/hive/root/31b54131-2886-4f94-862d-371a950882fe on fs with scheme hdfs
17/07/20 15:35:07 INFO session.SessionState: Deleted directory: /tmp/root/31b54131-2886-4f94-862d-371a950882fe on fs with scheme file
17/07/20 15:35:07 WARN hive.HiveSink: k1 : Failed connecting to EndPoint {metaStoreUri='thrift://192.168.100.223:9083', database='default', table='traffic_a', partitionVals=[20170720] }
org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://192.168.100.223:9083', database='default', table='traffic_a', partitionVals=[20170720] }
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
        at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:343)
        at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://192.168.100.223:9083', database='default', table='traffic_a', partitionVals=[20170720] }
        at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:383)
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:86)
        ... 6 more
Caused by: org.apache.hive.hcatalog.streaming.StreamingException: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
        at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:456)
        at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:376)
        ... 7 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:434)
        ... 8 more
Caused by: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
        at com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:436)
        at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:430)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:550)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
        at com.sun.proxy.$Proxy10.close(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2265)
        at com.sun.proxy.$Proxy10.close(Unknown Source)
        at org.apache.hadoop.hive.ql.metadata.Hive.close(Hive.java:387)
        at org.apache.hadoop.hive.ql.metadata.Hive.access$000(Hive.java:160)
        at org.apache.hadoop.hive.ql.metadata.Hive$1.remove(Hive.java:180)
        at org.apache.hadoop.hive.ql.metadata.Hive.closeCurrent(Hive.java:354)
        at org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1539)
        at org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.createPartitionIfNotExists(HiveEndPoint.java:486)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:318)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:278)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:215)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:192)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:105)
        at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:379)
        at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:376)
        at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more
17/07/20 15:35:07 ERROR flume.SinkRunner: Unable to deliver event. Exception follows.
org.apache.flume.EventDeliveryException: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://192.168.100.223:9083', database='default', table='traffic_a', partitionVals=[20170720] }
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:267)
        at org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
        at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://192.168.100.223:9083', database='default', table='traffic_a', partitionVals=[20170720] }
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:99)
        at org.apache.flume.sink.hive.HiveSink.getOrCreateWriter(HiveSink.java:343)
        at org.apache.flume.sink.hive.HiveSink.drainOneBatch(HiveSink.java:295)
        at org.apache.flume.sink.hive.HiveSink.process(HiveSink.java:253)
        ... 3 more
Caused by: org.apache.flume.sink.hive.HiveWriter$ConnectException: Failed connecting to EndPoint {metaStoreUri='thrift://192.168.100.223:9083', database='default', table='traffic_a', partitionVals=[20170720] }
        at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:383)
        at org.apache.flume.sink.hive.HiveWriter.<init>(HiveWriter.java:86)
        ... 6 more
Caused by: org.apache.hive.hcatalog.streaming.StreamingException: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
        at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:456)
        at org.apache.flume.sink.hive.HiveWriter.newConnection(HiveWriter.java:376)
        ... 7 more
Caused by: java.util.concurrent.ExecutionException: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)
        at org.apache.flume.sink.hive.HiveWriter.timedCall(HiveWriter.java:434)
        ... 8 more
Caused by: java.lang.NoSuchMethodError: com.facebook.fb303.FacebookService$Client.sendBaseOneway(Ljava/lang/String;Lorg/apache/thrift/TBase;)V
        at com.facebook.fb303.FacebookService$Client.send_shutdown(FacebookService.java:436)
        at com.facebook.fb303.FacebookService$Client.shutdown(FacebookService.java:430)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.close(HiveMetaStoreClient.java:550)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:154)
        at com.sun.proxy.$Proxy10.close(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient$SynchronizedHandler.invoke(HiveMetaStoreClient.java:2265)
        at com.sun.proxy.$Proxy10.close(Unknown Source)
        at org.apache.hadoop.hive.ql.metadata.Hive.close(Hive.java:387)
        at org.apache.hadoop.hive.ql.metadata.Hive.access$000(Hive.java:160)
        at org.apache.hadoop.hive.ql.metadata.Hive$1.remove(Hive.java:180)
        at org.apache.hadoop.hive.ql.metadata.Hive.closeCurrent(Hive.java:354)
        at org.apache.hadoop.hive.ql.session.SessionState.close(SessionState.java:1539)
        at org.apache.hadoop.hive.cli.CliSessionState.close(CliSessionState.java:66)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.createPartitionIfNotExists(HiveEndPoint.java:486)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:318)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:278)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:215)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:192)
        at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:105)
        at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:379)
        at org.apache.flume.sink.hive.HiveWriter$8.call(HiveWriter.java:376)
        at org.apache.flume.sink.hive.HiveWriter$11.call(HiveWriter.java:428)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        ... 1 more

But when I start the Flume and Hive,they work well,see info below:

[root@master conf]# flume-ng agent --conf conf --conf-file AvroHive.conf --name a1
-Dflume.root.logger=INFO,console
Info: Including Hadoop libraries found via (/usr/local/hadoop/bin/hadoop) for HDFS access
Info: Including Hive libraries found via (/usr/local/hadoop/hive-2.1.1-bin) for Hive access
Info: Including HCatalog libraries found via (/usr/local/hadoop/hive-2.1.1-bin/hcatalog) for Hive access
+ exec /usr/java/jdk1.8.0_91/bin/java -Xmx20m -cp 'conf:/usr/local/hadoop/flume-1.7.0-bin/lib/*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/hdfs:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/contrib/capacity-scheduler/*.jar:/usr/local/hadoop/hive-2.1.1-bin/lib/*:/usr/local/hadoop/hive-2.1.1-bin/hcatalog/share/hcatalog/*' -Djava.library.path=:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib org.apache.flume.node.Application --conf-file AvroHive.conf --name a1
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop/flume-1.7.0-bin/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.21.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hadoop/hive-2.1.1-bin/lib/log4j-slf4j-impl-2.4.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
17/07/20 15:34:12 INFO node.PollingPropertiesFileConfigurationProvider: Configuration provider starting
17/07/20 15:34:12 INFO node.PollingPropertiesFileConfigurationProvider: Reloading configuration file:AvroHive.conf
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Added sinks: k1 Agent: a1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:12 INFO conf.FlumeConfiguration: Processing:k1
17/07/20 15:34:13 INFO conf.FlumeConfiguration: Post-validation flume configuration contains configuration for agents: [a1]
17/07/20 15:34:13 INFO node.AbstractConfigurationProvider: Creating channels
17/07/20 15:34:13 INFO channel.DefaultChannelFactory: Creating instance of channel c1 type memory
17/07/20 15:34:13 INFO node.AbstractConfigurationProvider: Created channel c1
17/07/20 15:34:13 INFO source.DefaultSourceFactory: Creating instance of source r1, type avro
17/07/20 15:34:13 INFO sink.DefaultSinkFactory: Creating instance of sink: k1, type: hive
17/07/20 15:34:13 INFO node.AbstractConfigurationProvider: Channel c1 connected to [r1, k1]
17/07/20 15:34:13 INFO node.Application: Starting new configuration:{ sourceRunners:{r1=EventDrivenSourceRunner: { source:Avro source r1: { bindAddress: master, port: 4141 } }} sinkRunners:{k1=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@534ba5bf counterGroup:{ name:null counters:{} } }} channels:{c1=org.apache.flume.channel.MemoryChannel{name: c1}} }
17/07/20 15:34:13 INFO node.Application: Starting Channel c1
17/07/20 15:34:13 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: CHANNEL, name: c1: Successfully registered new MBean.
17/07/20 15:34:13 INFO instrumentation.MonitoredCounterGroup: Component type: CHANNEL, name: c1 started
17/07/20 15:34:13 INFO node.Application: Starting Sink k1
17/07/20 15:34:13 INFO node.Application: Starting Source r1
17/07/20 15:34:13 INFO source.AvroSource: Starting Avro source r1: { bindAddress: master, port: 4141 }...
17/07/20 15:34:13 INFO instrumentation.MonitoredCounterGroup: Monitored counter group for type: SINK, name: k1: Successfully registered new MBean.
17/07/20 15:34:13 INFO instrumentation.MonitoredCounterGroup: Component type: SINK, name: k1 started
17/07/20 15:34:13 INFO hive.HiveSink: k1: Hive Sink k1 started

my hive-site.xml is :

<configuration>	
	  <property>
        <name>hive.metastore.local</name>
        <value>false</value>
    </property>	
    <property>
        <name>javax.jdo.option.ConnectionURL</name>
		<value>jdbc:mysql://localhost:3306/hivedb?characterEncoding=UTF-8</value>       
    </property>
    <property>
        <name>javax.jdo.option.ConnectionDriverName</name>
        <value>com.mysql.jdbc.Driver</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionUserName</name>
        <value>root</value>
    </property>
    <property>
        <name>javax.jdo.option.ConnectionPassword</name>
        <value>root</value>
    </property>  
	
	<property>  
		<name>hive.metastore.warehouse.dir</name>  
		<value>/user/hive/warehouse</value>  
	</property>
	<property>
        <name>fs.defaultFS</name>
        <value>hdfs://192.168.100.223:9000</value>
    </property>
	
	<property>  
		<name>hive.metastore.uris</name>  
		<value>thrift://192.168.100.223:9083</value>  
	</property>
	
	<property>
        <name>hive.server2.thrift.port</name>
        <value>10000</value>
    </property>
    <property>
        <name>beeline.hs2.connection.user</name>
        <value>root</value>
        </property>
     <property>
        <name>beeline.hs2.connection.password</name>
        <value>123456</value>
     </property>
	 
	 <property>
        <name>datanucleus.autoCreateSchema</name>
        <value>true</value>
    </property>
    <property>
        <name>datanucleus.autoStartMechanism</name>
        <value>SchemaTable</value>
    </property>
    <property>
        <name>datanucleus.schema.autoCreateTables</name>
        <value>true</value>
    </property>
	
	<property>
		<name>hadoop.proxyuser.hadoop.hosts</name>
		<value>*</value>
	</property>
	<property>
		<name>hadoop.proxyuser.hadoop.groups</name>
		<value>*</value>
	</property>
	
	<property>
		<name>hive.exec.scratchdir</name>
		<value>/user/hive/tmp</value>    
	</property>
	
	<property>
		<name>hive.querylog.location</name>
		<value>/user/hive/log</value>   
	</property>
</configuration>

my flume configuration is:

a1.sinks = k1
a1.sources = r1
a1.channels = c1

a1.sources.r1.type = avro
a1.sources.r1.channels = c1
a1.sources.r1.bind = master
a1.sources.r1.port = 4141

a1.channels.c1.type = memory
a1.channels.c1.capacity = 10000
a1.channels.c1.transactionCapacity = 10000
a1.channels.c1.byteCapacityBufferPercentage = 20
a1.channels.c1.byteCapacity = 800000

a1.sinks.k1.type = hive
a1.sinks.k1.channel = c1
a1.sinks.k1.hive.metastore =thrift://192.168.100.223:9083
a1.sinks.k1.hive.database=default
a1.sinks.k1.hive.table=traffic_a
a1.sinks.k1.hive.partition=%Y%m%d
a1.sinks.k1.useLocalTimeStamp = false
a1.sinks.k1.hive.txnsPerBatchAsk=2
#a1.sinks.k1.hive.batchSize = 10
a1.sinks.k1.serializer = DELIMITED
a1.sinks.k1.serializer.delimiter ="\t"
a1.sinks.k1.serializer.serdeSeparator = '\t'
a1.sinks.k1.serializer.fieldnames = companyid,zoneid,historyrequest,price,createdate

mysql table traffic_a is:

CREATE TABLE traffic_a (companyid int,zoneid int,historyrequest string,price double,createdate string) PARTITIONED BY(dt string) clustered by (zoneid) into 2 buckets stored as orc TBLPROPERTIES ("transactional"="true");

the data I need to send to Flume is like this :

String data = "8	16	{\"AGE\":\"40\",\"AGEGROUP\":\"3\",\"CITY\":\"tianjing\",\"EDUCATION\":\"2\",\"GENDER\":\"0\",\"GEO\":\"hanghai\",\"HASHCODE\":\"e9e09afb6734ea1e07a52464d1271397\",\"INCOME\":\"6002\",\"INCOMELEVEL\":\"2\"}	5	\"2017-07-05 01:02:03\"";

I have tried to sole this problem for a few days! But I do not kown the error msg mean? Can anyone give some suggesion? Thanks a lot!

Announcements