Hi Experts, I was reading about "withBatchSize" parameter in apache storm at http://storm.apache.org/releases/1.1.1/storm-hive.html It has explanation as "Max number of events written to Hive in a single Hive transaction". So below are my questions, 1. What is events? Is it one record in stream which would be ingested or any thing else. 2. How to verify that number for batch size is being written into hdfs Thanks ... View more

Hi Experts, In my Storm based application I need to query oracle table periodically So I thought to use Tick tuple of storm. But it's not giving correct result and tick tuple is not producing. My storm version is 1.0.1.2.5.3.0-37 I tried as below, 1. Added getComponentConfiguration method in bolt but tick tuple is not generating. 2. So I changed the code and used Config from topology for generating tick tuple. But got tick tuple only once. Below is my code of tick tuple with bolt, public class TickTupleBolt implements IRichBolt{ private OutputCollector collector = null; private static final long serialVersionUID = 1L; private static final Logger LOG = LoggerFactory.getLogger(TickTupleBolt.class); public void prepare(Map stormConf, TopologyContext context,OutputCollector collector) { this.collector = collector; } public void execute(Tuple tuple) { LOG.info("Start of TickTupleBolt.execute"); try { if (isTickTuple(tuple)) { //if(tuple.getSourceStreamId().equals("__tick")){ LOG.info("**got tick tuple"); }else{ LOG.info("not got tick tuple"); } } catch (Exception e) { LOG.error("Bolt execute error: {}", e); collector.reportError(e); } LOG.info("End of TickTupleBolt.execute"); } public void cleanup() { // TODO Auto-generated method stub } public void declareOutputFields(OutputFieldsDeclarer declarer) { // TODO Auto-generated method stub } So can any body please let me know, 1. How to implement tick tuple in Storm 2. Is there any other way (apart from tick tuple) to do periodic job in storm Thanks. ... View more

I am with nifi 1.4. I am using ConsumeKafkaRecord_0_10 and PutHiveStreaming processor. I am getting pipe delimited messages which is parsed with CSVReader and written by using AvroRecordSetWriter. But now I am getting below error. 2017-11-06 11:11:02,699 ERROR [Timer-Driven Process Thread-6]o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=819a7fe2-015f-1000-7097-cd3f8614c9c2] Error connecting to Hive endpoint: table oneviewtest3 at thrift://parent.rolta.com:9083 2017-11-06 11:11:02,699 ERROR [Timer-Driven Process Thread-6] o.a.n.processors.hive.PutHiveStreaming PutHiveStreaming[id=819a7fe2-015f-1000-7097-cd3f8614c9c2] Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://parent.rolta.com:9083', database='default', table='oneviewtest3', partitionVals=[] }: org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://parent.rolta.com:9083', database='default', table='oneviewtest3', partitionVals=[] } org.apache.nifi.processors.hive.PutHiveStreaming$ShouldRetryException: Hive Streaming connect/write error, flow file will be penalized and routed to retry. org.apache.nifi.util.hive.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://parent.rolta.com:9083', database='default', table='oneviewtest3', partitionVals=[] } Below is my table create statement, create table oneviewtest3 (TNT_KEY BIGINT, PLANT_KEY BIGINT, UNIT_KEY BIGINT, TAG_DATA_SRC STRING, TAG_DATA_RLVNC STRING, TAG_TYPE_ID STRING, DB_RQST_ID STRING, BATCH_KEY BIGINT, OPC_TAG_ID STRING, OPC_TAG_VAL_TS STRING, OPC_SRC_STM_CODE BIGINT, OPC_TAG_VAL STRING, OPC_QLTY_VAL BIGINT, REF_ID STRING, REF_TYPE STRING, ERROR_CODE STRING, OPC_TAG_VAL_AGG_TYPE STRING, OPC_TAG_RSMPL_INTRVL_VAL BIGINT, OPC_TAG_RSMPL_INTRVL_TYPE STRING, OPC_TAG_VAL_TS_PRC STRING, FLEX_FIELD1 STRING, FLEX_FIELD2 STRING, FLEX_FIELD3 STRING, FLEX_FIELD4 STRING, FLEX_FIELD5 STRING, ETL_BTCH_ID BIGINT) CLUSTERED BY (OPC_TAG_ID) into 4 buckets stored as orc tblproperties ("transactional"="true","orc.compress"="NONE"); I am already following below in table create statement, 1.ORC format for table. So table has "stored as orc" 2.transactional = "true" is set in the table create statement 3.Bucketed but not sorted. So table has "clustered by (colName) into (n) buckets" And also done following on hive, hive.support.concurrency = true hive.enforce.bucketing = true hive.exec.dynamic.partition.mode = nonstrict hive.txn.manager = org.apache.hadoop.hive.ql.lockmgr.DbTxnManager hive.compactor.initiator.on = true hive.compactor.worker.threads = 2 I am using HDP 2.5.3 and nifi-hive-nar-1.4.0.nar. In hive-metastore I see error as, 2017-11-06 12:02:53,436 ERROR [pool-8-thread-192]: server.TThreadPoolServer (TThreadPoolServer.java:run(297)) - Error occurred during processing of message. java.lang.IllegalStateException: Unexpected DataOperationType: UNSET agentInfo=Unknown txnid:424 So can anybody please help me on it. ... View more

Hi, I just started with NiFi 1.4. I am trying to send pipe delimited message via kafka into Hive. So I am using ConsumeKafkaRecord_0_10 and PutHivStreaming processor. Consume Kafka reader sends data on success to PutHiveStreaming. Consume Kafka reader writing data in avro format but Put Hive streaming gives error as "The incoming flow file can not be read as an Avro filee: java.io.IOException: Not a data file." So can anybody please help me on this error. ... View more

Hi, I am using Kafka 0.10 and Flume 1.8. I am trying to get information on below (but could not get it). So can any body please help me. 1. Is there any way to send event to particular kafka topic partition 2. And if so, then can we read such events (coming to specific partition) with flume using hive sink ... View more

I am facing issue with kafka 0.10 and flume 1.8. We are trying to ingest NRT data into hive by streaming. But often get errors like socket exception and followed by java.lang.OutOfMemoryError. So i tried below things, 1. Exported java heap from flume-env.sh to 8GB but not worked. 2. Used multiple hive sinkers and channels (memory and file) but not worked. We are having 4 node cluster with 16GB memory on each node. So is there any thing else I can do in flume configuration or any alternative for kafka-flume would help me. Below is my flume properties flume1.sources = kafka-source-1 flume1.channels = hive-channel-1 hive-channel-2 hive-channel-3 hive-channel-4 hive-channel-5 hive-channel-6 hive-channel-7 hive-channel-8 hive-channel-9 flume1.sinks = hive-sink-1 hive-sink-2 hive-sink-3 hive-sink-4 hive-sink-5 hive-sink-6 hive-sink-7 hive-sink-8 hive-sink-9 flume1.sources.kafka-source-1.type = org.apache.flume.source.kafka.KafkaSource flume1.sources.kafka-source-1.zookeeperConnect = base1.rolta.com:2181 flume1.sources.kafka-source-1.topic = iot-streaming flume1.sources.kafka-source-1.batchSize = 100 flume1.sources.kafka-source-1.channels = hive-channel-1 flume1.sources.kafka-source-1.batchSize = 100 flume1.sources.kafka-source-1.batchDurationMillis = 10000 flume1.channels.hive-channel-1.type = file flume1.channels.hive-channel-1.checkpointDir = /home/labuser/flumedata/chkpoint/001 flume1.channels.hive-channel-1.dataDirs = /home/labuser/flumedata/data/001 flume1.channels.hive-channel-1.transactionCapacity = 100 flume1.channels.hive-channel-2.type = file flume1.channels.hive-channel-2.checkpointDir = /home/labuser/flumedata/chkpoint/002 flume1.channels.hive-channel-2.dataDirs = /home/labuser/flumedata/data/002 flume1.channels.hive-channel-2.transactionCapacity = 100 flume1.channels.hive-channel-3.type = file flume1.channels.hive-channel-3.checkpointDir = /home/labuser/flumedata/chkpoint/003 flume1.channels.hive-channel-3.dataDirs = /home/labuser/flumedata/data/003 flume1.channels.hive-channel-3.transactionCapacity = 100 flume1.channels.hive-channel-4.type = file flume1.channels.hive-channel-4.checkpointDir = /home/labuser/flumedata/chkpoint/004 flume1.channels.hive-channel-4.dataDirs = /home/labuser/flumedata/data/004 flume1.channels.hive-channel-4.transactionCapacity = 100 flume1.channels.hive-channel-5.type = file flume1.channels.hive-channel-5.checkpointDir = /home/labuser/flumedata/chkpoint/005 flume1.channels.hive-channel-5.dataDirs = /home/labuser/flumedata/data/005 flume1.channels.hive-channel-5.transactionCapacity = 100 flume1.channels.hive-channel-6.type = file flume1.channels.hive-channel-6.checkpointDir = /home/labuser/flumedata/chkpoint/006 flume1.channels.hive-channel-6.dataDirs = /home/labuser/flumedata/data/006 flume1.channels.hive-channel-6.transactionCapacity = 100 flume1.channels.hive-channel-7.type = file flume1.channels.hive-channel-7.checkpointDir = /home/labuser/flumedata/chkpoint/007 flume1.channels.hive-channel-7.dataDirs = /home/labuser/flumedata/data/007 flume1.channels.hive-channel-7.transactionCapacity = 100 flume1.channels.hive-channel-8.type = file flume1.channels.hive-channel-8.checkpointDir = /home/labuser/flumedata/chkpoint/008 flume1.channels.hive-channel-8.dataDirs = /home/labuser/flumedata/data/008 flume1.channels.hive-channel-8.transactionCapacity = 100 flume1.channels.hive-channel-9.type = file flume1.channels.hive-channel-9.checkpointDir = /home/labuser/flumedata/chkpoint/009 flume1.channels.hive-channel-9.dataDirs = /home/labuser/flumedata/data/009 flume1.channels.hive-channel-9.transactionCapacity = 100 flume1.sinks.hive-sink-1.channel = hive-channel-1 flume1.sinks.hive-sink-1.type = hive flume1.sinks.hive-sink-1.hive.metastore = thrift://base1.rolta.com:9083 flume1.sinks.hive-sink-1.hive.database = default flume1.sinks.hive-sink-1.hive.table = oneviewtest1 flume1.sinks.hive-sink-1.serializer = DELIMITED flume1.sinks.hive-sink-1.serializer.delimiter = "\\|" flume1.sinks.hive-sink-1.serializer.fieldnames = db_rqst_id,opc_tag_id,opc_qlty_val,opc_tag_val,opc_tag_val_ts,opc_src_stm_code,tag_data_src flume1.sinks.hive-sink-1.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-1.batchSize = 100 flume1.sinks.hive-sink-1.hive.partition = %y-%m-%d flume1.sinks.hive-sink-1.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-2.channel = hive-channel-2 flume1.sinks.hive-sink-2.type = hive flume1.sinks.hive-sink-2.hive.metastore = thrift://base1.rolta.com:9083 flume1.sinks.hive-sink-2.hive.database = default flume1.sinks.hive-sink-2.hive.table = oneviewtest1 flume1.sinks.hive-sink-2.serializer = DELIMITED flume1.sinks.hive-sink-2.serializer.delimiter = "\\|" flume1.sinks.hive-sink-2.serializer.fieldnames = db_rqst_id,opc_tag_id,opc_qlty_val,opc_tag_val,opc_tag_val_ts,opc_src_stm_code,tag_data_src flume1.sinks.hive-sink-2.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-2.batchSize = 100 flume1.sinks.hive-sink-2.hive.partition = %y-%m-%d flume1.sinks.hive-sink-2.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-3.channel = hive-channel-3 flume1.sinks.hive-sink-3.type = hive flume1.sinks.hive-sink-3.hive.metastore = thrift://base1.rolta.com:9083 flume1.sinks.hive-sink-3.hive.database = default flume1.sinks.hive-sink-3.hive.table = oneviewtest1 flume1.sinks.hive-sink-3.serializer = DELIMITED flume1.sinks.hive-sink-3.serializer.delimiter = "\\|" flume1.sinks.hive-sink-3.serializer.fieldnames = db_rqst_id,opc_tag_id,opc_qlty_val,opc_tag_val,opc_tag_val_ts,opc_src_stm_code,tag_data_src flume1.sinks.hive-sink-3.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-3.batchSize = 100 flume1.sinks.hive-sink-3.hive.partition = %y-%m-%d flume1.sinks.hive-sink-3.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-4.channel = hive-channel-4 flume1.sinks.hive-sink-4.type = hive flume1.sinks.hive-sink-4.hive.metastore = thrift://base1.rolta.com:9083 flume1.sinks.hive-sink-4.hive.database = default flume1.sinks.hive-sink-4.hive.table = oneviewtest1 flume1.sinks.hive-sink-4.serializer = DELIMITED flume1.sinks.hive-sink-4.serializer.delimiter = "\\|" flume1.sinks.hive-sink-4.serializer.fieldnames = db_rqst_id,opc_tag_id,opc_qlty_val,opc_tag_val,opc_tag_val_ts,opc_src_stm_code,tag_data_src flume1.sinks.hive-sink-4.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-4.batchSize = 100 flume1.sinks.hive-sink-4.hive.partition = %y-%m-%d flume1.sinks.hive-sink-4.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-5.channel = hive-channel-5 flume1.sinks.hive-sink-5.type = hive flume1.sinks.hive-sink-5.hive.metastore = thrift://base1.rolta.com:9083 flume1.sinks.hive-sink-5.hive.database = default flume1.sinks.hive-sink-5.hive.table = oneviewtest1 flume1.sinks.hive-sink-5.serializer = DELIMITED flume1.sinks.hive-sink-5.serializer.delimiter = "\\|" flume1.sinks.hive-sink-5.serializer.fieldnames = db_rqst_id,opc_tag_id,opc_qlty_val,opc_tag_val,opc_tag_val_ts,opc_src_stm_code,tag_data_src flume1.sinks.hive-sink-5.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-5.batchSize = 100 flume1.sinks.hive-sink-5.hive.partition = %y-%m-%d flume1.sinks.hive-sink-5.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-6.channel = hive-channel-6 flume1.sinks.hive-sink-6.type = hive flume1.sinks.hive-sink-6.hive.metastore = thrift://base1.rolta.com:9083 flume1.sinks.hive-sink-6.hive.database = default flume1.sinks.hive-sink-6.hive.table = oneviewtest1 flume1.sinks.hive-sink-6.serializer = DELIMITED flume1.sinks.hive-sink-6.serializer.delimiter = "\\|" flume1.sinks.hive-sink-6.serializer.fieldnames = db_rqst_id,opc_tag_id,opc_qlty_val,opc_tag_val,opc_tag_val_ts,opc_src_stm_code,tag_data_src flume1.sinks.hive-sink-6.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-6.batchSize = 100 flume1.sinks.hive-sink-6.hive.partition = %y-%m-%d flume1.sinks.hive-sink-6.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-7.channel = hive-channel-7 flume1.sinks.hive-sink-7.type = hive flume1.sinks.hive-sink-7.hive.metastore = thrift://base1.rolta.com:9083 flume1.sinks.hive-sink-7.hive.database = default flume1.sinks.hive-sink-7.hive.table = oneviewtest1 flume1.sinks.hive-sink-7.serializer = DELIMITED flume1.sinks.hive-sink-7.serializer.delimiter = "\\|" flume1.sinks.hive-sink-7.serializer.fieldnames = db_rqst_id,opc_tag_id,opc_qlty_val,opc_tag_val,opc_tag_val_ts,opc_src_stm_code,tag_data_src flume1.sinks.hive-sink-7.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-7.batchSize = 100 flume1.sinks.hive-sink-7.hive.partition = %y-%m-%d flume1.sinks.hive-sink-7.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-8.channel = hive-channel-8 flume1.sinks.hive-sink-8.type = hive flume1.sinks.hive-sink-8.hive.metastore = thrift://base1.rolta.com:9083 flume1.sinks.hive-sink-8.hive.database = default flume1.sinks.hive-sink-8.hive.table = oneviewtest1 flume1.sinks.hive-sink-8.serializer = DELIMITED flume1.sinks.hive-sink-8.serializer.delimiter = "\\|" flume1.sinks.hive-sink-8.serializer.fieldnames = db_rqst_id,opc_tag_id,opc_qlty_val,opc_tag_val,opc_tag_val_ts,opc_src_stm_code,tag_data_src flume1.sinks.hive-sink-8.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-8.batchSize = 100 flume1.sinks.hive-sink-8.hive.partition = %y-%m-%d flume1.sinks.hive-sink-8.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-9.channel = hive-channel-9 flume1.sinks.hive-sink-9.type = hive flume1.sinks.hive-sink-9.hive.metastore = thrift://base1.rolta.com:9083 flume1.sinks.hive-sink-9.hive.database = default flume1.sinks.hive-sink-9.hive.table = oneviewtest1 flume1.sinks.hive-sink-9.serializer = DELIMITED flume1.sinks.hive-sink-9.serializer.delimiter = "\\|" flume1.sinks.hive-sink-9.serializer.fieldnames = db_rqst_id,opc_tag_id,opc_qlty_val,opc_tag_val,opc_tag_val_ts,opc_src_stm_code,tag_data_src flume1.sinks.hive-sink-9.hive.txnsPerBatchAsk = 2 flume1.sinks.hive-sink-9.batchSize = 100 flume1.sinks.hive-sink-9.hive.partition = %y-%m-%d ... View more

Hi, I have done real time streaming into hive tables using storm. But now it takes long time to insert data into hive table. And the time lag for inserting data into table keeps on increasing. Below is my code, public class HiveSinker { private static final Logger LOG = LoggerFactory.getLogger(HiveSinker.class); public HiveBolt buildHiveBolt(String metaStoreURI, String dbName, String tblName, String[] colNames) { LOG.info("start of HiveSinker.buildHiveBolt"); DelimitedRecordHiveMapper mapper = new DelimitedRecordHiveMapper().withColumnFields(new Fields(colNames)).withTimeAsPartitionField("YYYY/MM/DD"); //HiveOptions hiveOptions = new HiveOptions(metaStoreURI,dbName,tblName,mapper).withTxnsPerBatch(10).withBatchSize(1000).withIdleTimeout(10).withCallTimeout(10000000); HiveOptions hiveOptions = new HiveOptions(metaStoreURI,dbName,tblName,mapper).withTxnsPerBatch(2).withBatchSize(100).withIdleTimeout(10).withCallTimeout(10000); HiveBolt hiveBolt = new HiveBolt(hiveOptions); LOG.info("end of HiveSinker.buildHiveBolt"); return hiveBolt; } } So I tried to change withTxnsPerBatch and withBatchSize but with no effect.So can anybody points out the issue. Thanks. ... View more

I am facing issue in storm and hive streaming with HDP 2.5. I am getting error as, Org.apache.storm.hive.common.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://base1.rolta.com:9083', database='default', table='table_mqtt', partitionVals=[2017/08/242] } at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:80) ~[stormjar.jar:?] at org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUtils.java:50) ~[stormjar.jar:?] at org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBolt.java:271) ~[stormjar.jar:?] at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:114) [stormjar.jar:?] at org.apache.storm.daemon.executor$fn__9364$tuple_action_fn__9366.invoke(executor.clj:734) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.daemon.executor$mk_task_receiver$fn__9285.invoke(executor.clj:466) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.disruptor$clojure_handler$reify__8798.onEvent(disruptor.clj:40) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.daemon.executor$fn__9364$fn__9377$fn__9430.invoke(executor.clj:853) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at org.apache.storm.util$async_loop$fn__656.invoke(util.clj:484) [storm-core-1.0.1.2.5.3.0-37.jar:1.0.1.2.5.3.0-37] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77] Caused by: org.apache.storm.hive.common.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://base1.rolta.com:9083', database='default', table='table_mqtt', partitionVals=[2017/08/242] } at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:264) ~[stormjar.jar:?] at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72) ~[stormjar.jar:?] ... 13 more Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://base1.rolta.com:9083', database='default', table='table_mqtt', partitionVals=[2017/08/242] } at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:575) ~[stormjar.jar:?] at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:544) ~[stormjar.jar:?] at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:259) ~[stormjar.jar:?] at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72) ~[stormjar.jar:?] ... 13 more Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[stormjar.jar:?] at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84) ~[stormjar.jar:?] at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378) ~[stormjar.jar:?] at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297) ~[stormjar.jar:?] at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204) ~[stormjar.jar:?] at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) ~[stormjar.jar:?] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3781) ~[stormjar.jar:?] at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3768) ~[stormjar.jar:?] at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1736) ~[stormjar.jar:?] at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:570) ~[stormjar.jar:?] at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:544) ~[stormjar.jar:?] at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:259) ~[stormjar.jar:?] at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72) ~[stormjar.jar:?] ... 13 more I tried below, 1. Done coding as link http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-hive.html 2. Then changed hive-conf.xml on all nodes as link http://www.openkb.info/2015/06/hive-transaction-feature-in-hive-10.html But then also getting error. Below is my pom. <dependencies> <dependency> <groupId>joda-time</groupId> <artifactId>joda-time</artifactId> <version>2.9.9</version> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-core</artifactId> <version>1.0.1</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-hive</artifactId> <version>1.0.1</version> </dependency> <dependency> <groupId>org.eclipse.paho</groupId> <artifactId>org.eclipse.paho.client.mqttv3</artifactId> <version>1.1.0</version> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.3.3</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.2.0</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.2.0</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> Can anybody please help me with suggestions. ... View more

Hi, I am having issue with storm and hive streaming. I did as post https://community.hortonworks.com/questions/96995/storm-hdfs-javalangruntimeexception-error-preparin.html but did not helped. I also looked at https://community.hortonworks.com/questions/111874/non-local-session-path-expected-to-be-non-null-try.html post. But not understood as which jar to be included. Can anybody please help me. Below is my POM. <dependencies> <dependency> <groupId>joda-time</groupId> <artifactId>joda-time</artifactId> <version>2.9.9</version> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-core</artifactId> <version>1.0.1</version> <scope>provided</scope> </dependency> <dependency> <groupId>org.apache.storm</groupId> <artifactId>storm-hive</artifactId> <version>1.0.1</version> </dependency> <dependency> <groupId>org.eclipse.paho</groupId> <artifactId>org.eclipse.paho.client.mqttv3</artifactId> <version>1.1.0</version> </dependency> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.3.3</version> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-client</artifactId> <version>2.2.0</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.hadoop</groupId> <artifactId>hadoop-hdfs</artifactId> <version>2.2.0</version> <exclusions> <exclusion> <groupId>org.slf4j</groupId> <artifactId>slf4j-log4j12</artifactId> </exclusion> </exclusions> </dependency> </dependencies> Below is my maven shade plugin <build> <plugins> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-source-plugin</artifactId> </plugin> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-compiler-plugin</artifactId> <configuration> <source>1.8</source> <target>1.8</target> </configuration> </plugin> <!-- Maven shade plugin --> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.4.3</version> <configuration> <createDependencyReducedPom>true</createDependencyReducedPom> </configuration> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> </goals> <configuration> <!-- <artifactSet> <excludes> <exclude>org.slf4j:slf4j-log4j12:*</exclude> <exclude>log4j:log4j:jar:</exclude> <exclude>org.slf4j:slf4j-simple:jar</exclude> <exclude>org.apache.storm:storm-core</exclude> </excludes> </artifactSet> --> <transformers> <transformer implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/> <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> <mainClass></mainClass> </transformer> </transformers> </configuration> </execution> </executions> </plugin> </plugins> </build> Please see below error. java.lang.RuntimeException: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx------ at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444) ~[stormjar.jar:?] at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.createPartitionIfNotExists(HiveEndPoint.java:358) ~[stormjar.jar:?] at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:276) ~[stormjar.jar:?] at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) ~[stormjar.jar:?] at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) ~[stormjar.jar:?] at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) ~[stormjar.jar:?] at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:238) ~[stormjar.jar:?] at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:235) ~[stormjar.jar:?] at org.apache.storm.hive.common.HiveWriter$9.call(HiveWriter.java:366) ~[stormjar.jar:?] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_77] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_77] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77] Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable. Current permissions are: rwx------ at org.apache.hadoop.hive.ql.session.SessionState.createRootHDFSDir(SessionState.java:529) ~[stormjar.jar:?] at org.apache.hadoop.hive.ql.session.SessionState.createSessionDirs(SessionState.java:478) ~[stormjar.jar:?] at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:430) ~[stormjar.jar:?] ... 12 more 2017-08-30 14:51:58.022 o.a.s.d.executor [INFO] BOLT fail TASK: 3 TIME: TUPLE: source: mqttprocesser:4, stream: default, id: {}, [My count is, 11] 2017-08-30 14:51:58.023 o.a.s.d.executor [INFO] Execute done TUPLE source: mqttprocesser:4, stream: default, id: {}, [My count is, 11] TASK: 3 DELTA: 2017-08-30 14:51:58.023 o.a.s.d.executor [INFO] Processing received message FOR 3 TUPLE: source: mqttprocesser:4, stream: default, id: {}, [My count is, 12] 2017-08-30 14:51:58.050 h.metastore [INFO] Trying to connect to metastore with URI thrift://base1.rolta.com:9083 2017-08-30 14:51:58.052 h.metastore [INFO] Connected to metastore. 2017-08-30 14:51:58.124 o.a.h.h.q.l.PerfLogger [INFO] <PERFLOG method=Driver.run from=org.apache.hadoop.hive.ql.Driver> 2017-08-30 14:51:58.124 o.a.h.h.q.l.PerfLogger [INFO] <PERFLOG method=TimeToSubmit from=org.apache.hadoop.hive.ql.Driver> 2017-08-30 14:51:58.124 o.a.h.h.q.l.PerfLogger [INFO] <PERFLOG method=compile from=org.apache.hadoop.hive.ql.Driver> 2017-08-30 14:51:58.202 STDIO [ERROR] FAILED: NullPointerException Non-local session path expected to be non-null 2017-08-30 14:51:58.202 o.a.h.h.q.Driver [ERROR] FAILED: NullPointerException Non-local session path expected to be non-null java.lang.NullPointerException: Non-local session path expected to be non-null at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204) at org.apache.hadoop.hive.ql.session.SessionState.getHDFSSessionPath(SessionState.java:590) at org.apache.hadoop.hive.ql.Context.<init>(Context.java:129) at org.apache.hadoop.hive.ql.Context.<init>(Context.java:116) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:382) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:303) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1067) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1129) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1004) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:994) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.runDDL(HiveEndPoint.java:404) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.createPartitionIfNotExists(HiveEndPoint.java:369) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:276) at org.apache.hive.hcatalog.streaming.HiveEndPoint$ConnectionImpl.<init>(HiveEndPoint.java:243) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnectionImpl(HiveEndPoint.java:180) at org.apache.hive.hcatalog.streaming.HiveEndPoint.newConnection(HiveEndPoint.java:157) at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:238) at org.apache.storm.hive.common.HiveWriter$5.call(HiveWriter.java:235) at org.apache.storm.hive.common.HiveWriter$9.call(HiveWriter.java:366) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) ... View more

Hi All, So I am able to save data in Solr using Apache Storm. But it seems that same value is getting saved multiple times. Now i am going through links on it but can anybody please help me to know what ia the best way to save data using Apache Storm and overcome this issue. I am using shufflegrouping. Can anybody please help. ... View more