Support Questions

Find answers, ask questions, and share your expertise

HiveBolt getting the error "Unable to acquire lock on"

Explorer

Hi Experts,

i am running a very simple example where a Storm topology is getting data through kafka and is then writhing them into Hive.

After few problems with the deployment of the Topology on the cluster (i am running the HDP 2.6.1 on Esxi) a managed to start top receives data but then i start to receive the error "Unable to acquire lock" while the data where supposed to be written into Hive.

The java application that i am using is instantiating the HiveBolt like this:

// Hive connection configuration
String metaStoreURI = "thrift://sandbox.hortonworks.com:9083";
String dbName = "core";
String tblName = "stock_prices";
// Fields for possible partition
String[] partNames = {"name"};
// Fields for possible column data
String[] colNames = {"day", "open", "high", "low", "close", "volume","adj_close"};
// Record Writer configuration
DelimitedRecordHiveMapper mapper = new DelimitedRecordHiveMapper().withColumnFields(new Fields(colNames)).withPartitionFields(new Fields(partNames));

HiveOptions hiveOptions;
hiveOptions = new HiveOptions(metaStoreURI, dbName, tblName, mapper)
                .withTxnsPerBatch(2)
                .withBatchSize(100)
                .withIdleTimeout(10)
                .withCallTimeout(10000000);

TopologyBuilder builder = new TopologyBuilder();
builder.setSpout(KAFKA_SPOUT_ID, kafkaSpout,4);
builder.setBolt(STOCK_PROCESS_BOLT_ID, new StockDataBolt()).shuffleGrouping(KAFKA_SPOUT_ID);
builder.setBolt(HIVE_BOLT_ID, new HiveBolt(hiveOptions)).shuffleGrouping(STOCK_PROCESS_BOLT_ID);

The worker.log shows the following error:

2017-10-26 19:34:44.370 h.metastore [INFO] Trying to connect to metastore with URI thrift://sandbox.hortonworks.com:9083

2017-10-26 19:34:44.371 h.metastore [INFO] Connected to metastore.

2017-10-26 19:34:44.754 o.a.s.h.b.HiveBolt [ERROR] Failed to create HiveWriter for endpoint: {metaStoreUri='thrift://sandbox.hortonworks.com:9083', database='core', table='stock_prices', partitionVals=[AAPL] }

org.apache.storm.hive.common.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://sandbox.hortonworks.com:9083', database='core', table='stock_prices', partitionVals=[AAPL] }

	at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:80) ~[storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUtils.java:50) ~[storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBolt.java:262) [storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:112) [storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.daemon.executor$fn__10432$tuple_action_fn__10434.invoke(executor.clj:730) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.daemon.executor$mk_task_receiver$fn__10353.invoke(executor.clj:462) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.disruptor$clojure_handler$reify__9867.onEvent(disruptor.clj:40) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.daemon.executor$fn__10432$fn__10445$fn__10498.invoke(executor.clj:849) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]

	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

Caused by: org.apache.storm.hive.common.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://sandbox.hortonworks.com:9083', database='core', table='stock_prices', partitionVals=[AAPL] }

	at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:264) ~[storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72) ~[storm-hive-1.1.1.jar:1.1.1]

	... 13 more

Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://sandbox.hortonworks.com:9083', database='core', table='stock_prices', partitionVals=[AAPL] }

	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:575) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]

	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:544) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]

	at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:259) ~[storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72) ~[storm-hive-1.1.1.jar:1.1.1]

	... 13 more

Caused by: org.apache.thrift.transport.TTransportException

	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3781) ~[hive-metastore-0.14.0.jar:0.14.0]

	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3768) ~[hive-metastore-0.14.0.jar:0.14.0]

	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1736) ~[hive-metastore-0.14.0.jar:0.14.0]

	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:570) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]

	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:544) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]

	at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:259) ~[storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72) ~[storm-hive-1.1.1.jar:1.1.1]

	... 13 more

2017-10-26 19:34:44.755 o.a.s.d.executor [ERROR] 

org.apache.storm.hive.common.HiveWriter$ConnectFailure: Failed connecting to EndPoint {metaStoreUri='thrift://sandbox.hortonworks.com:9083', database='core', table='stock_prices', partitionVals=[AAPL] }

	at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:80) ~[storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUtils.java:50) ~[storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBolt.java:262) ~[storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:112) [storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.daemon.executor$fn__10432$tuple_action_fn__10434.invoke(executor.clj:730) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.daemon.executor$mk_task_receiver$fn__10353.invoke(executor.clj:462) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.disruptor$clojure_handler$reify__9867.onEvent(disruptor.clj:40) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.daemon.executor$fn__10432$fn__10445$fn__10498.invoke(executor.clj:849) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.1.0.2.6.1.0-129.jar:1.1.0.2.6.1.0-129]

	at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]

	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_141]

Caused by: org.apache.storm.hive.common.HiveWriter$TxnBatchFailure: Failed acquiring Transaction Batch from EndPoint: {metaStoreUri='thrift://sandbox.hortonworks.com:9083', database='core', table='stock_prices', partitionVals=[AAPL] }

	at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:264) ~[storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72) ~[storm-hive-1.1.1.jar:1.1.1]

	... 13 more

Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://sandbox.hortonworks.com:9083', database='core', table='stock_prices', partitionVals=[AAPL] }

	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:575) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]

	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:544) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]

	at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:259) ~[storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72) ~[storm-hive-1.1.1.jar:1.1.1]

	... 13 more

Caused by: org.apache.thrift.transport.TTransportException

	at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:77) ~[libthrift-0.9.3.jar:0.9.3]

	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3781) ~[hive-metastore-0.14.0.jar:0.14.0]

	at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3768) ~[hive-metastore-0.14.0.jar:0.14.0]

	at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1736) ~[hive-metastore-0.14.0.jar:0.14.0]

	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:570) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]

	at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:544) ~[hive-hcatalog-streaming-0.14.0.jar:0.14.0]

	at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:259) ~[storm-hive-1.1.1.jar:1.1.1]

	at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72) ~[storm-hive-1.1.1.jar:1.1.1]

	... 13 more

Where you can find the Unable to acquire lock problem.

I am pretty new to the topic and I did tried to look in other question if there was the solution already but i could not find it.

I did all the setup described in the post https://community.hortonworks.com/questions/59411/how-to-use-puthivestreaming.html

but i am still getting the error.

Can you guy suggest anything?

I honestly do not know where to look anymore and i am stuck on a very basic thing (i guess ).

Thanks in advance

2 REPLIES 2

Explorer

same issue here

New Contributor

Any update on this? I am facing same issue

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.