Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Transactionbatch.beginNextTransaction() error in hive streaming code using StrictJsonWriter

Highlighted

Transactionbatch.beginNextTransaction() error in hive streaming code using StrictJsonWriter

New Contributor

Hi,

I am trying to ingest Json Files in to hive table using streaming code. I was facing following error to acquire lock on metastore (Using HDP 2.6 sandbox). I have no clue on this error. Can anyone help me on this? Thanks in advance.

org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://masternode1.rudram.us:9083', database='default', table='usage1', partitionVals=[] }: null at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:636) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:603) at com.rudram.bigdata.hivestruct.StructWriter.struct(StructWriter.java:130) at com.rudram.bigdata.hivestruct.StructWriter.main(StructWriter.java:200) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:297) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.thrift.transport.TTransportException at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:86) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:429) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:318) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:219) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3910) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3897) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1869) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:152) at com.sun.proxy.$Proxy33.lock(Unknown Source) at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:631) ... 9 more [WARNING] java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:297) at java.lang.Thread.run (Thread.java:748) Caused by: com.rudram.bigdata.hivestruct.REIHiveException: Failed to write. at com.rudram.bigdata.hivestruct.StructWriter.struct (StructWriter.java:148) at com.rudram.bigdata.hivestruct.StructWriter.main (StructWriter.java:200) at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke (Method.java:498) at org.codehaus.mojo.exec.ExecJavaMojo$1.run (ExecJavaMojo.java:297) at java.lang.Thread.run (Thread.java:748)

5 REPLIES 5

Re: Transactionbatch.beginNextTransaction() error in hive streaming code using StrictJsonWriter

Expert Contributor

Check metastore log from standalone metastore. It may have more information.

Re: Transactionbatch.beginNextTransaction() error in hive streaming code using StrictJsonWriter

New Contributor

I fixed this issue by changing Hadoop and Hive versions in my Pom. Now, I am facing "Unable to get path for end partionvals=[UA,201702]" and "Unable to flush recordupdater" issues. But, it was working fine before. Can you help me with this. @Eugene Koifman

Re: Transactionbatch.beginNextTransaction() error in hive streaming code using StrictJsonWriter

Expert Contributor

I would need more context. What exceptions are you getting? It seems unusual that you have modify pom files in an whole platform installation. Are you calling Streaming API from your own app?

Re: Transactionbatch.beginNextTransaction() error in hive streaming code using StrictJsonWriter

New Contributor

I was using streaming API on HDP 2.5 before that I ran same code on cloudera sandbox. It worked fine on Cloudera but issues are coming when I tried to run it on HDP. Then I figured out may be version issue(Unable to acquire lock) and I changed everything to my HDP version and it worked fine. Now after running 1000 files it started troubling again (unable to flush recordupdater and sometime unable to find path to end partitionvals=[UA,201702]). @Eugene Koifman

trace:

2017-11-08 16:41:42,380 ERROR [pool-29-thread-1] org.apache.hadoop.hdfs.DFSClient Failed to close inode 368731 org.apache.hadoop.ipc.RemoteException: No lease on /apps/hive/warehouse/usage/airline_id=UA/departure_time_month=201701/delta_0003470_0003471/bucket_00001_flush_length (inode 368731): File does not exist. [Lease. Holder: DFSClient_NONMAPREDUCE_1674329775_60, pendingcreates: 299] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3521) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSNamesystem.java:3611) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:3578) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:905) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:544) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2313) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2309) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2307) at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1552) at org.apache.hadoop.ipc.Client.call(Client.java:1496) at org.apache.hadoop.ipc.Client.call(Client.java:1396) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233) at com.sun.proxy.$Proxy129.complete(Unknown Source) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.complete(ClientNamenodeProtocolTranslatorPB.java:501) at sun.reflect.GeneratedMethodAccessor303.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:278) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:194) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:176) at com.sun.proxy.$Proxy130.complete(Unknown Source) at org.apache.hadoop.hdfs.DFSOutputStream.completeFile(DFSOutputStream.java:2361) at org.apache.hadoop.hdfs.DFSOutputStream.closeImpl(DFSOutputStream.java:2338) at org.apache.hadoop.hdfs.DFSOutputStream.close(DFSOutputStream.java:2303) at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:947) at org.apache.hadoop.hdfs.DFSClient.closeOutputStreams(DFSClient.java:979) at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:1192) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2852) at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2869) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

trace 2:

Exception in thread "main" org.apache.hive.hcatalog.streaming.StreamingException: partition values=[UAL, 201725]. Unable to get path for end point: [UAL, 201725] at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.getPathForEndPoint(AbstractRecordWriter.java:278) at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.<init>(AbstractRecordWriter.java:98) at org.apache.hive.hcatalog.streaming.StrictJsonWriter.<init>(StrictJsonWriter.java:82) at org.apache.hive.hcatalog.streaming.StrictJsonWriter.<init>(StrictJsonWriter.java:70) at com.rudram.bigdata.hivestruct.StructWriter.struct(StructWriter.java:69) at com.rudram.bigdata.hivestruct.StructWriter.main(StructWriter.java:86) Caused by: NoSuchObjectException(message:partition values=[UAL, 201725]) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_result$get_partition_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_result$get_partition_resultStandardScheme.read(ThriftHiveMetastore.java) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_partition_result.read(ThriftHiveMetastore.java) at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_partition(ThriftHiveMetastore.java:1872) at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_partition(ThriftHiveMetastore.java:1857) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getPartition(HiveMetaStoreClient.java:1200) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:155) at com.sun.proxy.$Proxy4.getPartition(Unknown Source) at org.apache.hive.hcatalog.streaming.AbstractRecordWriter.getPathForEndPoint(AbstractRecordWriter.java:273) ... 5 more

Re: Transactionbatch.beginNextTransaction() error in hive streaming code using StrictJsonWriter

New Contributor

"org.apache.hive.hcatalog.streaming.StreamingIOFailure: Unable to flush recordUpdater"

Don't have an account?
Coming from Hortonworks? Activate your account here