Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

hdfs.protocol.AlreadyBeingCreatedException

hdfs.protocol.AlreadyBeingCreatedException

Explorer

Hey there,

 

I am creating a parquet table by loading data from a non parquet table into it. Here is the error log:

 

using this query:

insert overwrite table 1gb_parquet_table select * from 1gb_nonparquet_table

 

 

Log Type: stderr
  
  Log Length: 22143
  org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException): Failed to create file [/tmp/hive-hive/hive_2014-08-29_11-39-54_165_7508319328508212888-1/_task_tmp.-ext-10002/_tmp.000001_0] for [DFSClient_attempt_1409323138357_0001_m_000001_0_1194751406_1] on client [129.73.17.194], because this file is already being created by [DFSClient_attempt_1409323138357_0001_m_000001_0_1283605123_1] on [129.73.17.194]
  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2548)
  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2306)
  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2235)
  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2188)
  at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:505)
  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:354)
  at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
  at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)
  
  at org.apache.hadoop.hive.ql.io.HiveFileFormatUtils.getHiveRecordWriter(HiveFileFormatUtils.java:286)
  at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:516)
  at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:860)
  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:598)
  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616)
  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616)
  at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:616)
  at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.close(ExecMapper.java:207)
  at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:61)
  at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430)
  at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342)
  at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
  at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
  Caused by: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException): Failed to create file [/tmp/hive-hive/hive_2014-08-29_11-39-54_165_7508319328508212888-1/_task_tmp.-ext-10002/_tmp.000001_0] for [DFSClient_attempt_1409323138357_0001_m_000001_0_1194751406_1] on client [129.73.17.194], because this file is already being created by [DFSClient_attempt_1409323138357_0001_m_000001_0_1283605123_1] on [129.73.17.194]
  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2548)
  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2306)
  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2235)
  at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2188)
  at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:505)
  at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:354)
  at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
  at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
  at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
  at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
  at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:415)
  at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
  at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

 I continously get this error until I recieve a java heap error the table I am generating has 1800 columns not sure if that hurts it.

1 REPLY 1

Re: hdfs.protocol.AlreadyBeingCreatedException

New Contributor

Same problem.  Is this a bug related to the newest version of Cloudera, which puts .hive folders in the same location as the data for a Hive table?