question Re: While loading the data from external hive table to orc table, in Support Questions

While loading the data from external hive table to orc table,

kotesh_banoth — Wed, 28 Dec 2016 17:10:47 GMT

Re: While loading the data from external hive table to orc table,

Eran — Wed, 28 Dec 2016 22:04:21 GMT

If you are looking for a way to load data from external table to ORC here is an example:

CREATE TABLE New_ORC_table STORED AS ORC tblproperties(“orc.compress”=“SNAPPY”) AS SELECT * FROM old_external_table;

Re: While loading the data from external hive table to orc table,

skurup — Wed, 28 Dec 2016 22:04:42 GMT

@kotesh banoth

Need more information here

Re: While loading the data from external hive table to orc table,

TimothySpann — Wed, 28 Dec 2016 23:18:48 GMT

NIFI 1.1 can do that

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi.processors.hive.ConvertAvroToORC/index.html

Re: While loading the data from external hive table to orc table,

srai1 — Thu, 29 Dec 2016 00:49:52 GMT

@kotesh banoth please complete your sentence 🙂

Re: While loading the data from external hive table to orc table,

kotesh_banoth — Thu, 29 Dec 2016 11:14:06 GMT

Status: Failed Vertex re-running, vertexName=Map 1, vertexId=vertex_1481963681664_2652_2_00 Vertex failed, vertexName=Reducer 2, vertexId=vertex_1481963681664_2652_2_01, diagnostics=[Task failed, taskId=task_1481963681664_2652_2_01_000002, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to create RecordUpdater for HiveOutputFormat that does not implement AcidOutputFormat at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:347) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Hive Runtime Error while closing operators: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to create RecordUpdater for HiveOutputFormat that does not implement AcidOutputFormat at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.close(ReduceRecordProcessor.java:310) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:164) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to create RecordUpdater for HiveOutputFormat that does not implement AcidOutputFormat at org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:577) at org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:1010)

Re: While loading the data from external hive table to orc table,

kotesh_banoth — Thu, 29 Dec 2016 11:14:26 GMT

Re: While loading the data from external hive table to orc table,

kotesh_banoth — Thu, 29 Dec 2016 11:15:41 GMT

Getting error as below: Status: Failed Vertex re-running, vertexName=Map 1, vertexId=vertex_1481963681664_2652_2_00 Vertex failed, vertexName=Reducer 2, vertexId=vertex_1481963681664_2652_2_01, diagnostics=[Task failed, taskId=task_1481963681664_2652_2_01_000002, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Hive Runtime Error while closing operators: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to create RecordUpdater for HiveOutputFormat that does not implement AcidOutputFormat at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:173)

Re: While loading the data from external hive table to orc table,

kotesh_banoth — Thu, 29 Dec 2016 11:16:35 GMT

Thanks, my question was incomplete . please look into error which i had posted now.

Re: While loading the data from external hive table to orc table,

srai1 — Fri, 30 Dec 2016 01:54:01 GMT

@kotesh banoth can you help me with the output of "describe formatted <tablename>" for both external and managed tables ?

Re: While loading the data from external hive table to orc table,

skurup — Fri, 30 Dec 2016 02:21:35 GMT

The query fails here. Below is the stack for the error generated

public static RecordUpdater getAcidRecordUpdater(JobConf jc, TableDesc tableInfo, int bucket,
  FileSinkDesc conf, Path outPath,
  ObjectInspector inspector,
  Reporter reporter, int rowIdColNum)
  throws HiveException, IOException {
HiveOutputFormat<?, ?> hiveOutputFormat = getHiveOutputFormat(jc, tableInfo);
AcidOutputFormat<?, ?> acidOutputFormat = null;
if (hiveOutputFormat instanceof AcidOutputFormat) {
  acidOutputFormat = (AcidOutputFormat)hiveOutputFormat;
} else {
  throw new HiveException("Unable to create RecordUpdater for HiveOutputFormat that does not " +
      "implement AcidOutputFormat");

Are these tables have ACID support enabled ?