Support Questions
Find answers, ask questions, and share your expertise

How to move the data from the source ACID table to a non-ACID table

Explorer

Hello Everyone,

I need some help with below workaround.

This is regarding hive table(ACID transactional) where i want to copy data from PROD to PRE_PROD.


1. How to move the data from the source ACID table to a non-ACID table.

2. How to move the data from non-ACID table to ACID table.


Please suggest me with the commands as i am struggling to get it from many days.


Many thanks in advance!!!

3 REPLIES 3

Re: How to move the data from the source ACID table to a non-ACID table

Super Guru

@Shrikant BM

1. How to move the data from the source ACID table to a non-ACID table.

hive> create table <db_name>.<non_acid_table> stored as orc as select * from <acid_table>;

Now you can copy the non_acid_table data in HDFS from Prod to pre_prod using Distcp command.

2. How to move the data from non-ACID table to ACID table.

2.1.
Create your acid_table if not exists.
2.2.
hive> Insert into acid_Table select * from non_acid_table;

-

If the answer is helpful to resolve the issue, Login and Click on Accept button below to close this thread.This will help other community users to find answers quickly 🙂

Re: How to move the data from the source ACID table to a non-ACID table

Explorer

@Shu Many thanks for the support!!!

I tried running the create statement "create table <db_name>.<non_acid_table> stored as orc as select * from <acid_table>; " but I came across GC ERROR as below

2019-06-07 08:50:00,087 [INFO] [TezChild] |orc.ReaderImpl|: Reading ORC rows from hdfs://xxxxx/apps/hive/warehouse/master.db/assets/delta_0028169_0028169_0000/bucket_00000 with {include: [true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true, true], offset: 0, length: 9223372036854775807}

2019-06-07 08:51:17,413 [FATAL] [IPC Client (592617454) connection to /x.x.x.x from application_1558704226227_196441] |yarn.YarnUncaughtExceptionHandler|: Thread Thread[IPC Client (592617454) connection to /x.x.x.x from application_1558704226227_196441,5,main] threw an Error. Shutting down now...

java.lang.OutOfMemoryError: GC overhead limit exceeded

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.OutOfMemoryError: GC overhead limit exceeded


I have tried giving below command and rerun the statement but came across same issue

SET hive.tez.container.size=32768;


Can you please suggest on this?


Thanks in Advance!!!

Re: How to move the data from the source ACID table to a non-ACID table

Super Guru

@Shrikant BM

Try increasing the container size

set hive.tez.container.size=10240;
set hive.tez.java.opts=-Xmx10g;

if you are still facing the issue, try increasing the size and run the statements again.