Created on 07-18-2024 01:47 AM - edited 07-18-2024 01:49 AM
Good day,
In SIT, I have manually create a hive table. Follow by manually create a partition on it. Then I copy the bucket file from other table and put inside the partition. At this moment, if I run select statement, I getting 0 result, I believe this is normal because data haven't write to meta store.
Then I run the following commands:
ANALYZE TABLE raw_zone_sit.jxtemp PARTITION (dl_created_yr='2023', dl_created_mth='12') COMPUTE STATISTICS;
After I run this command, I able to select data from my table.
After that I apply the same way in production, but I am still getting empty result.
Currently I am trying to recover all the meta store in production, this is because 1 of the person accidently delete all the meta store in production. Thus now I am trying to recover on it.
For your information, parquet tables working fine. But I have some ORC tables also, now pending ORC table to resolve.
Please help.
Created 07-23-2024 11:49 PM
@AhXian Ideally it is not supported to copy files into and out of an ACID table location. In this case we can use a workaround:
1. Create an external table similar(same columns) as the original ACID table.
2. Upload the data file into the external table location.
3. MSCK REPAIR table <external table name>
4. See if you can read the data from the external table now.
5. If [4] works, insert the data into the managed table by using:
insert into table managed_table_name partition(xx,yy) select * from external_table_name;
Created 07-24-2024 05:43 AM
Yes, 5 steps mentioned above
Created 07-25-2024 05:44 AM
Hi @smruti , Yes, I have done it in production environment, its not working, I still unable to select from it. But its working in SIT.
Created 07-25-2024 06:12 AM
@AhXian
1. Are you able to read data from the external table in Prod(step 4)?
2. In step 5, I meant for you to insert data into a new ACID table, which can be created using similar DDL as the original table.
3. After INSERT(step 5), see if you could read the data from the new table.
Ideally it should work
Created 07-25-2024 07:00 PM
hi @smruti , I unable to read data from external table in step 4.
Thus, I cant use the insert data query to insert into a new ACID table in step 5.
Created 07-25-2024 08:34 PM
@AhXian I hope we have copied the bucket files correctly, and also ran MSCK REPAIR. Could you also run ANALYZE.. COMPUTE STATISTICS on the external table once. Make sure the DESCRIBE command on the external table reflects the number of data files correctly.
Please share describe and "hdfs dfs -ls -R <externa table location>" o/p