Support Questions
Find answers, ask questions, and share your expertise

Can we log partial records loaded into hive table

New Contributor

Hi ,I am trying to load data/table from source database into my database table.While loading the data into my table some error occured,which results in failure of table load.

Is there anything in hive which will provide me at which row the data loading got failed.

Let us take an example:suppose my source table has 1000 records ,while loading data into my table error occured at 500 record,so can i record from hive that loading failed because of line 500 or something like that.


What do you see in the HiveServer2 logs?

Expert Contributor

How are you loading the table? Are you loading them from CSV files, or using sqoop. Each of those ways you could enable logging at job level to track changes and restart the batch. If you have written your own scripting, on error you should try and write those files to error directories so you could go back and look for errors, fix them and reload them.

New Contributor

I am loading the data from one hive table to another hive table using insert and select statement.

i.e insert into table main_table select id,name from source_table.

I have written my own script,Can you please elloborate with some example.It will be more helpful.

Expert Contributor

by Inserting into table using the construct INSERT INTO SELECT FROM... you are essentially bacthing/microbatching the data. You would have to create scripts/code to handle exceptions as you are loading. There are multiple ways of doing this. But there is no automated way unless you use ETL/ELT tools.

Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.