try adding this parameter to your sqoop job -D sqoop.export.records.per.statement=100 you can micro batch your transactions. ... View more

if you are using 2.6 version or later, you can turn on ACID and execute delete commands. you can audit your delete via your app if it is needed. Otherwise, delete, merge and update can run on Hive directly with ACID. ... View more

by Inserting into table using the construct INSERT INTO SELECT FROM... you are essentially bacthing/microbatching the data. You would have to create scripts/code to handle exceptions as you are loading. There are multiple ways of doing this. But there is no automated way unless you use ETL/ELT tools. ... View more

How are you loading the table? Are you loading them from CSV files, or using sqoop. Each of those ways you could enable logging at job level to track changes and restart the batch. If you have written your own scripting, on error you should try and write those files to error directories so you could go back and look for errors, fix them and reload them. ... View more

This works perfectly with Field Cloud. If you want to run some queries on phoenix by following this and Phoenix and Hbase tutorials this is an awesome demoable material ... View more

Also, how big is data in your table.. are you doing some sort of Limit or where clause when you run the query? ... View more

Are these tables External Tables? In the case of external tables you would have manually clean the folders by removing the files and folders that are referenced by the table ( using hadoop fs -rm command) ... View more

Is this an external table? if it is an external table then write a perl/shell script that runs daily based on a pattern it will remove files older than a certain date. If it is not, and if it is a managed table with ORC/Parquet format, then it depends on how you load the table. While loading the table you could setup logic to populate the date in the table partition by that date and as @Avijeet Dash mentioned drop the partition that is 7 days old. ... View more

sudo su - hdfs, then execute your commands. the xdl3 user does not have write access to /xdl/tmp directory. Also i hope you dont have any acls setup. ... View more