If you want to delete files and have them not go in the trash, use: hdfs dfs -rm -r -skipTrash <directory name> ... View more

I'm not aware of an existing script already in HDP to do this for you. However, I did run across this: https://github.com/nmilford/clean-hadoop-tmp Note, that script is written in ruby. You could follow the logic an write it in Python, Perl or Bash. ... View more

@sankar rao The file stored in /tmp should be automatically removed when the job finishes. However, if the job does not finish properly (due to an error or some other problem), the files may not always be deleted. See here: https://cwiki.apache.org/confluence/display/Hive/AdminManual+Configuration Hive uses temporary folders both on the machine running the Hive client and the default HDFS instance. These folders are used to store per-query temporary/intermediate data sets and are normally cleaned up by the hive client when the query is finished. However, in cases of abnormal hive client termination, some data may be left behind. The configuration details are as follows: On the HDFS cluster this is set to /tmp/hive-<username> by default and is controlled by the configuration variable hive.exec.scratchdir On the client machine, this is hardcoded to /tmp/<username> Note that when writing data to a table/partition, Hive will first write to a temporary location on the target table's filesystem (using hive.exec.scratchdir as the temporary location) and then move the data to the target table. This applies in all cases - whether tables are stored in HDFS (normal case) or in file systems like S3 or even NFS. ... View more

@Gnanasekaran G As @Benjamin Leonhardi said, there is very little overhead to using an external table to do this. The only thing stored in the Hive Metastore is the schema about the CSV and the pointer to where the data is on HDFS. The data is left where you put it on HDFS. Using an external table is a very common way of solving this problem. Having said that, you can use Pig to load CSV data directly from HDFS. You have to define the schema for the CSV within the Pig script and you can write the data to a Hive ORC table. Be aware that the Hive ORC table must be created before you can write to it with Pig. Here is a tutorial that covers this: http://hortonworks.com/hadoop-tutorial/how-to-use-basic-pig-commands/ Here is an example of loading CSV data via Pig: STOCK_A = LOAD '/user/maria_dev/NYSE_daily_prices_A.csv' USING PigStorage(',') AS (exchange:chararray, symbol:chararray, date:chararray, open:float, high:float, low:float, close:float, volume:int, adj_close:float); DESCRIBE STOCK_A; ... View more

According to the documentation, you can't update partitioned or bucketed columns. https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-Update Partitioning columns cannot be updated. Bucketing columns cannot be updated. ... View more