deleting files after ingesting data

mark_hadoop — Wed, 09 Aug 2017 20:54:50 GMT

Hi,

I want to understand the below.

1. I have placed a log file in to hdfs

2. Then processed the log file in hdfs to csv format, say location of the csv is "/my/loc/" (both steps (1&2) using nifi).

3. created a table in hive "create external table load_table (col1, col2, ...) fields terminated by ',' lines terminated by '\n' stored as textfile location '/my/loc'

-----I want to do as below-----

4. copy the above data from load_table to orig_table(internal table)

5. remove the csv file (as the data is stream data in to csv, it may grow endless).

Things need clarification:

I have used puthiveql processor to do step 4, as insert overwrite table orig_table select col1, col2, ... from load_table; but I was getting error that "extraneous input ";" expecting EOF near <EOF>. I thought because of ";" in the insert statement, removed it but got "org.apache.hadoop.hive.ql.exec.movetask" error.

Also, for step 5, how can I remove csv, as streaming data is coming and storing in csv, this may grow huge.

Thank you

Re: deleting files after ingesting data

balavignesh_nag — Thu, 10 Aug 2017 14:01:06 GMT

@Hadoop User

Could you help with the complete log. It will be easier to sort it out if the complete log file is available.

Re: deleting files after ingesting data

mark_hadoop — Fri, 24 Nov 2017 00:16:12 GMT

@Bala Vignesh N V I have sorted it, thanks for your help

question Re: deleting files after ingesting data in Archives of Support Questions (Read Only)

deleting files after ingesting data

Re: deleting files after ingesting data

Re: deleting files after ingesting data