hi, I need to create ETL pipeline, files are generated after hour and then these files are moved to HDFS now i want to load these files into Hive table using below command but i want to automate this process.

[#1]

LOAD DATA INPATH '/Project/' OVERWRITE INTO TABLE tb1;

[#2]

Let me know how you generate the files.

[#3]

JSON files are generated from twitter API using python now i want to load these files in HIVE tables but need to automate this process.

[#4]

You can write your processes into script file.

test.sh

# Run the python to generate JSON file

# hive -e "LOAD DATA INPATH '/Project/' OVERWRITE INTO TABLE tb1;"

# hive -e "select * from your table"

Run test.sh !