This bug was because of permission issue. (Reinstallation of Ambari agent) The error was because of /tmp/parquet-0.log and /tmp/parquet-0.log.lock access denying for hive user. ... View more

Hi @Mehdi Hosseinzadeh, From the requirements prospective, following is the simplistic approach which will be inline with technologies which you proposed. Read the data From HTTP using Spark Streaming job and write into Kafka Read & process data from Kafka Topic as batches/stream save the data into HDFS as parquet / Avaro /ORC etc.. Build an external Tables in Hive(on top of the data which processed in step 2) so that data is available as and when it is placed in HDFS Accessing the data from external tables has been discussed here ... View more