Hi @heta desai You could be using HDF (NiFi) as your primary ingestion tool and not have to worry about the other options necessarily. That said, sqoop is primarily used to move data from an existing RDBMS to Hadoop (or vice versa). Flume was the main tool previously to ingest flat files, csv, etc, but has fallen out of favour and is often being replaced with HDF/NiFi now. Kafka is a distributed messaging system which can be used as a pub/sub model for data ingest, including streaming. So all three are a bit different. The right tool for the job depends on your use case, but as I said, HDF/NiFi can pretty much cover the gambit, so if you are starting out now, you may want to look at that first. Here is another good write up on the same subject: https://community.hortonworks.com/questions/23337/best-tools-to-ingest-data-to-hadoop.html As always, if you find this post useful, please accept the answer. ... View more

Hi @Narasimma varman the InvokeHTTP Processor in NiFi can be used. More on the InvokeHTTP processor here: https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.InvokeHTTP/ In fact, there is a good tutorial walking through how to get data from an API endpoint here: https://hortonworks.com/hadoop-tutorial/learning-ropes-apache-nifi/#section_5 As always, if you find this source useful, please accept the answer. ... View more

Hi @Cam Mach There isn't one for Kudu out of the box. However, you could either create a custom one, or create a flow that would land data on the box and then call the Kudu APIs (ExecuteScript processor maybe?) for the ingest. As always if you find this post useful, please accept the answer. ... View more

Hi @brian adi You can use the SplitJson processor to get the array into individual flowfile records, then use EvaluateJsonPath to pull out the values of interest, and ReplaceText to get the content back into a CSV format. Here is a good example with screenshots showing how to do this using NiFi https://community.hortonworks.com/articles/64069/converting-a-large-json-file-into-csv.html As always, if you find this post useful, don't forget to accept the answer. ... View more