Sensor Data Analysis using HDP and HDF - Part 3

[#1]

Creating NIFI workflow

Use below steps to create the whole workflow.

1. Open the Nifi Web UI, download the below processors and configure as described.

a. MergeContent

b. InferAvroSchema

c. PutHive3streaming

108016-screen-shot-2019-04-18-at-111554-am.png

2. Now let’s configure them one by one. Right click each processor and configure as shown below.

a. MergeContent

107994-screen-shot-2019-04-18-at-110038-am.png

b. InferAvroSchema – In the CSV header definition put all the headers of your data separated by delimiter.

108024-screen-shot-2019-04-18-at-111642-am.png

c. PutHive3Streaming – Here we need to configure the record reader. Right click on the processor > configure > Record reader > create new service.

108064-screen-shot-2019-04-18-at-111727-am.png

d. Add CSV reader and save.

108007-screen-shot-2019-04-18-at-111800-am.png

e. Click on the record reader again and then enable by clicking on middle sign on upper right-side corner.

107995-screen-shot-2019-04-18-at-111831-am.png

f. Connect all three processor and address the errors if any. For any configuration reference, you can try importing the attached workflow.

I have attached my workflow in case you want to directly import into your Nifi.

Rasp_new.xml

Links to series

Part 1, Part 2, Part 3, Part 4, Part 5