Apache NiFi Journey from HDF 3.0 to 3.1 - Part 1

[#1]

Apache NiFi evolution from version 1.2 included in HDF 3.0 and version 1.5 included in HDF is significant. I find myself quite often puzzled when required to provide differences between releases and just reading the release notes history at https://cwiki.apache.org/confluence/display/NIFI/Release+Notes and looking at the latest list of NiFi processors is not trivial to determine which new processors were added.

I put together matrix which I hope will help developers to take advantage of new processor to improve old and develop new flows.

In a nutshell, main functionality added is around:

• AzureEventHub
• Kafka 0.11 and 1.0 processors
• Record processors
• RethinkDB
• Flatten Json
• Execute Spark Interactive
• Execute Groovy Script

My favorite improvements are around record processors, flattening JSON and executing Spark interactively.

The following is a table of the matrix, arranged alphabetically from A-D:

• See here for the Matrix Table from E-J
• See here for the Matrix Tabke from K-Z

For

NiFi 1.5	NiFi 1.4	NiFi 1.3	NiFi 1.2
AttributeRollingWindow	AttributeRollingWindow	AttributeRollingWindow	AttributeRollingWindow
AttributesToJSON	AttributesToJSON	AttributesToJSON	AttributesToJSON
Base64EncodeContent	Base64EncodeContent	Base64EncodeContent	Base64EncodeContent
CaptureChangeMySQL	CaptureChangeMySQL	CaptureChangeMySQL	CaptureChangeMySQL
CompareFuzzyHash	CompareFuzzyHash	CompareFuzzyHash	CompareFuzzyHash
CompressContent	CompressContent	CompressContent	CompressContent
ConnectWebSocket	ConnectWebSocket	ConnectWebSocket	ConnectWebSocket
ConsumeAMQP	ConsumeAMQP	ConsumeAMQP	ConsumeAMQP
ConsumeAzureEventHub
ConsumeEWS	ConsumeEWS	ConsumeEWS	ConsumeEWS
ConsumeIMAP	ConsumeIMAP	ConsumeIMAP	ConsumeIMAP
ConsumeJMS	ConsumeJMS	ConsumeJMS	ConsumeJMS
ConsumeKafka	ConsumeKafka	ConsumeKafka	ConsumeKafka
ConsumeKafka_0_10	ConsumeKafka_0_10	ConsumeKafka_0_10	ConsumeKafka_0_10
ConsumeKafka_0_11	ConsumeKafka_0_11	ConsumeKafkaRecord_0_10	ConsumeKafkaRecord_0_10
ConsumeKafkaRecord_0_10	ConsumeKafkaRecord_0_10
ConsumeKafkaRecord_0_11	ConsumeKafkaRecord_0_11
ConsumeKafka_1_0
ConsumeKafkaRecord_1_0
ConsumeMQTT	ConsumeMQTT	ConsumeMQTT	ConsumeMQTT
ConsumePOP3	ConsumePOP3	ConsumePOP3	ConsumePOP3
ConsumeWindowsEventLog	ConsumeWindowsEventLog	ConsumeWindowsEventLog	ConsumeWindowsEventLog
ControlRate	ControlRate	ControlRate	ControlRate
ConvertAvroSchema	ConvertAvroSchema	ConvertAvroSchema	ConvertAvroSchema
ConvertAvroToJSON	ConvertAvroToJSON	ConvertAvroToJSON	ConvertAvroToJSON
ConvertAvroToORC	ConvertAvroToORC	ConvertAvroToORC	ConvertAvroToORC
ConvertCharacterSet	ConvertCharacterSet	ConvertCharacterSet	ConvertCharacterSet
ConvertCSVToAvro	ConvertCSVToAvro	ConvertCSVToAvro	ConvertCSVToAvro
ConvertExcelToCSVProcessor	ConvertExcelToCSVProcessor	ConvertExcelToCSVProcessor	ConvertExcelToCSVProcessor
ConvertJSONToAvro	ConvertJSONToAvro	ConvertJSONToAvro	ConvertJSONToAvro
ConvertJSONToSQL	ConvertJSONToSQL	ConvertJSONToSQL	ConvertJSONToSQL
ConvertRecord	ConvertRecord	ConvertRecord	ConvertRecord
CreateHadoopSequenceFile	CreateHadoopSequenceFile	CreateHadoopSequenceFile	CreateHadoopSequenceFile
CountText
DebugFlow	DebugFlow	DebugFlow	DebugFlow
DeleteDynamoDB	DeleteDynamoDB	DeleteDynamoDB	DeleteDynamoDB
DeleteGCSObject	DeleteGCSObject	DeleteGCSObject	DeleteGCSObject
DeleteHDFS	DeleteHDFS	DeleteHDFS	DeleteHDFS
DeleteElasticsearch5	DeleteElasticsearch5
DeleteRethinkDB	DeleteRethinkDB
DeleteS3Object	DeleteS3Object	DeleteS3Object	DeleteS3Object
DeleteMongo
DeleteSQS	DeleteSQS	DeleteSQS	DeleteSQS
DetectDuplicate	DetectDuplicate	DetectDuplicate	DetectDuplicate
DistributeLoad	DistributeLoad	DistributeLoad	DistributeLoad
DuplicateFlowFile	DuplicateFlowFile	DuplicateFlowFile	DuplicateFlowFile