Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

We have a requirement to process MVS EBCEDIC file and add two new fields in nifi. Is there anyway it can be achieved?

avatar
New Contributor

We have a requirement where we need to pick an MVS file from a server and add two new indicator fields. We are using nifi tool to process the data. Is this requirement achievable using nifi tool? If yes, what options are available?

1 ACCEPTED SOLUTION

avatar
Super Mentor
@Bhagyashree Chandrashekhar

NiFi at its core is aligned with any specific data format. NiFi can ingest data of any format (it is just treated as binary) The data is controlled in NiFi via a NiFi FlowFile. A FlowFile consist of Metadata about the actual content and the content itself. It si this FlowFile metadata that is the unit of transfer between processor components in a NiFi dataflow.

When it comes to different available processor components, they may care about the content format if their job is to manipulate that content in anyway. I a m not familiar with EBCEDIC data, but form what i have read of EBCEDIC legacy file format is that it can be either EBCEDIC-Binary or EBCEDIC-ASCII. If the format is ASCII, you may be able to use processors like ReplaceText to manipulate the actual content.

If it is binary, you may need to write your own custom NiFi processor component that would be able to process that data type. Or you might be able to use a one of NiFi's Scripting processors to use an external script to convert the format of these files to ASCII where you would then be able to modify them with existing processors.

There are no EBCEDIC specific processor in NIFi as of now.

Thank you,

Matt

View solution in original post

1 REPLY 1

avatar
Super Mentor
@Bhagyashree Chandrashekhar

NiFi at its core is aligned with any specific data format. NiFi can ingest data of any format (it is just treated as binary) The data is controlled in NiFi via a NiFi FlowFile. A FlowFile consist of Metadata about the actual content and the content itself. It si this FlowFile metadata that is the unit of transfer between processor components in a NiFi dataflow.

When it comes to different available processor components, they may care about the content format if their job is to manipulate that content in anyway. I a m not familiar with EBCEDIC data, but form what i have read of EBCEDIC legacy file format is that it can be either EBCEDIC-Binary or EBCEDIC-ASCII. If the format is ASCII, you may be able to use processors like ReplaceText to manipulate the actual content.

If it is binary, you may need to write your own custom NiFi processor component that would be able to process that data type. Or you might be able to use a one of NiFi's Scripting processors to use an external script to convert the format of these files to ASCII where you would then be able to modify them with existing processors.

There are no EBCEDIC specific processor in NIFi as of now.

Thank you,

Matt