About manishg

manishg · ‎09-27-2023

Got it. Its on Data Provenance dialog box.

manishg · ‎09-27-2023

Where is this graph available on ui? And does it get updated after every run?

manishg · ‎09-27-2023

I have a nifi flow. I observe that if input file is split into smaller files and fed into the flow one by one, then overall time taken(sum of time taken for individual files) is considerably low compared to when I feed single big file. What can be a possible cause for this performance difference? Note: Flow has many processors that use avro readers/writers. I calculate time using following in a LogMessage processor: ${now():toNumber():minus(${lineageStartDate}):format("HH:mm:‌ss", "GMT")}

manishg · ‎09-21-2023

Please ignore this question. My understanding of the problem itself was wrong.

manishg · ‎09-21-2023

I have records with following structure: { field1 : val1 field2: val2 fields3: x::y::z field4: a::b::c ..... } field3 and field4 are basically pairs..ie x is paired with a, y with b, z with c. I want to break this record to reflect above pairing. So it would result in 3 records, with first one having field3 as x and field4 as a, second record having field3 as y and field4 as b and so on. Can it be achieved with nifi dsl?

manishg · ‎09-16-2023

I am using ListFile processor to pick up input files to my flow. And it picks up all available files in the directory. Is it possible to configure it to pick only one file, let its processing to be completed and then pick next file. So basically only one file at a time to be processed by system. Or if any processor other than ListFile can be used for this purpose?

manishg · ‎09-15-2023

This served my purpose. ${now():toNumber():minus(${lineageStartDate}):format("HH:mm:‌ss", "GMT")} I added it to a LogMessage processor, and I can see the result in logs, though sending it over a mail would be terrific.

manishg · ‎09-15-2023

What is most convenient way of determining total time taken by a group of consecutive processors? Do I need to log time before first and after last processor, and then take the diff? Or there is some inbuild feature that can help me on this?

manishg · ‎09-14-2023

Its a custom processor, and it reads avro records(using AvroReader) and writes them too(using AvroRecordSetWriter).

manishg · ‎09-14-2023

Its ListFile.

Online	Offline
Last Visited	‎06-10-2024 06:50 AM

Member Since	‎07-27-2023 04:53 AM
Last Visited	‎06-10-2024 06:50 AM
Posts	55
Kudos received	19

Cloudera Community

Re: Nifi: file not picked from input directory aft...

Re: Performance diff between single big file vs mu...

Re: Performance diff between single big file vs mu...

Performance diff between single big file vs multip...

Re: nifi dsl: breaking a record into multiple reco...

nifi dsl: breaking a record into multiple records

Process only one file at a time

Re: Detemine total time taken by a group of proces...

Detemine total time taken by a group of processors...

Re: 'Records Processed' missing for a processor

Re: Nifi: file not picked from input directory aft...