Hi, I work with spatial data which is in netCDF format. To manipulate and analyze the data I use the NCO toolkit, which consists of a bunch of commands. I run these in the Windows CMD. Though this works fine, it is somewhat slow and cumbersome. Thus I would like to automate this with NiFi. I figured the ExecuteStreamCommand would do the job.
The netCDF data comes in via InvokeHTTP (generates a flowfile with ‘Content Type:application/x-netcdf’), and is subsequently fed to the ExecuteStreamCommand. I have it configured as follows
‘ncks’ is the executable command from the NCO Toolkit. This is on my path, such that I am sure NiFi can reach it. When I run the flow as follows
I receive errors on the ExecuteStreamCommand processor. These are as follows
2023-04-09 20:46:43,879 ERROR [Thread-187] o.a.n.p.standard.ExecuteStreamCommand ExecuteStreamCommand[id=65610950-0187-1000-7ef5-c5d0d5057e63] Failed to write FlowFile to Standard Input Stream
java.io.IOException: De pipe is beëindigd
at java.base/java.io.FileOutputStream.writeBytes(Native Method)
at java.base/java.io.FileOutputStream.write(FileOutputStream.java:354)
at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
at java.base/java.io.BufferedOutputStream.write(BufferedOutputStream.java:123)
at org.apache.nifi.stream.io.StreamUtils.copy(StreamUtils.java:36)
at org.apache.nifi.processors.standard.ExecuteStreamCommand.lambda$readStdoutReadable$1(ExecuteStreamCommand.java:633)
at java.base/java.lang.Thread.run(Thread.java:834)
2023-04-09 20:46:43,880 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.ExecuteStreamCommand ExecuteStreamCommand[id=65610950-0187-1000-7ef5-c5d0d5057e63] Transferring StandardFlowFileRecord[uuid=04a81cbc-9756-4346-b53a-b1d06555b0b7,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1681066003862-24, container=default, section=24], offset=0, length=7225],offset=0,name=edf174fc-eded-4480-bb09-5cf0adb9dc20,size=7225] to nonzero status. Executable command ncks ended in an error:
2023-04-09 20:46:44,363 INFO [Flow Service Tasks Thread-2] o.a.nifi.controller.StandardFlowService Saved flow controller org.apache.nifi.controller.FlowController@1344547c // Another save pending = false
Now to me these errors are weird… It says the executable command ended in an error, thus I double-checked, downloaded the output flowfile from InvokeHTTP and ran ncks ‘filename’ in CMD. This works fine. I also ran the flow whereby I ignored STDIN and added the filename to the ‘Command Arguments’ (I had the file on disk), this works fine as well, though I must note the resulting flow file still has ‘Content Type:application/x-netcdf’ which I did not expect… (this is mostly just annoying since I cannot view the contents when the files are queued, however UpdateAttribute can fix this problem)
Now I wonder whether filesize/computation-time makes for too much trouble. Though it is not necessary a very heavy file, nor does the 'ncks' computation take long (185 KB, and subsecond respectively)… thus I do not think this is an issue. At this point I feel like there is nothing wrong with the command ‘ncks’. The error ‘De pipe is beëindigd’ translates to ‘the pipe has been ended’. I figured I look into the NiFi source code to understand where things(the pipe) break, it is here
I note that it is a bug previously discovered, however I really do not understand why it occurs. I know I could potentially work around the issue, I could write the file to disk and access it from there, however this really is not my preferred solution. I could also write something in Python, etc. to extract the data with ExecuteScript. Again, not preferred.
My question boils down to two things, am I right the issue is with the bug, or am I doing something obvious/else wrong? If the problem is with the bug, how could I alter the source code such that it does work for my case? I do have some Java experience, however it is not my ‘best’ language as I mainly do analysis which is not in Java.
If needed I can supply additional info(pictures, etc.), I have tried quite a lot of things and spent a good amount of time on this problem. However thus far unsuccessful.
