NiFi ExecuteStreamCommand Processor Create Process...

techNerd · ‎01-23-2022

Hi, I am currently trying to count the number of files in the directory. The command that I keyed to count in the windows command prompt is "dir /a:-d /s /b "D:\Testing\UploadFile" | find /c ":"" I am trying to use NiFi to apply the above command using the ExecuteStreamCommand processor. Currently I am facing the "Access Denied error" as shown in the picture. Execute Stream Command Processor Error Execute Stream Command Processor Properties Appreciate if anyone could help with an example and explanation. Thanks. Note: the fileslocation set in the properties is the path to the file.

techNerd · ‎01-06-2022

Scenario: I have a List File Processor looking out for incoming file ("file 1"). I scheduled it to start picking up the "file 1" 20s after the "file 1" is downloaded. Let assume that the List File Processor noticed the incoming "file 1" and started the delay 20 sec before picking the "file 1". In the middle of 20 sec delay, there is an new incoming file ("file 2") noticed by the List File Processor. List File Processor will be interrupt and reset the 20 sec schedule time when ("file 2") appear within the previous 20 sec delay Could the List File Processor in the middle of 20 sec delay be interrupt and restart the initial delay? Appreciate help with explanation and example. Thanks.

techNerd · ‎01-04-2022

Hi, Would like to seek help in removing of additional backslash generated while transforming the incoming flowfile attribute (path + filename) into json content generating json file? Is there any regex I could used? looking forward to anyone helps with explanation with different methods and examples. Thanks. appreciate. Current Json output: { "combination of file path and filename" : "C:\\User\\test\\tester\\gameName.txt" } Envisage Json output: { "combination of file path and filename" : "C:\User\test\tester\gameName.txt" } Scenario: 1) Using GetFile NiFi Processor to retrieve a file from a specific location Directory : C:\User\test Note: Recurse Subdirectories = true 2) Using UpdateAttribute NiFi Processor to combine the file path and filename together The combination of file path and filename = "C:\User\test\tester\gameName.txt" 3) Using AttributesToJSON NiFi Processor to generate a json file that contain the no. 2 updated attribute. Example of current json file: { "combination of file path and filename" : "C:\\User\\test\\tester\\gameName.txt" }

techNerd · ‎12-07-2021

Hi, I would like to ask if there is ways for NiFi to know the number of files on the monitored directory (that have sub-directories that contains varies number of large files in different sub-directories) and waited for all the large files that are fetched initially processed before merging. Scenario: Monitor the number of files inside a directory (with sub-directories included). Used AttributeToJson NiFi Processor to change the incoming file content to the File Original Location attribute With the known number of files, Using NiFi MergeRecord Processor to merge all the file location. Note: Number of files varies. Maybe 4 files, 6 files, 10 files, etc. Depending on daily incoming files required The files might be 3GB, 5GB, 10 GB depending on daily incoming files The incoming Folder that contains all the files is not ZIP Could someone help me with example and explanation? Thanks. Appreciate a lot.

techNerd · ‎09-22-2021

Hi, I am trying to send files into Minio Object storage. Here are the scenario example: 1) Create files using NiFi GenerateFlowFile processor 2) Send the created files into Minio Object Storage using NiFi PutS3Object processor Note: In Minio Object Storage I created a bucket name called "test " It came out with this error: PutS3Object[id=0ba16837-017c-1000-46e7-b2c426d2b028] Error checking S3 Multipart Upload list for /test: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListMultipartUploadsHandler Here is the NiFi setup: GenerateFlowFile Processor: PutS3Object processor: My Minio Interface where bucket is created : Could someone please help me with this error with explanation and example ? Thanks & appreciate. Auto-refresh

techNerd · ‎08-25-2021

Hi, Would like to seek an example of caching the 2 singe flow file attribute (filename attribute) and store it for later part processing. Scenario: 1) Folder containing (file1.xml and file2.txt) 2) Using NiFi GetFile processor to get both file 3) Get the attribute of both files (filename) 4) store it for future storage into database Looking forward for the help with example and explanation. Appreciate the helps. Thanks

techNerd · ‎08-19-2021

Hi, In NiFi SFTP Processors (GetSFTP, PutSFTP, ListSFTP, FetchSFTP), we are required to input the credential (Username & Password) of the other platform that we are going to access (Computer A access to Computer B) In situation where other platform (Computer B) have it password changing every 3 month unknowingly and without affecting the automate data flow is there a way that we can handle this situation ? Thanks

techNerd · ‎06-27-2021

Hi @MattWho , Thanks for your help. Appreciated.

techNerd · ‎06-17-2021

Hi @MattWho Thanks for your help. Appreciate.

techNerd · ‎06-17-2021

Hi @MattWho The following is my use case: There are 2 computer A & B. Computer A has a folder named "LocalFolder" (example: C:\User\LocalFolder\) Computer B has a GetSFTP NiFi processor that listen to and extract files inside the Computer A "LocalFolder" Folder In odd cases, Computer A will generate a new folder named "newFolder" located in the "C:\User\LocalFolder" (example: C:\User\LocalFolder\newFolder) which is not known to Computer B (sudden appear). Is there any way computer B will 1) Get known of the new generated folder by Computer A and Computer B will generate the folder that is same at Computer A size 2) Extract all the file in computer A folder to Computer B newly generated folder Thank & it will be a good help if there is Example with internal NiFi configuration and explanation. Appreciate

techNerd · ‎06-16-2021

Hi @MattWho, Apologies as I made a mistake on the example. The example should be as follow: Input: <particular> <name>Nicholas</name> <height unit = "meter">183</height> <weight unit = "kilogram" unit = "lb">75</weight> </particular> <particular> <name>Debbie</name> <height unit = "meter">163</height> <weight unit = "kilogram" unit = "lb">45</weight> </particular> Output: <particular> <name>Nicholas</name> <height>183</height> <weight>75</weight> </particular> <particular> <name>Debbie</name> <height>163</height> <weight>45</weight> </particular> Are there some ways that NiFi processor could remove the attributes (unit or etc..) inside the XML tag name as mention in the example above? It will be a great help if there is an example with internal processor configuration shown and explanation. Appreciate your help. Thanks.

techNerd · ‎06-15-2021

Hi I currently have to extract folder from other computer via NiFi SFTP. I been finding for solution and notice that the NiFi processor (GetFile, GetSFTP) only allow to extract files instead of folder. Is there any nifi processor that is able to extract the folder? Example: Computer 1 directory: C:\User\localFolder Computer 2 Using GetSFTP processor (remote path: C:\User) to extract the folder Thanks

techNerd · ‎06-15-2021

Hi, is there some way that nifi could remove the attributes inside the XML tag name? Please advise with some example to help me. Appreciate. Thanks Example: <particular> <name>Nicholas</name> <height>183</height> <weight>75</weight> </particular> <particular> <name>Debbie</name> <height163</height> <weight>45</weight> </particular> to something like: <particular> <name>Nicholas</name> <height unit = "meter">183</height> <weight unit = "kilogram" unit = "lb">75</weight> </particular> <particular> <name>Debbie</name> <height unit = "meter">163</height> <weight unit = "kilogram" unit = "lb">45</weight> </particular>

techNerd · ‎05-14-2021

@MattWho Hi Matt, Thanks for the reply. I am sending a 1kb file over the rest endpoint of a different IP machine. What I am asking here is how do I configure the invokeHTTP request using IP address? Example: Machine 1 ( 192.168.4.1) which contain NiFi (GetFile processor --> invokeHTTP processor) to Machine 2 (192.168.3.1) which contain the Wildfly restful webservice. Thanks.

techNerd · ‎05-13-2021

Current setup, there are 2 computer machine 1) First machine (IP of 192.168.4.1) It contain the NiFi setup as show in the image below: Inside the NiFi setup, I get a file from a local directory and I invoke a HTTP POST to send over to the second machine which hit the Wildfly Restful Webservice endpoint. Below image show the configuration I have done inside the Invoke HTTP: (Note: the Remote URL: http://192.168.3.1:8081/JaxRS-Nifi/restapi/multipart/upload ) 2) Second machine (IP of 192.168.3.1) It contain a Wildfly Restful Webservice which allow my first machine to hit the webservice endpoint. Error that I am facing: I have a problem which the first machines doesn't send a HTTP POST over to the Second machines. I am currently looking for solutions on what are the different configurations should I do to make a NiFi HTTP over to hit the Restful Webservice endpoint? Thanks. Appreciate if there is anyone who can help me.

techNerd · ‎04-20-2021

Hi, I got a problem while sending a Zero Byte flow file through Nifi PutSFTP processor. Wondering if there is some configuration I have to set? thanks. Use case: get a file from a directory (GetFile Processor) send the file through SFTP (PutSFTP Processor) if success, change the filename to new filename (UpdateAttribute Processor) Remove the all of the content (ReplaceText Processor). Zero Byte flow file send the new filename through SFTP (PutSFTP Processor) I gotten this error while sending Zero Byte flow file over using PutSFTP Processor PutSFTP[id=1a091db4-1f9e-14cd-5206-1514485febc8] Rejecting StandardFlowFileRecord[uuid=247457a8-6608-4271-a626-3c016a1bbe7c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1618805512561-40, container=default, section=40], offset=15043, length=0],offset=0,name=AS_MPS_20200927_010010.xml.done,size=0] because it is zero bytes It will be great if someone could help me. Thanks in advance.

techNerd · ‎02-07-2021

Hi, Thanks for the good explanation. What would be the jolt specification for the following input/output. There are two input Json: 1st Json input: { "Name" : "Alvin", "Status" : "Single", "Life" : [ { "Sport" : "Swimming", "Singing" : "K-box", "Food" : "Burger", "Alcohol" : "Rum" }, { "Sport" : "Boxing", "Singing" : "party world", "Food" : "Chicken Wing", "Alcohol" : "Whisky" }, { "Sport" : "Running", "Singing" : "KTV", "Food" : "Muffin", "Alcohol" : "Martel" }] } 2nd Json input: { "Name" : "Alvin", "Status" : "Single", "Life" : { "Sport" : "Swimming", "Singing" : "K-box", "Food" : "Burger", "Alcohol" : "Rum" } } This two Json message input should go to a same JoltTransformJson processor and come out with the following output: 1st Json output: { "Name" : "Alvin", "Status" : "Single", "Sport" : [ "Swimming", "Boxing", "Running"], "Singing" : [ "K-box", "party world" , "KTV"], "Food" : [ "Burger", "Chicken Wing" , "Muffin"], "Alcohol" : [ "Rum", "Whisky", "Martel"] } 2nd Json output: { "Name" : "Alvin", "Status" : "Single", "Sport" : [ "Swimming"], "Singing" : [ "K-box"], "Food" : [ "Burger"], "Alcohol" : [ "Rum"] } How can I configure the JoltTransformJson processor to get the above output? Or is there any other ways to do it? Please advise with step and example. appreciate

techNerd · ‎02-05-2021

There are two input Json: 1st Json input: { "Name" : "Alex", "Status" : "Single", "Life" : [ { "Sport" : "Swimming", "Singing" : "K-box", "Food" : "Burger", "Alcohol" : "Rum" }, { "Sport" : "Boxing", "Singing" : "party world", "Food" : "Chicken Wing", "Alcohol" : "Whisky" }, { "Sport" : "Running", "Singing" : "KTV", "Food" : "Muffin", "Alcohol" : "Martel" }] } 2nd Json input: { "Name" : "Alex", "Status" : "Single", "Life" : { "Sport" : "Swimming", "Singing" : "K-box", "Food" : "Burger", "Alcohol" : "Rum" } } This two Json message input should go to a same JoltTransformJson processor and come out with the following output: 1st Json output: { "Name" : "Alex", "Status" : "Single", "Sport" : [ "Swimming", "Boxing", "Running"], "Singing" : [ "K-box", "party world" , "KTV"], "Food" : [ "Burger", "Chicken Wing" , "Muffin"], "Alcohol" : [ "Rum", "Whisky", "Martel"] } 2nd Json output: { "Name" : "Alex", "Status" : "Single", "Sport" : [ "Swimming"], "Singing" : [ "K-box"], "Food" : [ "Burger"], "Alcohol" : [ "Rum"] } How can I configure the JoltTransformJson processor to get the above output? Or is there any other ways to do it? Please advise with step and example. appreciate.

Online	Offline
Last Visited	‎01-24-2022 04:00 AM

Date Registered	‎02-05-2021 05:38 AM
Last Visited	‎01-24-2022 04:00 AM
Total Messages Posted	18
Total Kudos Received	1

NiFi ExecuteStreamCommand Processor Create Process...

NiFi List File Processor - Interrupt run schedule ...

Attribute to Json - removal of additional backslas...

Monitor the number of large size files (that varie...

Error that send files into Minio Object Storage

Merge 2 different incoming single flow file attrib...

NiFi SFTP - How to deal with credential changes ev...

Re: can nifi pull a folder instead of file

Re: How can I remove XML Tag Name attributes in Ni...

Re: can nifi pull a folder instead of file

Re: How can I remove XML Tag Name attributes in Ni...

can nifi pull a folder instead of file

How can I remove XML Tag Name attributes in NiFi?

Re: How can I use NiFi to Invoke a HTTP and send a...

How can I use NiFi to Invoke a HTTP and send a RES...

Zero Byte Error with PUTSFTP. Is there a way to se...

Re: Jolt quick reference for Nifi Jolt Processors

How can I used JoltTransformJson Processor to come...