Member since
02-05-2021
18
Posts
1
Kudos Received
0
Solutions
01-23-2022
10:55 PM
Hi, I am currently trying to count the number of files in the directory. The command that I keyed to count in the windows command prompt is "dir /a:-d /s /b "D:\Testing\UploadFile" | find /c ":"" I am trying to use NiFi to apply the above command using the ExecuteStreamCommand processor. Currently I am facing the "Access Denied error" as shown in the picture. Execute Stream Command Processor Error Execute Stream Command Processor Properties Appreciate if anyone could help with an example and explanation. Thanks. Note: the fileslocation set in the properties is the path to the file.
... View more
Labels:
- Labels:
-
NiFi Registry
01-06-2022
10:52 PM
Scenario: I have a List File Processor looking out for incoming file ("file 1"). I scheduled it to start picking up the "file 1" 20s after the "file 1" is downloaded. Let assume that the List File Processor noticed the incoming "file 1" and started the delay 20 sec before picking the "file 1". In the middle of 20 sec delay, there is an new incoming file ("file 2") noticed by the List File Processor. List File Processor will be interrupt and reset the 20 sec schedule time when ("file 2") appear within the previous 20 sec delay Could the List File Processor in the middle of 20 sec delay be interrupt and restart the initial delay? Appreciate help with explanation and example. Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
01-04-2022
08:18 PM
Hi, Would like to seek help in removing of additional backslash generated while transforming the incoming flowfile attribute (path + filename) into json content generating json file? Is there any regex I could used? looking forward to anyone helps with explanation with different methods and examples. Thanks. appreciate. Current Json output: { "combination of file path and filename" : "C:\\User\\test\\tester\\gameName.txt" } Envisage Json output: { "combination of file path and filename" : "C:\User\test\tester\gameName.txt" } Scenario: 1) Using GetFile NiFi Processor to retrieve a file from a specific location Directory : C:\User\test Note: Recurse Subdirectories = true 2) Using UpdateAttribute NiFi Processor to combine the file path and filename together The combination of file path and filename = "C:\User\test\tester\gameName.txt" 3) Using AttributesToJSON NiFi Processor to generate a json file that contain the no. 2 updated attribute. Example of current json file: { "combination of file path and filename" : "C:\\User\\test\\tester\\gameName.txt" }
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
12-07-2021
07:33 PM
Hi, I would like to ask if there is ways for NiFi to know the number of files on the monitored directory (that have sub-directories that contains varies number of large files in different sub-directories) and waited for all the large files that are fetched initially processed before merging. Scenario: Monitor the number of files inside a directory (with sub-directories included). Used AttributeToJson NiFi Processor to change the incoming file content to the File Original Location attribute With the known number of files, Using NiFi MergeRecord Processor to merge all the file location. Note: Number of files varies. Maybe 4 files, 6 files, 10 files, etc. Depending on daily incoming files required The files might be 3GB, 5GB, 10 GB depending on daily incoming files The incoming Folder that contains all the files is not ZIP Could someone help me with example and explanation? Thanks. Appreciate a lot.
... View more
- Tags:
- NiFi
Labels:
- Labels:
-
Apache NiFi
09-22-2021
01:57 AM
Hi, I am trying to send files into Minio Object storage. Here are the scenario example: 1) Create files using NiFi GenerateFlowFile processor 2) Send the created files into Minio Object Storage using NiFi PutS3Object processor Note: In Minio Object Storage I created a bucket name called "test " It came out with this error: PutS3Object[id=0ba16837-017c-1000-46e7-b2c426d2b028] Error checking S3 Multipart Upload list for /test: Failed to parse XML document with handler class com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListMultipartUploadsHandler Here is the NiFi setup: GenerateFlowFile Processor: PutS3Object processor: My Minio Interface where bucket is created : Could someone please help me with this error with explanation and example ? Thanks & appreciate. Auto-refresh
... View more
Labels:
- Labels:
-
Apache NiFi
08-25-2021
03:39 AM
Hi, Would like to seek an example of caching the 2 singe flow file attribute (filename attribute) and store it for later part processing. Scenario: 1) Folder containing (file1.xml and file2.txt) 2) Using NiFi GetFile processor to get both file 3) Get the attribute of both files (filename) 4) store it for future storage into database Looking forward for the help with example and explanation. Appreciate the helps. Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
-
NiFi Registry
08-19-2021
07:06 PM
Hi, In NiFi SFTP Processors (GetSFTP, PutSFTP, ListSFTP, FetchSFTP), we are required to input the credential (Username & Password) of the other platform that we are going to access (Computer A access to Computer B) In situation where other platform (Computer B) have it password changing every 3 month unknowingly and without affecting the automate data flow is there a way that we can handle this situation ? Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
06-17-2021
08:06 PM
Hi @MattWho Thanks for your help. Appreciate.
... View more
06-17-2021
06:58 PM
Hi @MattWho The following is my use case: There are 2 computer A & B. Computer A has a folder named "LocalFolder" (example: C:\User\LocalFolder\) Computer B has a GetSFTP NiFi processor that listen to and extract files inside the Computer A "LocalFolder" Folder In odd cases, Computer A will generate a new folder named "newFolder" located in the "C:\User\LocalFolder" (example: C:\User\LocalFolder\newFolder) which is not known to Computer B (sudden appear). Is there any way computer B will 1) Get known of the new generated folder by Computer A and Computer B will generate the folder that is same at Computer A size 2) Extract all the file in computer A folder to Computer B newly generated folder Thank & it will be a good help if there is Example with internal NiFi configuration and explanation. Appreciate
... View more
06-16-2021
07:09 PM
Hi @MattWho, Apologies as I made a mistake on the example. The example should be as follow: Input: <particular> <name>Nicholas</name> <height unit = "meter">183</height> <weight unit = "kilogram" unit = "lb">75</weight> </particular> <particular> <name>Debbie</name> <height unit = "meter">163</height> <weight unit = "kilogram" unit = "lb">45</weight> </particular> Output: <particular> <name>Nicholas</name> <height>183</height> <weight>75</weight> </particular> <particular> <name>Debbie</name> <height>163</height> <weight>45</weight> </particular> Are there some ways that NiFi processor could remove the attributes (unit or etc..) inside the XML tag name as mention in the example above? It will be a great help if there is an example with internal processor configuration shown and explanation. Appreciate your help. Thanks.
... View more
06-15-2021
09:17 PM
Hi I currently have to extract folder from other computer via NiFi SFTP. I been finding for solution and notice that the NiFi processor (GetFile, GetSFTP) only allow to extract files instead of folder. Is there any nifi processor that is able to extract the folder? Example: Computer 1 directory: C:\User\localFolder Computer 2 Using GetSFTP processor (remote path: C:\User) to extract the folder Thanks
... View more
Labels:
- Labels:
-
Apache NiFi
06-15-2021
06:36 PM
Hi, is there some way that nifi could remove the attributes inside the XML tag name? Please advise with some example to help me. Appreciate. Thanks Example: <particular> <name>Nicholas</name> <height>183</height> <weight>75</weight> </particular> <particular> <name>Debbie</name> <height163</height> <weight>45</weight> </particular> to something like: <particular> <name>Nicholas</name> <height unit = "meter">183</height> <weight unit = "kilogram" unit = "lb">75</weight> </particular> <particular> <name>Debbie</name> <height unit = "meter">163</height> <weight unit = "kilogram" unit = "lb">45</weight> </particular>
... View more
- Tags:
- NiFi
- xml
- xml tag name
Labels:
- Labels:
-
Apache NiFi
05-14-2021
08:54 AM
@MattWho Hi Matt, Thanks for the reply. I am sending a 1kb file over the rest endpoint of a different IP machine. What I am asking here is how do I configure the invokeHTTP request using IP address? Example: Machine 1 ( 192.168.4.1) which contain NiFi (GetFile processor --> invokeHTTP processor) to Machine 2 (192.168.3.1) which contain the Wildfly restful webservice. Thanks.
... View more
05-13-2021
11:57 PM
Current setup, there are 2 computer machine 1) First machine (IP of 192.168.4.1) It contain the NiFi setup as show in the image below: Inside the NiFi setup, I get a file from a local directory and I invoke a HTTP POST to send over to the second machine which hit the Wildfly Restful Webservice endpoint. Below image show the configuration I have done inside the Invoke HTTP: (Note: the Remote URL: http://192.168.3.1:8081/JaxRS-Nifi/restapi/multipart/upload ) 2) Second machine (IP of 192.168.3.1) It contain a Wildfly Restful Webservice which allow my first machine to hit the webservice endpoint. Error that I am facing: I have a problem which the first machines doesn't send a HTTP POST over to the Second machines. I am currently looking for solutions on what are the different configurations should I do to make a NiFi HTTP over to hit the Restful Webservice endpoint? Thanks. Appreciate if there is anyone who can help me.
... View more
- Tags:
- NiFi
Labels:
- Labels:
-
Apache NiFi
04-20-2021
12:41 AM
Hi, I got a problem while sending a Zero Byte flow file through Nifi PutSFTP processor. Wondering if there is some configuration I have to set? thanks. Use case: get a file from a directory (GetFile Processor) send the file through SFTP (PutSFTP Processor) if success, change the filename to new filename (UpdateAttribute Processor) Remove the all of the content (ReplaceText Processor). Zero Byte flow file send the new filename through SFTP (PutSFTP Processor) I gotten this error while sending Zero Byte flow file over using PutSFTP Processor PutSFTP[id=1a091db4-1f9e-14cd-5206-1514485febc8] Rejecting StandardFlowFileRecord[uuid=247457a8-6608-4271-a626-3c016a1bbe7c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1618805512561-40, container=default, section=40], offset=15043, length=0],offset=0,name=AS_MPS_20200927_010010.xml.done,size=0] because it is zero bytes It will be great if someone could help me. Thanks in advance.
... View more
Labels:
02-07-2021
06:45 PM
Hi, Thanks for the good explanation. What would be the jolt specification for the following input/output. There are two input Json: 1st Json input: { "Name" : "Alvin", "Status" : "Single", "Life" : [ { "Sport" : "Swimming", "Singing" : "K-box", "Food" : "Burger", "Alcohol" : "Rum" }, { "Sport" : "Boxing", "Singing" : "party world", "Food" : "Chicken Wing", "Alcohol" : "Whisky" }, { "Sport" : "Running", "Singing" : "KTV", "Food" : "Muffin", "Alcohol" : "Martel" }] } 2nd Json input: { "Name" : "Alvin", "Status" : "Single", "Life" : { "Sport" : "Swimming", "Singing" : "K-box", "Food" : "Burger", "Alcohol" : "Rum" } } This two Json message input should go to a same JoltTransformJson processor and come out with the following output: 1st Json output: { "Name" : "Alvin", "Status" : "Single", "Sport" : [ "Swimming", "Boxing", "Running"], "Singing" : [ "K-box", "party world" , "KTV"], "Food" : [ "Burger", "Chicken Wing" , "Muffin"], "Alcohol" : [ "Rum", "Whisky", "Martel"] } 2nd Json output: { "Name" : "Alvin", "Status" : "Single", "Sport" : [ "Swimming"], "Singing" : [ "K-box"], "Food" : [ "Burger"], "Alcohol" : [ "Rum"] } How can I configure the JoltTransformJson processor to get the above output? Or is there any other ways to do it? Please advise with step and example. appreciate
... View more
02-05-2021
06:11 AM
1 Kudo
There are two input Json: 1st Json input: { "Name" : "Alex", "Status" : "Single", "Life" : [ { "Sport" : "Swimming", "Singing" : "K-box", "Food" : "Burger", "Alcohol" : "Rum" }, { "Sport" : "Boxing", "Singing" : "party world", "Food" : "Chicken Wing", "Alcohol" : "Whisky" }, { "Sport" : "Running", "Singing" : "KTV", "Food" : "Muffin", "Alcohol" : "Martel" }] } 2nd Json input: { "Name" : "Alex", "Status" : "Single", "Life" : { "Sport" : "Swimming", "Singing" : "K-box", "Food" : "Burger", "Alcohol" : "Rum" } } This two Json message input should go to a same JoltTransformJson processor and come out with the following output: 1st Json output: { "Name" : "Alex", "Status" : "Single", "Sport" : [ "Swimming", "Boxing", "Running"], "Singing" : [ "K-box", "party world" , "KTV"], "Food" : [ "Burger", "Chicken Wing" , "Muffin"], "Alcohol" : [ "Rum", "Whisky", "Martel"] } 2nd Json output: { "Name" : "Alex", "Status" : "Single", "Sport" : [ "Swimming"], "Singing" : [ "K-box"], "Food" : [ "Burger"], "Alcohol" : [ "Rum"] } How can I configure the JoltTransformJson processor to get the above output? Or is there any other ways to do it? Please advise with step and example. appreciate.
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)