Member since
07-29-2020
574
Posts
323
Kudos Received
176
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1955 | 12-20-2024 05:49 AM | |
2184 | 12-19-2024 08:33 PM | |
2001 | 12-19-2024 06:48 AM | |
1324 | 12-17-2024 12:56 PM | |
1851 | 12-16-2024 04:38 AM |
10-16-2024
11:51 AM
@SAMSAL Thanks so much - Here are my further findings... Here is my current directory structure - I tried a bunch of different ways to zip this and put the .nar in the corresponding NIFI_HOME/lib directory. Here is each test case running on my Mac OS. Please ignore the naming conventions as I was trying to go through this quickly 😊 Test Case 1: Normal zip Failed zip -r archive_name.zip TransformOpenskyStates.py NAR-INF/ META-INF/ cp archive_name.zip ../Downloads/nifi-2.0.0-M4/lib/archive_name.nar Restarting NiFi logs: 2024-10-15 15:52:07,787 ERROR [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.lang.NullPointerException: Cannot invoke "org.apache.nifi.nar.ExtensionMapping.size()" because "extensionMapping" is null at org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:64) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:833) at org.apache.nifi.NiFi.<init>(NiFi.java:172) at org.apache.nifi.NiFi.<init>(NiFi.java:83) at org.apache.nifi.NiFi.main(NiFi.java:332) ---- Test Case 2: Excluding Mac OS metadata files in Zip Command Failed zip -r archive_name.zip TransformOpenskyStates.py NAR-INF/ META-INF/ -x "*.DS_Store" -x "__MACOSX/*" cp archive_name.zip ../Downloads/nifi-2.0.0-M4/lib/archive_name.nar Restarting NiFi logs: 2024-10-15 15:52:07,787 ERROR [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.lang.NullPointerException: Cannot invoke "org.apache.nifi.nar.ExtensionMapping.size()" because "extensionMapping" is null at org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:64) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:833) at org.apache.nifi.NiFi.<init>(NiFi.java:172) at org.apache.nifi.NiFi.<init>(NiFi.java:83) at org.apache.nifi.NiFi.main(NiFi.java:332) ---- Test Case 3: Normal Zip Command Outer Folder Failed zip -r test.zip TransformOS-nar/ cp test.zip ../Downloads/nifi-2.0.0-M4/lib/archive_name.nar Restarting NiFi logs: java.lang.NullPointerException: Cannot invoke "java.util.jar.Manifest.getMainAttributes()" because "manifest" is null at org.apache.nifi.nar.NarUnpacker.createBundleCoordinate(NarUnpacker.java:258) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:140) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:90). at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:84) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:75) at org.apache.nifi.NiFi.<init>(NiFi.java:142) at org.apache.nifi.NiFi.<init>(NiFi.java:83) at org.apache.nifi.NiFi.main(NiFi.java:332) ---- Test Case 4: Normal Zip Command Outer Folder with exclusions Failed zip -r archive_name.zip TransformOS-nar/ -x "*.DS_Store" -x "__MACOSX/*" cp archive_name.zip ../Downloads/nifi-2.0.0-M4/lib/archive_name.nar Restarting NiFi logs: java.lang.NullPointerException: Cannot invoke "java.util.jar.Manifest.getMainAttributes()" because "manifest" is null at org.apache.nifi.nar.NarUnpacker.createBundleCoordinate(NarUnpacker.java:258) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:140) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:90). at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:84) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:75) at org.apache.nifi.NiFi.<init>(NiFi.java:142) at org.apache.nifi.NiFi.<init>(NiFi.java:83) at org.apache.nifi.NiFi.main(NiFi.java:332) ---- Test Case 5: Running Zip Command inside TransformOS-nar directory with * Passed!! zip -r archive_name.zip * cp archive_name.zip ../Downloads/nifi-2.0.0-M4/lib/archive_name.nar -------------------------------------------- Now you are probably wondering what the difference is between Test Case 1 and Test Case 5 is... To go in further detail, here's the output of each zip command - Zip command output for Test Case 1: drewnicolette@MacBook-Pro-2:~/TransformOS-nar $ zip -r archive_name.zip TransformOpenskyStates.py NAR-INF/ META-INF/ adding: TransformOpenskyStates.py (deflated 60%) adding: NAR-INF/ (stored 0%) adding: NAR-INF/.DS_Store (deflated 96%) adding: NAR-INF/bundled-dependencies/ (stored 0%) adding: META-INF/ (stored 0%) adding: META-INF/MANIFEST.MF (deflated 13%) adding: META-INF/.DS_Store (deflated 97%) Zip command output for Test Case 5: drewnicolette@MacBook-Pro-2:~/TransformOS-nar $ zip -r archive_name.zip * adding: META-INF/ (stored 0%) adding: META-INF/MANIFEST.MF (deflated 13%) adding: META-INF/.DS_Store (deflated 97%) adding: NAR-INF/ (stored 0%) adding: NAR-INF/.DS_Store (deflated 96%) adding: NAR-INF/bundled-dependencies/ (stored 0%) adding: TransformOpenskyStates.py (deflated 60%) It seems like it's zipping up the same files and the byte count for each .zip file are the same! Please see screenshot below! Solution: Order matters when zipping up the file! If you look above in Test Case 1, when I ran the zip command, it added TransformOpenskyStates.py first, while in Test Case 5 it added META-INF/ first. I believe NiFi code is expecting META-INF/ first?? @MattWho - Do you know anything about this?
... View more
10-09-2024
08:37 AM
1 Kudo
Can you break the data into smaller chunks ? this way you speed up the process by taking advantage of parallelism and multi threading.
... View more
10-04-2024
07:59 AM
1 Kudo
i had the same issue i made sudo apt install python3-venv restarted apache nifi and it worked
... View more
10-04-2024
04:54 AM
@mjmoore
Have you been able to resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future.
... View more
09-20-2024
07:49 AM
1 Kudo
Hey @SAMSAL Amazing response - thank you! I'll go over a couple of things you have mentioned, firstly the overwriting... I was wrong about that one. When I converted it to CSV instead using the 2 sheets, it actually appends the information from sheet 2 to the end of sheet 1 (leaving 2 rows as a gap first) so in your example above you would actually end up with something like this in CSV (when reimported into XL): I'm not sure if I can work with that, as I don't see any way in any of the readers to be able to "split" that (or only reading from row 2 to X for example) whilst keeping the flow etc and just using some of the values The other issue is with the Avro you have set up, because we don't know what the column headers will be (field names) I'm unable to define a schema in Avro for this as they can add and remove these columns at will - the only thing I will be able to rely on is that both the sheets will have identical columns. When I convert direct to JSON, I get something more like this: [ {
"column_0" : "ID",
"column_1" : "NAME",
"column_2" : "ADDRESS"
}, {
"column_0" : "1",
"column_1" : "Sam",
"column_2" : "TX"
}, {
"column_0" : "2",
"column_1" : "Ali",
"column_2" : "FL"
}] Again, not sure this is something I can use in this format but I'm just not familiar enough with the NiFi elements at this stage I guess I'll try the enrichment fork method though and see if there is some mileage in that and then I can feed back
... View more
09-17-2024
11:19 AM
@moonspa Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
... View more
09-12-2024
06:22 AM
1 Kudo
Thanks a lot, @SAMSAL! The proposed solution worked for us.
... View more
09-11-2024
06:17 AM
@moshell281 Has the reply helped resolve your issue? If so, please mark the appropriate reply as the solution, as it will make it easier for others to find the answer in the future. Thanks.
... View more
09-09-2024
03:47 PM
Thanks @SAMSAL for the clarification and the alternatives for looping in Nifi. We will consider using DuplicateFlowFile processor. Do you think nifi documentation should be updated to explicitly mention that it is not only for load tests but also can be used in production flows when there is need to clone flow files ?
... View more
09-08-2024
08:38 PM
can I keep the JSON file as a constatnts in nifi itself? so that I can avoid the invokeHTTP call. every time I received a flow data, I just need to check the id from the flow data and using that i need to find the correspondent entry from the conastant json file and append those in to the flow data.
... View more