@nifier - I emulated the same setup as you via a testing S3 bucket. Using an access key id, and secret access key that had full access to all of S3, I was able to receive all the objects (including recursive ones) from that bucket.   A couple questions I want to follow up with ...  1. In your ListS3 processor configuration, do you have anything set for prefix or delimiter? Just want to make sure because that could be filtering some files/directories coming from S3. 2. What is your IAM role and the bucket policy you are trying to consume from? Are you certain that the role you are using can access to all the objects in the bucket? ... View more

@AndreyDE  Is one flowfile going into the SplitText processor and outputting 10000 flowfiles? How big is the flowfile going into the SplitText processor? Or is the source of the pipeline recursively getting all objects in your S3 bucket?  I need to a little bit more about the input going into SplitText?  ... View more

Hi @AndreyDE , What's your input into the SplitFile processor? I used your example and getting a valid output -  Make sure the file going into the SplitText is not re-reading the same file over and over again and also if you are using generateFlowFile make sure the scheduling isn't set to 0 sec because it will keep outputting a bunch of flowfiles.  Please accept this solution if it's correct, thanks! ... View more

@SAMSAL  Thanks so much - Here are my further findings... Here is my current directory structure -  I tried a bunch of different ways to zip this and put the .nar in the corresponding NIFI_HOME/lib directory. Here is each test case running on my Mac OS. Please ignore the naming conventions as I was trying to go through this quickly 😊   Test Case 1: Normal zip  Failed zip -r archive_name.zip TransformOpenskyStates.py NAR-INF/ META-INF/ cp archive_name.zip ../Downloads/nifi-2.0.0-M4/lib/archive_name.nar Restarting NiFi logs: 2024-10-15 15:52:07,787 ERROR [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.lang.NullPointerException: Cannot invoke "org.apache.nifi.nar.ExtensionMapping.size()" because "extensionMapping" is null at org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:64) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:833) at org.apache.nifi.NiFi.<init>(NiFi.java:172) at org.apache.nifi.NiFi.<init>(NiFi.java:83) at org.apache.nifi.NiFi.main(NiFi.java:332) ---- Test Case 2: Excluding Mac OS metadata files in Zip Command Failed  zip -r archive_name.zip TransformOpenskyStates.py NAR-INF/ META-INF/ -x "*.DS_Store" -x "__MACOSX/*" cp archive_name.zip ../Downloads/nifi-2.0.0-M4/lib/archive_name.nar Restarting NiFi logs: 2024-10-15 15:52:07,787 ERROR [main] org.apache.nifi.web.server.JettyServer Failed to start web server... shutting down. java.lang.NullPointerException: Cannot invoke "org.apache.nifi.nar.ExtensionMapping.size()" because "extensionMapping" is null at org.apache.nifi.documentation.DocGenerator.generate(DocGenerator.java:64) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:833) at org.apache.nifi.NiFi.<init>(NiFi.java:172) at org.apache.nifi.NiFi.<init>(NiFi.java:83) at org.apache.nifi.NiFi.main(NiFi.java:332) ---- Test Case 3: Normal Zip Command Outer Folder Failed  zip -r test.zip TransformOS-nar/ cp test.zip ../Downloads/nifi-2.0.0-M4/lib/archive_name.nar Restarting NiFi logs: java.lang.NullPointerException: Cannot invoke "java.util.jar.Manifest.getMainAttributes()" because "manifest" is null at org.apache.nifi.nar.NarUnpacker.createBundleCoordinate(NarUnpacker.java:258) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:140) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:90). at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:84) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:75) at org.apache.nifi.NiFi.<init>(NiFi.java:142) at org.apache.nifi.NiFi.<init>(NiFi.java:83) at org.apache.nifi.NiFi.main(NiFi.java:332) ---- Test Case 4: Normal Zip Command Outer Folder with exclusions Failed  zip -r archive_name.zip TransformOS-nar/ -x "*.DS_Store" -x "__MACOSX/*" cp archive_name.zip ../Downloads/nifi-2.0.0-M4/lib/archive_name.nar Restarting NiFi logs: java.lang.NullPointerException: Cannot invoke "java.util.jar.Manifest.getMainAttributes()" because "manifest" is null at org.apache.nifi.nar.NarUnpacker.createBundleCoordinate(NarUnpacker.java:258) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:140) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:90). at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:84) at org.apache.nifi.nar.NarUnpacker.unpackNars(NarUnpacker.java:75) at org.apache.nifi.NiFi.<init>(NiFi.java:142) at org.apache.nifi.NiFi.<init>(NiFi.java:83) at org.apache.nifi.NiFi.main(NiFi.java:332) ---- Test Case 5: Running Zip Command inside TransformOS-nar directory with * Passed!! zip -r archive_name.zip * cp archive_name.zip ../Downloads/nifi-2.0.0-M4/lib/archive_name.nar -------------------------------------------- Now you are probably wondering what the difference is between Test Case 1 and Test Case 5 is... To go in further detail, here's the output of each zip command - Zip command output for Test Case 1: drewnicolette@MacBook-Pro-2:~/TransformOS-nar $ zip -r archive_name.zip TransformOpenskyStates.py NAR-INF/ META-INF/ adding: TransformOpenskyStates.py (deflated 60%) adding: NAR-INF/ (stored 0%) adding: NAR-INF/.DS_Store (deflated 96%) adding: NAR-INF/bundled-dependencies/ (stored 0%) adding: META-INF/ (stored 0%) adding: META-INF/MANIFEST.MF (deflated 13%) adding: META-INF/.DS_Store (deflated 97%) Zip command output for Test Case 5: drewnicolette@MacBook-Pro-2:~/TransformOS-nar $ zip -r archive_name.zip * adding: META-INF/ (stored 0%) adding: META-INF/MANIFEST.MF (deflated 13%) adding: META-INF/.DS_Store (deflated 97%) adding: NAR-INF/ (stored 0%) adding: NAR-INF/.DS_Store (deflated 96%) adding: NAR-INF/bundled-dependencies/ (stored 0%) adding: TransformOpenskyStates.py (deflated 60%) It seems like it's zipping up the same files and the byte count for each .zip file are the same! Please see screenshot below! Solution: Order matters when zipping up the file! If you look above in Test Case 1, when I ran the zip command, it added TransformOpenskyStates.py first, while in Test Case 5 it added META-INF/ first.  I believe NiFi code is expecting META-INF/ first?? @MattWho - Do you know anything about this? ... View more

Hi SAMSAL,    Thanks for the recommendation. I tried just doing the python extension and it worked. However, when I compile the nar again it still doesn't show up.  Can you try it with this .py file I am pasting below? Thanks! Filename - TransformOpenskyStates.py from nifiapi.flowfiletransform import ( FlowFileTransform, FlowFileTransformResult ) from nifiapi.properties import ProcessContext import json FIELD_MAP = [ "icao24", "callsign", "origin_country", "time_position", "last_contact", "longitude", "latitude", "baro_altitude", "on_ground", "velocity", "true_track", "vertical_rate", "sensors", "geo_altitude", "squawk", "spi", "position_source" ] RETURN_SCHEMA = [ "icao24", "callsign", "origin_country", "reporting_time", "time_position", "last_contact", "longitude", "latitude", "on_ground" ] class TransformOpenskyStates(FlowFileTransform): class Java: implements = ['org.apache.nifi.python.processor.FlowFileTransform'] class ProcessorDetails: version = '0.0.1-SNAPSHOT' description = ''' Transform the data returned by the OpenSky Network API. ''' tags = ["opensky", "transform", "tutorial"] dependencies = [] def __init__(self, **kwargs): super().__init__() def transform( self, context: ProcessContext, flow_file ) -> FlowFileTransformResult: ''' Parameters: context (ProcessContext) flow_file Returns: FlowFileTransformResult ''' contents = json.loads(flow_file.getContentsAsBytes()) def sanitize_value(value): if isinstance(value, str): return value.strip() return value states = [] for record in contents["states"]: record = dict(zip(FIELD_MAP, record)) record["reporting_time"] = contents["time"] # Choose only fields listed in the RETURN_SCHEMA sanitized = {} for key, value in record.items(): if key not in RETURN_SCHEMA: continue sanitized[key] = sanitize_value(value) states.append(sanitized) return FlowFileTransformResult( "success", contents=json.dumps(states) )   ... View more

@SAMSAL  - Can you help out with this? ... View more

@RAGHUY    Thank you for your response. I tried manually resetting the policies and also resetting the cache for hbase but still getting the same thing. I am not making changes to any hbase policies because that is using a group for authorization. I am only making changes to ldap membership group in ldap. Then I rerun user sync and I see the user has been removed from that group in ranger admin UI. However I am still able to get acccess to the table the first time but the second time I can’t it works. The opposite happens too when I add a user to a group the first time it doesn’t work and the second time it works. I the HBase rest server logs I am seeing unauthorized and some issues with Kerberos thread unexpectedly exiting. Do you think that has something to do with it? ... View more

Maybe you could reset the state via NiFi-REST-API at beginning of your flow or separately on a schedule cron every morning Could be: POST "https://[ip:port]/nifi-api/processors/${processor-id}/state/clear-requests"   This is the request which NiFi itself uses when you go to the processor in the UI and choose the menu-option "view state" -> "clear state". ... View more