@saquibsk, It seems you are using Literal Value  in the Replacement Value Strategy of the second UpdateRecord. I think it needs to be "Record Path Value" since you are referencing another field from the content data itself.   If that helps please accept solution. Thanks ... View more

hi @piyush7829 , Can you provide more info on what the issue is? Are you getting error messages in the failure , retry relationships? Are you not getting any response at all? Are you sure that your api takes the data as request body and not as post form data key value as the processor description indicates : Also I dont see that you are setting the Request Content-Type property to: multipart/form-data Here are some posts that might help get it working: https://community.cloudera.com/t5/Support-Questions/How-to-send-an-http-POST-multipart-form-data-request-with-a/m-p/240035   https://palindromicity.blogspot.com/2020/04/sending-multipart-form-data-with.html   https://stackoverflow.com/questions/51929208/how-to-send-multipart-form-data-via-invokehttp-nifi   If you find this is helpful please accept solution. Thanks ... View more

Greetings, So I have been experimenting with Python extension (2.0.0 -M1) for sometime now and I have learned quite a lot since I posted the post above,  and I think now I know how to resolve those issues which I will share with the community, but before doing so, I just wanted to kindly ask - if I may -  the Nifi Leadership and the committers out there  to do better job when documenting the different features and capabilities of Nifi specially when its newly released feature. They have done great job on some of the documentation like in the General Nifi User Guide , System Admin. user Guide, How to start with Nifi...etc. but when it comes to others like the "NiFi Python Developer’s Guide" (https://nifi.apache.org/documentation/nifi-2.0.0-M1/html/python-developer-guide.htm ) I find it extremely lacking and any one who is new to this and new to the Python to Java communication, would struggle quite a bit and spend a lot of time to figure out how things can be implemented based on what the guide is highlighting. I'm hoping this can reach someone who can address this concern  @cjervis , @VidyaSargur , @pvillard. Anyway,  enough talking about boring documentations and lets get to code: First, if you are planning to use Python Extension I recommend you get familiarized with the Java\Python communication specially the py4j library. it seems everything written in python is being passed\translated to Java which makes sense because Nifi is a java based application.  The Python extension guide referenced above talks briefly about it but you can find more info here: https://www.py4j.org/ Second, as I mentioned in the above post that both issues seem to be related to the same cause since both issues produce the same error even though they are addressing totally different development problem. To summarize the issue, is when you are dealing with complex types like Python Collections  as in the first case or external Classes like nifi Relationship: nifiapi.relationship Those complex types will get passed to the jvm through py4j , and if they dont adhere to the Java Gateway interface you will end up with this confusing error:     AttributeError: '...' object has no attribute '_get_object_id'     If you get familiar with how py4j works, you understand that in order to have the right class types everything has to go through\from java_gateway(). Either you convert your complex types to java compatible ones or you instantiate them through the gateway. In the first case with python dict type, I managed to resolve it using the java_gateway MapConverter() but it felt like Im doing some kind of hack copying some code from the python framework folder from the "controller.py" file in order to have access to the  java_gateway from my python extension processor which looked like this:     java_port = int(os.getenv('JAVA_PORT')) auth_token = os.getenv('AUTH_TOKEN') gateway = JavaGateway( callback_server_parameters=CallbackServerParameters(port=0, auth_token=auth_token), gateway_parameters=GatewayParameters(port=java_port, read_timeout=None, enable_memory_management=True, auth_token=auth_token), auto_convert=True) ....     However the second case I got stuck because I did not know how to convert or pass the nifiapi Relationship object. Well, after given up for days\weeks where there was no help even from the community, my stubbornness sometimes pays off 🙂 . It turns out buried somewhere in the same "Controller.py" the following line of code:     # Initialize the JvmHolder class with the gateway jvm. # This must be done before executing the module to ensure that the nifiapi module # is able to access the JvmHolder.jvm variable. This enables the nifiapi.properties.StandardValidators, etc. to be used # However, we have to delay the import until this point, rather than adding it to the top of the ExtensionManager class # because we need to ensure that we've fetched the appropriate dependencies for the pyenv environment for the extension point. from nifiapi.__jvm__ import JvmHolder JvmHolder.jvm = gateway.jvm JvmHolder.gateway = gateway     Vowalla! I raise my hat to whoever wrote the comments above the code and I wish if they had access to slap those comments in the user guide as it would have saved me a lot of time. It seems the nifiapi has static class called JvmHolder that has access to the java_gateway(). I know it feels silly but Im going to say: it felt like I found a buried treasure! So that would have simplified the code above in two ways: 1- I could have used the gateway from the JvmHolder and do the conversion using MapConverter as follows:     from nifiapi.__jvm__ import JvmHolder input_dict = {'name':record['name']} mc_run_map_dict = MapConverter().convert(input_dict,JvmHolder.gateway._gateway_client)     2- Or even better , It seems when you have access to the java_gateway , you can instantiate your java compatible python dict using gateway.jvm.java.util.HashMap(). Thanks to : https://www.py4j.org/advanced_topics.html#map     from nifiapi.__jvm__ import JvmHolder map = JvmHolder.gateway.jvm.java.util.HashMap() map.put("name",record['name']) return RecordTransformResult(schema=None,record=record,relationship="success",partition=map)     It seems we are making progress without having to do more treasure hunting, but what about the custom Relationship? The python developer guide promised us that we can define our own relationship by implementing the getRelationships(self) : https://nifi.apache.org/documentation/nifi-2.0.0-M1/html/python-developer-guide.html#relationships and simply by returning a list of "nifiapi.relationship.Relationship objects" it should magically works - or at least thats what I thought - but it did not! So I had to go on another treasure hunting and what do you know, buried in the "PythonProcessorAdapter.py"  I found the following code:     self.relationships = gateway.jvm.java.util.HashSet() success = gateway.jvm.org.apache.nifi.processor.Relationship.Builder() \ .name("success") \ .description("All FlowFiles will go to this relationship") \ .build() self.relationships.add(success)     Vowalla again! That solved my second issue and I was able to define and return my own custom relationship to which I shouted "freeeeeeeeeeeeedoooooooooooooooooooooooooom". I hope that helps someone out there and save them the headache, frustration and whatever hair left on their head. If you know of any other buried treasures, please share and don't be stingy. By the way if you love treasure hunting, I still have unsolved issue here that  I would love to find an answer for: https://community.cloudera.com/t5/Support-Questions/python-extension-generate-multiple-flowfiles-from-bytes/m-p/383095 I hope the community accepts that Im accepting my own solution. Sorry if I kept it long or got quite emotional 😉 its just been quite the struggle. Thank you all for reading and keep learning. S ... View more

Hi, It depends on how you are setting the UpdateRecord processor. If you can provide screenshot it will help us figure out the issue. It seems you are trying to overwrite the field SPRINT_LIST and its complaining that you cannot convert Array to String because its already an array in the first record. May I suggest a different approach because the UpdateRecord you have to worry about setting the Record Writer\Reader and also you have to define the avro schema accordingly if you are planning to add new fields. Instead you can just use the Json Jolt Transformation processor with the following spec:   [ { "operation": "shift", "spec": { "*": { "SPRINT_LIST": { "0": { "#true": "[&3].is_sprint_array" }, "@": "[&2].SPRINT_LIST" }, "*": "[&1].&" } } }, { "operation": "default", "spec": { "*": { "is_sprint_array": "false" } } } ] No need to set record reader\writer as long you know the input is always its valid json and has same schema. This will give you the following output:   [ { "INTEGRATION_ID" : "1", "ISSUE_STATUS_CAT" : "Done", "SPRINT_LIST" : [ "24-w1-2", "24-w1-3", "24-w1-4" ], "is_sprint_array" : "true" }, { "INTEGRATION_ID" : "2", "ISSUE_STATUS_CAT" : "Done", "SPRINT_LIST" : "24-w1-2", "is_sprint_array" : "false" } ]   If that helps please accept solution. Thanks ... View more