Greetings,
So I have been experimenting with Python extension (2.0.0 -M1) for sometime now and I have learned quite a lot since I posted the post above, and I think now I know how to resolve those issues which I will share with the community, but before doing so, I just wanted to kindly ask - if I may - the Nifi Leadership and the committers out there to do better job when documenting the different features and capabilities of Nifi specially when its newly released feature. They have done great job on some of the documentation like in the General Nifi User Guide , System Admin. user Guide, How to start with Nifi...etc. but when it comes to others like the "NiFi Python Developer’s Guide" (https://nifi.apache.org/documentation/nifi-2.0.0-M1/html/python-developer-guide.htm ) I find it extremely lacking and any one who is new to this and new to the Python to Java communication, would struggle quite a bit and spend a lot of time to figure out how things can be implemented based on what the guide is highlighting. I'm hoping this can reach someone who can address this concern @cjervis , @VidyaSargur , @pvillard.
Anyway, enough talking about boring documentations and lets get to code:
First, if you are planning to use Python Extension I recommend you get familiarized with the Java\Python communication specially the py4j library. it seems everything written in python is being passed\translated to Java which makes sense because Nifi is a java based application. The Python extension guide referenced above talks briefly about it but you can find more info here: https://www.py4j.org/
Second, as I mentioned in the above post that both issues seem to be related to the same cause since both issues produce the same error even though they are addressing totally different development problem. To summarize the issue, is when you are dealing with complex types like Python Collections as in the first case or external Classes like nifi Relationship:
Those complex types will get passed to the jvm through py4j , and if they dont adhere to the Java Gateway interface you will end up with this confusing error:
AttributeError: '...' object has no attribute '_get_object_id'
If you get familiar with how py4j works, you understand that in order to have the right class types everything has to go through\from java_gateway(). Either you convert your complex types to java compatible ones or you instantiate them through the gateway. In the first case with python dict type, I managed to resolve it using the java_gateway MapConverter() but it felt like Im doing some kind of hack copying some code from the python framework folder from the "controller.py" file in order to have access to the java_gateway from my python extension processor which looked like this:
java_port = int(os.getenv('JAVA_PORT'))
auth_token = os.getenv('AUTH_TOKEN')
gateway = JavaGateway(
callback_server_parameters=CallbackServerParameters(port=0, auth_token=auth_token),
gateway_parameters=GatewayParameters(port=java_port, read_timeout=None, enable_memory_management=True, auth_token=auth_token),
auto_convert=True)
....
However the second case I got stuck because I did not know how to convert or pass the nifiapi Relationship object. Well, after given up for days\weeks where there was no help even from the community, my stubbornness sometimes pays off 🙂 . It turns out buried somewhere in the same "Controller.py" the following line of code:
# Initialize the JvmHolder class with the gateway jvm.
# This must be done before executing the module to ensure that the nifiapi module
# is able to access the JvmHolder.jvm variable. This enables the nifiapi.properties.StandardValidators, etc. to be used
# However, we have to delay the import until this point, rather than adding it to the top of the ExtensionManager class
# because we need to ensure that we've fetched the appropriate dependencies for the pyenv environment for the extension point.
from nifiapi.__jvm__ import JvmHolder
JvmHolder.jvm = gateway.jvm
JvmHolder.gateway = gateway
Vowalla! I raise my hat to whoever wrote the comments above the code and I wish if they had access to slap those comments in the user guide as it would have saved me a lot of time.
It seems the nifiapi has static class called JvmHolder that has access to the java_gateway(). I know it feels silly but Im going to say: it felt like I found a buried treasure!
So that would have simplified the code above in two ways:
1- I could have used the gateway from the JvmHolder and do the conversion using MapConverter as follows:
from nifiapi.__jvm__ import JvmHolder
input_dict = {'name':record['name']}
mc_run_map_dict = MapConverter().convert(input_dict,JvmHolder.gateway._gateway_client)
2- Or even better , It seems when you have access to the java_gateway , you can instantiate your java compatible python dict using gateway.jvm.java.util.HashMap(). Thanks to : https://www.py4j.org/advanced_topics.html#map
from nifiapi.__jvm__ import JvmHolder
map = JvmHolder.gateway.jvm.java.util.HashMap()
map.put("name",record['name'])
return RecordTransformResult(schema=None,record=record,relationship="success",partition=map)
It seems we are making progress without having to do more treasure hunting, but what about the custom Relationship? The python developer guide promised us that we can define our own relationship by implementing the getRelationships(self) :
https://nifi.apache.org/documentation/nifi-2.0.0-M1/html/python-developer-guide.html#relationships
and simply by returning a list of "nifiapi.relationship.Relationship objects" it should magically works - or at least thats what I thought - but it did not! So I had to go on another treasure hunting and what do you know, buried in the "PythonProcessorAdapter.py" I found the following code:
self.relationships = gateway.jvm.java.util.HashSet()
success = gateway.jvm.org.apache.nifi.processor.Relationship.Builder() \
.name("success") \
.description("All FlowFiles will go to this relationship") \
.build()
self.relationships.add(success)
Vowalla again! That solved my second issue and I was able to define and return my own custom relationship to which I shouted "freeeeeeeeeeeeedoooooooooooooooooooooooooom".
I hope that helps someone out there and save them the headache, frustration and whatever hair left on their head.
If you know of any other buried treasures, please share and don't be stingy. By the way if you love treasure hunting, I still have unsolved issue here that I would love to find an answer for: https://community.cloudera.com/t5/Support-Questions/python-extension-generate-multiple-flowfiles-from-bytes/m-p/383095
I hope the community accepts that Im accepting my own solution.
Sorry if I kept it long or got quite emotional 😉 its just been quite the struggle.
Thank you all for reading and keep learning.
S
