Created 11-21-2018 08:50 AM
Hi, I have a processgroup which acts as a subroutine for many other processgroups. I started version control and imported this version into the other processgroups. Now I made changes which lead to a new version which should be substituted in all other processgroups.
Found in "NiFi Summary" -> "PROCESS GROUPS" the possibility to find all affected processgroups.
The question is: Is there an other possibility for substitution except going from here to every single processgroup for "Change version"?
Using NiFi 1.6.0
Thanks for any help!
Created 06-04-2019 06:43 AM
Oh, Is it your NiFi-Registry using HTTPS instead of HTTP? I'm going to try nipyapi on HTTPS too 🙂
Created on 06-04-2019 12:33 PM - edited 08-17-2019 04:30 PM
Hi @ Justen
Check this out.
import nipyapi nipyapi.config.registry_config.host = 'https://localhost:18443/nifi-registry-api' # additional code for SSL(HTTPS) nipyapi.security.set_service_ssl_context(service='registry', ca_file='G:/TEMP/ssl/localhost.cer', client_cert_file='G:/TEMP/ssl/sys_admin.cer', client_key_file='G:/TEMP/ssl/sys_admin.key', client_key_password=None)
And it should works like this
So, we should know somethings like:
1. HTTPS: HTTP + SSL(TLS is deprecated)
2. SSL: Know about Certificate. https://www.websecurity.symantec.com/security-topics/what-is-ssl-tls-https
3. asymmetric public-private key cryptosystem: common use case is `ssh-keygen`. https://en.wikipedia.org/wiki/RSA_(cryptosystem)
4. CA(certificate authority): Everyone can create a Certificate but whose Certificate is safe or 'legal'? So CA is here. https://www.google.com/search?q=ca+certificate
5. (optional) To setup a nifi-registry in HTTPS: https://www.youtube.com/watch?v=qD03ao3R-a4
To learn that, just google it.
In HTTPS-nifi-registry case, use this method to add certificate to access HTTPS. https://nipyapi.readthedocs.io/en/latest/nipyapi-docs/nipyapi.html#module-nipyapi.security
I setup a nifi-registry followed by the youtube video above. Use tools - https://keystore-explorer.org/ - and the certificates should look like this
Parameters:
1. client_cert_file: the certificate use to access nifi-registry, like 'username'
2. client_key_file: a private key for `client_cert_file`, 'password'
3. client_key_password: the password for private key - `client_key_file`, private key can be encrypted by a password or not be encrypted. In this case, the private key isn't encrypted so set to None.
4. ca_file: The CA who sign the `client_cert_file`. In this case, the CA file is created by ourselves too which is the 'localhost' certificate.
You can have all those files by asking your admin guy.
I hope I explain clearly 🙂
Created 06-19-2019 06:48 AM
Thanks for this detailed and informative answer! So my admin solved the SSL-problem among others by installing Python3.6.
Now I come until this error:
ExecuteStreamCommand[id=e0bc9898-7dfb-3992-9e35-46fd7e983658] Transferring flow file StandardFlowFileRecord[uuid=f680a1c1-5ba9-426e-8b4f-a1d89e1a3f30,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1560791493717-1, container=default, section=1], offset=646526, length=-1],offset=0,name=tmp_nifi_update_flow_ver.py,size=0] to nonzero status. Executable command /usr/bin/python3.6 ended in an error: Traceback (most recent call last): File "tmp_nifi_update_flow_ver.py", line 56, in <module> update_flow_ver(pg_id, bucket_name, flow_name, target_version) File "tmp_nifi_update_flow_ver.py", line 42, in update_flow_ver flow = nipyapi.versioning.get_flow_in_bucket(bucket_id=bucket.identifier, identifier=flow_name, identifier_type='name') AttributeError: 'NoneType' object has no attribute 'identifier'
Can you give me some advice on this?
Created 06-19-2019 07:10 AM
bucket = nipyapi.versioning.get_registry_bucket(identifier=bucket_name, identifier_type='name') # from: I should add some verification here if bucket is None: print(f"ERROR: bucket is None which means {bucket_name} not exists") sys.exit(-1) # end flow = nipyapi.versioning.get_flow_in_bucket(bucket_id=bucket.identifier, identifier=flow_name, identifier_type='name')
Nothing fatal, just check the "bucket" whether it exists or not.
Created 06-19-2019 08:38 AM
@Archon Gum
I'm sorry, but the bucket definitely exists!
Your solution came so close to my target... Do you have another idea?
By the way. If I add your supplement this error occurs:
ExecuteStreamCommand[id=e0bc9898-7dfb-3992-9e35-46fd7e983658] Transferring flow file StandardFlowFileRecord[uuid=6a70ebcc-13a0-431b-aa85-014bf8fb825f,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1560791493717-1, container=default, section=1], offset=715298, length=59],offset=0,name=tmp_nifi_update_flow_ver.py,size=59] to nonzero status. Executable command /usr/bin/python3.6 ended in an error: Traceback (most recent call last): File "tmp_nifi_update_flow_ver.py", line 61, in <module> update_flow_ver(pg_id, bucket_name, flow_name, target_version) File "tmp_nifi_update_flow_ver.py", line 45, in update_flow_ver sys.exit(-1) NameError: name 'sys' is not defined
Created 06-19-2019 08:45 AM
import sys
import the 'sys' module
Created on 06-19-2019 08:59 AM - edited 08-17-2019 04:30 PM
Created 07-16-2019 07:51 AM
Hi @Archon Gum , I know I'm very persistent but I see no other way than aksing you for help again.
Ran the script from shell and received the message you added:
python3.6 tmp_nifi_update_flow_ver.py --pg_id f5163735-016b-1000-0000-00002a32b939 --bucket_name Justen_Bucket --flow_name TEST_IJ_PG_1 --target_version latest ERROR: bucket is None which means Justen_Bucket not exists
Even when I use another bucket this message appears.
It seems the nipyapi.versioning.get_registry_bucket(identifier=bucket_name, identifier_type='name') doesn't return the real buckets in registry.
Is it possible to test this in another way? Could it be an authorization problem again?
Sorry, but I see no way to solve this problem myself.
It would be a pity if all your efforts were in vain... Thank you!
Created 07-17-2019 06:05 AM
@ Justen Do you use slack? Leave your email address and I invite you to my channel, we discuss there.
Created 07-17-2019 07:42 AM
@Archon Gum I'm glad you answered!
Yes, I'm using slack - since about 5 minutes... ;-))
Here's my email address which I will remove in about 2 hours: removed
I'm looking forward to receive an appointment for our discussion because sometimes I'm not available, sorry!