Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

​NiFi / NiFi-Registry - Best approach for "Change version" of a processgroup used by many other processgroups

​NiFi / NiFi-Registry - Best approach for "Change version" of a processgroup used by many other processgroups

Rising Star

Hi, I have a processgroup which acts as a subroutine for many other processgroups. I started version control and imported this version into the other processgroups. Now I made changes which lead to a new version which should be substituted in all other processgroups.

Found in "NiFi Summary" -> "PROCESS GROUPS" the possibility to find all affected processgroups.

The question is: Is there an other possibility for substitution except going from here to every single processgroup for "Change version"?

Using NiFi 1.6.0

Thanks for any help!

18 REPLIES 18
Highlighted

Re: ​NiFi / NiFi-Registry - Best approach for "Change version" of a processgroup used by many other processgroups

Explorer

It seems no such function yet. I currently move all my stuff in a 'root' group and version control it. For example, let's assume there are two environments - dev and prod. And I just import the same version-controlled 'root' group twice and named them 'dev' and 'prod'. When 'root' group version changed, all processes and groups inside - including version-controlled groups - changed too.108584-1.jpg

dev has 2 version-controlled groups inside and prod has 1 currently


108621-2.jpg


change prod version to 7 to sync

108612-3.jpg

Highlighted

Re: ​NiFi / NiFi-Registry - Best approach for "Change version" of a processgroup used by many other processgroups

Explorer

https://raw.githubusercontent.com/archongum/NiFi-templates/master/VersioningUpdateFlowVer.xml

Apply "change version" of all process groups which share the same version-control flow(bucket/flow)

Some input attributes:

  1. process group id, a parent group which contains all process groups which share the same version-control flow. Set to 'root' to loop all process groups but cost more time.
  2. bucket name
  3. flow name
  4. target version

This template based on https://github.com/Chaffelson/nipyapi

Highlighted

Re: ​NiFi / NiFi-Registry - Best approach for "Change version" of a processgroup used by many other processgroups

Rising Star

@Archon Gum

Hi, thanks for this awsome information! Something like this I was looking for but could not find...

Did you develope this solution?


Tested it on my local installation (1.9.1) and it worked perfect.

Now I have to prepare my offical DEV-Sytem with python and test it there (Cluster with 1.9.2) with more extensive flows.


I will let you know here whether the test was successfully.

Bye!

Highlighted

Re: ​NiFi / NiFi-Registry - Best approach for "Change version" of a processgroup used by many other processgroups

Explorer

Glad it works. I just wrote the nifi template and the python script inside it. Thanks to the `nipyapi` module.

Highlighted

Re: ​NiFi / NiFi-Registry - Best approach for "Change version" of a processgroup used by many other processgroups

Rising Star

@Archon Gum

Hi, I was told Python 2.7.5 is already available on my DEV. Installed nipyapi and changed CommandPath in ExecuteStreamCommand. On execution this error happens:

108659-1557900281889.png
Question: Is it absolutely necessary having Python3.x or can you see another cause for this exception?

Thanks for your help and this solution!

Highlighted

Re: ​NiFi / NiFi-Registry - Best approach for "Change version" of a processgroup used by many other processgroups

Rising Star

Hi @Archon Gum
do you mind if I ask you one more time for your help?

I compared the bucket.py in Python37 (local) and Python275 (DEV-System with Cluster) but there is no difference.


Maybe you could give me a hint to solve this problem? Thanks!

Highlighted

Re: ​NiFi / NiFi-Registry - Best approach for "Change version" of a processgroup used by many other processgroups

Explorer

Hi

I tried but I can't reproduce it:

# 1. set identifier to None
nipyapi.versioning.get_registry_bucket(identifier=None, identifier_type='name')
# return: TypeError: 'in <string>' requires string as left operand, not NoneType

# 2. empty all bucket in nifi-registry and try to get bucket
nipyapi.versioning.get_registry_bucket(identifier='test', identifier_type='name')
# return: None

I can't reach the code in quote:

1. If nifi-registry has buckets, each bucket's name will set to this 'name' parameter and can not be None at all.

2. If nifi-registry is empty(no bucket), the code won't be executed (because return a empty list)

109074-1559183972253.png


My suggestions:

1. check `nipyapi.config.registry_config.host = 'http://your-server:18080/nifi-registry-api'`

2. try to use a new nifi-registry and run the code again to find out whether the error occurs. (I recommand using docker: https://hub.docker.com/r/apache/nifi-registry )

3. it seems python version is not the problem here.

Highlighted

Re: ​NiFi / NiFi-Registry - Best approach for "Change version" of a processgroup used by many other processgroups

Rising Star

Hi @Archon Gum

thank you so much for your answer and your efforts!


Refering to your suggestions:

1. I checked the information and it is correct. BUT when I test to acces the REST-API of registry in an InvokeHTTP-Processor I have to type https.

So I changed this in "nipyapi.config.registry_config.host" and ended up with an certification failure. Here a part of it:


AttributeError: 'module' object has no attribute 'X509_up_ref'
2019-06-04 05:40:24,709 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),)': /nifi-registry-api/buckets
2019-06-04 05:40:24,709 WARNING Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),)': /nifi-registry-api/buckets
WARNING:urllib3.connectionpool:Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),)': /nifi-registry-api/buckets
From cffi callback <function _verify_callback at 0x7f3a008085f0>


So I have to speak with the admin.


2. Because I'm not the "master of the installation" I can't do this just like that. But I will bear this in mind.


3. I agree.


If I get this work I will let you know. So long!

Highlighted

Re: ​NiFi / NiFi-Registry - Best approach for "Change version" of a processgroup used by many other processgroups

Explorer

Oh, Is it your NiFi-Registry using HTTPS instead of HTTP? I'm going to try nipyapi on HTTPS too :)

Don't have an account?
Coming from Hortonworks? Activate your account here