Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Nifi CICD with mutiple flows having controller services

Highlighted

Nifi CICD with mutiple flows having controller services

New Contributor

@MattWho 

Hi,

I am looking for better suggestions,How CICD configured or performed for real time Nifi projects.

 

In my project we are having 3 different individual flows mostly uses same controller services.

What is the best way to have controller services Globally or local for each flow?

  • If controller services are configured globally ,what is the best way of mapping while deploying flow in another environment.
  • Pro and cons of having controller services globally / locally.

Thanks in advance.

 

 

 

 

3 REPLIES 3
Highlighted

Re: Nifi CICD with mutiple flows having controller services

Master Guru

@anil35759 

 

What is the best way to have controller services Globally or local for each flow?
- Controller Services (CS) are setup at the Process Group (PG) level.  Sub process groups will have access (potentially depending on granularity of authorizations that have been setup) to CSs created at parent process group level.  So if you want to isolate what CS a particular group/team has access to you should create a different PG for each group/team in which they will create their dataflows.  Those dataflows will only hav access to the CSs created within their PG.  This prevents each team from messing with the configuration of a CS that may affect another team's dataflow.

  • If controller services are configured globally ,what is the best way of mapping while deploying flow in another environment.  
    --- Not sure how you are currently "deploying" your flows, so this is a bit hard to answer.
    --- Generally speaking the deployment of flows from one NiFi environment to another is best handled via the NiFi-Registry service that is shared amongst all environments.  A dataflow is build within a PG and then that PG is version controlled to the NiFi-Registry (all referenced CS within the PG are included in that version controlled flow within NiFi-Registry).  Then within another NiFi environment, that flow can be imported from the same NiFi-Registry.  The added advantage to this deployment model is that new versions of that PG can be published to the NiFi-Registry and the other NiFi instances using that same registry will be made aware that a newer version is available and user can take action to update to the newer available version.

Pro and cons of having controller services globally / locally.
--- When you say "Globally", I assume you mean creating the CSs at the root PG level so that they are accessible by all sub PGs added within the UI?  Disadvantages here:
1. One team may make a change to the CS that they do not realize will affect another team's dataflow. 
2. Making a change to a CS requires that the CS is disabled.  If team one disables the CS to edit it, any referenced component of the CS is also stopped.  After editing the team may enable the CS only and not restart the other teams referenced components.
3. If granular authorizations are setup user in one team may not be able to successfully edit a CS that is being actively used by a component in another teams PG that they do not have authorization to stop.

So all the above really revolves around access controls and not much to do with the actual execution of your dataflows.  If you have one team that manages all your dataflows, I see no downside to having a single shared global CS except when you deploy each flow separately.  as part if deployment of a dataflow (how ever you do it), components will need the referenced CSs.  So if you may end up during deployment with multiple copies of the same CS with each separate dataflow deployment.

 

 

Hope this helps,

Matt

Re: Nifi CICD with mutiple flows having controller services

New Contributor

@MattWho 

There is only single team uses all the flows.

--We are using Nifi toolkit api with Nifi registry for deploying flows from one environment to another.

 

Currently we are using CSs globally(CSs at the root PG level) and those are accessible for all the flows.

No we end up with a scenario how well CSs can be managed and mapped while deploying flow independently .

 

Thanks,

Anil

 

 

 

 

 

Highlighted

Re: Nifi CICD with mutiple flows having controller services

Master Guru

@anil35759 

Your observations are exactly how Nifi is designed to work when importing version comtrolled flows from NiFi-Registry.

 

The first time you "deploy" a flow from NiFi-Registry it will add all the Controller Services (CS) utilized by the newly imported flow's components.  Subsequent version changes will not result in same behavior unless there is a new CS as part of the new version.

So if you then deploy another version controlled flow that uses the same CS that the other already imported flow uses, you will end up with a second set of CS in the target NiFi.  There is not cross dependency on separately version controlled flows in NiFi-Registry. So each version controlled Process Group (PG) in NiFi will contain the complete set of referenced CS.  Post import of a CS to target NiFi, you may make changes to the CSs imported.  It would be dangerous for NiFi to assume that when importing another or the sane flow again from NiFi-Registry that it should reuse existing CS within the target NiFi. 

Matt

Don't have an account?
Coming from Hortonworks? Activate your account here