@MattWho  Wow! I think this pattern would work best for my usecase. I hadn't even considered the first challenge you brought up of production flows having their Parameter Context unassigned if I were to update their version. That would've been painful to find out after deploying many instances.   Back in NiFi 1 I used to handle situations such as this with variables, since they could just be directly attached to Process Groups and so I never had to worry about creating separate objects (parameters) and ensuring they get attached, or that every new instance of a versioned flow had to have its own unique context created. It's been a couple years but I believe I even questioned Pierre about this in one of his appearances in the Israeli NiFi meet-ups.   In regards to product work, I've ran into this case of trying to use NiFi as the underlying tool for different SaaS platforms multiple times already. There could definitely be some QoL changes made to make such a use-case easier to implement with NiFi's flow registry, I guess the responsibility lies in people like me opening issues to bring them though 🙂   Thank you very much for the suggestions Matt! Green ... View more

@MattWho  So just to verify I understand everything: 1) I start with a dev process group somewhere in my canvas (let us say it is named my-flow), with a parameter context attached to it, say PC-DEV, and then I commit this PG to the registry so I will be able to deploy thousands of unique production copies of it  2) When I deploy my-flow from the registry in the same nifi environment, the newly created process group will automatically have PC-DEV attached to it (<-- this is what I'm worried about) 3) In order to uniquely set up this flow, I now need to create a new PC which mimics PC-DEV so that I will be able to configure a unique set of parameters for this deployed instance   Currently testing on nifi 2.7.2, I actually see that there's a toggle when importing from registry to "Keep existing Parameter Contexts", which partially answers my concern. If it is toggled ON, then the new deployment will automatically use an existing PC. If toggled OFF, it will instead automatically create a new copy of the commit-time PC (without sensitive values) which will be named "<original pc name> (1)" with the number increasing per copy. The OFF behaviour is more in-line with what I wish to do in my environment, though the naming is still problematic as I'd prefer the PC name to more directly correlate to the deployed instance (when I'll deploy instances via my code, I'll probably add the instance ID to the PC name)   Your 2nd note has raised some concerns for me. Did I understand correctly that if I commit a new version for a flow and in it I added a new paramter somewhere in the procsesors, then automatically when I update the other flows that new parameter will be added to all of their respective parameter contexts? (E.G. I update my dev flow with a new parameter, all production copies of the dev parameter context will now have a new parameter added, though with the value being the same as the dev value I commited)   Frankly I still feel as though my best course of action is to simply commit the flows without any PC attached but with parameter strings (#{..}) pre-configured. This way when I deploy a flow with my code via the REST api, I can more easily tailor a new parameter context to the new flow. This way there are also no concerns of accidentally attaching default/dev values to production flows. Though if I add new parameters, I will need to manually add them to all the existing PCs.   I'm not sure there's any one clear answer to my question so I'll just mark yours as the solution. Thanks Matt 🙂 ... View more

Hi @MattWho , thank you for the detailed reply 🙂 I might not have been very clear in my original question so I'll rephrase with more context- My use-case is not deploying a single instance of a flow from NiFi A to NiFi B, rather I've got just one NiFi instance and I'd like to deploy thousands of instances of the same versioned flow (and perhaps in the future, I'll have more clusters in different regions and would like to deploy the flows there too).   In this context, it is problematic for me that Parameter Contexts get created/picked based on the name of the original PC at commit time, since I want each flow instance to have its own unique values for the parameters.   Therefore, my current plan is commiting the flows without a PC attached (though with parameters configured in the processors), and at creation time (via my codebase) creating a new PC (uniquely named per flow) with the specific values for that instance.   My question was whether there is a more preferable way to do this - creating many instances of a parameterized versioned flow in the same nifi environment and ensuring each instance can get its own unique set of parameters.   Thanks for helping out, Green ... View more

I am not clear on what kind of example I could share with you. 403 seems to imply the user you got the token for does not have permissions to run the operation. You should verify if your user has the correct permissions, specifically "Operate the component" permissions, though I believe these are included if the user has modify permissions.   ... View more

So just to verify, in the UI you are able to start/stop process groups? If your user is able to do so by clicking on the PG, it should be able to do so with the REST API ... View more

So GET requests work fine for you but POSTs fail? Perhaps your user does not have write permissions? I recommend opening devtools and looking at the Network tab to see what requests get sent when you do certain operations (starting/stopping a process group) in the UI. You should be able to replicate them with the REST API if you can do them manually in the UI. ... View more

Hi @MuruganFinastra , Are you sure you are attaching your token correctly to your requests? You should be using an Authorization header with a value of "Bearer <token>" for all requests once you fetch the token. ... View more

Hey @eylon , did you ever find a solution to this problem? We are facing the exact same issue right now where we'd like to write to elastic without records due to a changing schema, yet our files contain batches of jsons and so we can't use PutElasticsearchJson as we'd like. ... View more