Created on 07-24-201809:02 PM - edited on 10-22-201903:21 AM by VidyaSargur
It's become enough of an ask that I had to post to answer it. The ask is something like this, "What's with all this UI stuff, I want devops, automation, command line et al." So did I in 2002. It's nice to see everything and have a nice diagram on a website without any extra tools or ssh. Okay that didn't convince anyone. So here is a proper DevOps solution for you.
Once you unzip it, you can run one of two ways. With no parameters and you will bring up an interactive console.
Now you an type help to see a nice list of commands. I think of this like the Spark Shell or Apache Zeppelin where you can experiment, find out what you want then you can use that single command with your automation suite. The toolkit lets you automate a number of actions in Apache NiFi and it's registry.
# Name Id Description- ------------------ ------------------------------------ -------------------------1 NiFi 1.7 Features 54b37ad8-274b-4d9d-a09c-0ee2816f271c NiFi 1.72 Rainbow Processing 5ebc2183-954e-4887-a28c-9d0ee54a02ed server rainbow processing
You can run this from the interactive command line or as a one-off command. You would have to capture the list of buckets from the list, use it to get flows and then use the list of flows to get versions. This could easily be in a for-loop in shell, Python, Go or automation scripting tool of your choice. I would probably do this in NiFi.
Transfers a version of a flow directly from one flow to another, without needing
to export/import. If --sourceProps is not specified, the source flow is assumed
to be in the same registry as the destination flow. If --sourceFlowVersion is
not specified, then the latest version will be transferred.
-f,--flowIdentifier <arg> A flow identifier
-kp,--keyPasswd <arg> The key password of the keystore being used
-ks,--keystore <arg> A keystore to use for TLS/SSL connections
-ksp,--keystorePasswd <arg> The password of the keystore being used
-kst,--keystoreType <arg> The type of key store being used (JKS or
-ot,--outputType <arg> The type of output to produce (json or
-p,--properties <arg> A properties file to load arguments from,
command line values will override anything
in the properties file, must contain full
path to file
-pe,--proxiedEntity <arg> The identity of an entity to proxy
-sf,--sourceFlowIdentifier <arg> A flow identifier from the source registry
-sfv,--sourceFlowVersion <arg> A version of a flow from the source registry
-sp,--sourceProps <arg> A properties file to load for the source
-ts,--truststore <arg> A truststore to use for TLS/SSL connections
-tsp,--truststorePasswd <arg> The password of the truststore being used
-tst,--truststoreType <arg> The type of trust store being used (JKS or
-u,--baseUrl <arg> The URL to execute the command against
-verbose,--verbose Indicates that verbose output should be
An Example List of My Local Apache NiFi Flows
NIFI TOOLKIT Flow Analyzer
To run this with my massive amount of flows, I edited the flow-analyze.sh and upped java memory to below:
The rest of this article is a big command line dump, seems a huge text list is the way to go:
Total Bytes Utilized by System=519 GB
Max Back Pressure Size=1 GB
Min Back Pressure Size=1 GB
Average Back Pressure Size=0.990458015 GB
Max Flowfile Queue Size=10000
Min Flowfile Queue Size=10000
Avg Flowfile Queue Size=9904.580152672