Community Articles

Find and share helpful community-sourced technical articles.
Announcements
Celebrating as our community reaches 100,000 members! Thank you!
Labels (1)
avatar
Master Guru

HDF DevOps

It's become enough of an ask that I had to post to answer it. The ask is something like this, "What's with all this UI stuff, I want devops, automation, command line et al." So did I in 2002. It's nice to see everything and have a nice diagram on a website without any extra tools or ssh. Okay that didn't convince anyone. So here is a proper DevOps solution for you.

Option 1: REST

The full documentation for the NiFi REST API is here: https://nifi.apache.org/docs/nifi-docs/rest-api/

The follow is some examples I have accessed via CURL (if you have security, you will need to account for that, see the specifications.

curl http://hw13125.local:8080/nifi-api/resources

curl http://hw13125.local:8080/nifi-api/tenants/user-groups

curl http://hw13125.local:8080/nifi-api/tenants/users

curl http://hw13125.local:8080/nifi-api/flow/about

curl http://hw13125.local:8080/nifi-api/flow/banners

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/bulletin-board

curl http://hw13125.local:8080/nifi-api/flow/cluster/summary

curl http://hw13125.local:8080/nifi-api/flow/config

{"flowConfiguration":{"supportsManagedAuthorizer":false,"supportsConfigurableAuthorizer":false,"supportsConfigurableUsersAndGroups":false,"autoRefreshIntervalSeconds":30,"currentTime":"16:12:51 EDT","timeOffset":-14400000,"defaultBackPressureObjectThreshold":10000,"defaultBackPressureDataSizeThreshold":"1 GB"}}%

curl http://hw13125.local:8080/nifi-api/flow/controller/bulletins

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/history\?offset\=1\&count\=100

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/prioritizers

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/processor-types

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/registries

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/reporting-tasks

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/search-results\?\q\=mxnet

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/status

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/templates

curl http://hw13125.local:8080/nifi-api/system-diagnostics

curl http://hw13125.local:8080/nifi-api/flow/controller/bulletins

curl http://hw13125.local:8080/nifi-api/flow/status

curl http://hw13125.local:8080/nifi-api/flow/cluster/summary

curl http://hw13125.local:8080/nifi-api/site-to-site

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/process-groups/root

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/process-groups/root/controller-services

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/process-groups/root/status

curl http://princeton1.field.hortonworks.com:8080/nifi-api/flow/process-groups/7a01d441-0164-1000-ec7a-54...

Option 2: Python: http://nipyapi.readthedocs.io/en/latest/readme.html Now in version 0.91.

This library is awesome, very easy to use and I love it. See here for a deep dive: https://community.hortonworks.com/articles/177301/big-data-devops-apache-nifi-flow-versioning-and-au...

Option 3: Forget about it, just use Ambari, NiFi, Cloudbreak and DPS. Problem solved. WebGUIs are killer.

Option 4: The New NiFi Toolkit CLI

Let's examine the New NiFi CLI. I am using the version for Apache NiFi 1.7.

To install the CLI, you need to download Apache NiFi Toolkit (https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli)

(https://www.apache.org/dyn/closer.lua?path=/nifi/1.7.1/nifi-toolkit-1.7.1-bin.zip)

Once you unzip it, you can run one of two ways. With no parameters and you will bring up an interactive console.

83400-nificlistart.png

Now you an type help to see a nice list of commands. I think of this like the Spark Shell or Apache Zeppelin where you can experiment, find out what you want then you can use that single command with your automation suite. The toolkit lets you automate a number of actions in Apache NiFi and it's registry.

Below are a number of non-interactive commands:

./bin/cli.sh nifi pg-list -u http://hw13125.local:8080 -ot json

./bin/cli.sh registry list-buckets -u http://localhost:18080

./bin/cli.sh nifi pg-status -u http://hw13125.local:8080 --processGroupId f10700ba-3d5e-30a8-ea5d-33c59771d4f1

./bin/cli.sh nifi pg-get-services -u http://hw13125.local:8080 --processGroupId f10700ba-3d5e-30a8-ea5d-33c59771d4f1

./bin/cli.sh registry list-flows -bucketId 36cb79a4-f735-4f77-ba55-606718a9c3c9 -u http://localhost:18080

./bin/cli.sh registry list-buckets -u http://princeton1.field.hortonworks.com:18080/

./bin/cli.sh registry list-flows -u http://princeton1.field.hortonworks.com:18080/ -bucketIdentifier 36cb79a4-f735-4f77-ba55-606718a9c3c9

#  Name  Id                     Description-  ------------------  ------------------------------------  -------------------------1  NiFi 1.7 Features  54b37ad8-274b-4d9d-a09c-0ee2816f271c  NiFi 1.72  Rainbow Processing  5ebc2183-954e-4887-a28c-9d0ee54a02ed  server rainbow processing

./bin/cli.sh registry export-flow-version -u http://princeton1.field.hortonworks.com:18080/ -f 5ebc2183-954e-4887-a28c-9d0ee54a02ed -o rainbow.json -ot json

 

How to Backup Registry

You can run this from the interactive command line or as a one-off command. You would have to capture the list of buckets from the list, use it to get flows and then use the list of flows to get versions. This could easily be in a for-loop in shell, Python, Go or automation scripting tool of your choice. I would probably do this in NiFi.

registry list-buckets -u http://localhost:18080

registry list-flows -u http://localhost:18080 -b 36cb79a4-f735-4f77-ba55-606718a9c3c9

registry export-flow-version -f 5ebc2183-954e-4887-a28c-9d0ee54a02ed -o rainbow.json -ot json

 

List What’s Running


nifi pg-list -u http://princeton1.field.hortonworks.com:8080

You will get a list of all the Processor Groups.

An Example Processor Group List from HDF NiFi Server in the Cloud

83403-nificli-pglist.png

List of Commands

commands:

 demo quick-import
 nifi current-user
 nifi get-root-id
 nifi list-reg-clients
 nifi create-reg-client
 nifi update-reg-client
 nifi get-reg-client-id
 nifi pg-import
 nifi pg-start
 nifi pg-stop
 nifi pg-get-vars
 nifi pg-set-var
 nifi pg-get-version
 nifi pg-change-version
 nifi pg-get-all-versions
 nifi pg-list
 nifi pg-status
 nifi pg-get-services
 nifi pg-enable-services
 nifi pg-disable-services
 registry current-user
 registry list-buckets
 registry create-bucket
 registry delete-bucket
 registry list-flows
 registry create-flow
 registry delete-flow
 registry list-flow-versions
 registry export-flow-version
 registry import-flow-version
 registry sync-flow-versions
 registry transfer-flow-version
 session keys
 session show
 session get
 session set
 session remove
 session clear
 exit
 help


 

Transfer Between Servers (NiFi Registries)

registry transfer-flow-version

Transfers a version of a flow directly from one flow to another, without needing
to export/import. If --sourceProps is not specified, the source flow is assumed
to be in the same registry as the destination flow. If --sourceFlowVersion is
not specified, then the latest version will be transferred.

usage: transfer-flow-version
 -f,--flowIdentifier <arg>          A flow identifier
 -h,--help                          Help
 -kp,--keyPasswd <arg>              The key password of the keystore being used
 -ks,--keystore <arg>               A keystore to use for TLS/SSL connections
 -ksp,--keystorePasswd <arg>        The password of the keystore being used
 -kst,--keystoreType <arg>          The type of key store being used (JKS or
                                    PKCS12)
 -ot,--outputType <arg>             The type of output to produce (json or
                                    simple)
 -p,--properties <arg>              A properties file to load arguments from,
                                    command line values will override anything
                                    in the properties file, must contain full
                                    path to file
 -pe,--proxiedEntity <arg>          The identity of an entity to proxy
 -sf,--sourceFlowIdentifier <arg>   A flow identifier from the source registry
 -sfv,--sourceFlowVersion <arg>     A version of a flow from the source registry
 -sp,--sourceProps <arg>            A properties file to load for the source
 -ts,--truststore <arg>             A truststore to use for TLS/SSL connections
 -tsp,--truststorePasswd <arg>      The password of the truststore being used
 -tst,--truststoreType <arg>        The type of trust store being used (JKS or
                                    PKCS12)
 -u,--baseUrl <arg>                 The URL to execute the command against
 -verbose,--verbose                 Indicates that verbose output should be
                                    provided

 

An Example List of My Local Apache NiFi Flows

83399-nifipglistlocal.png

NIFI TOOLKIT Flow Analyzer

bin/flow-analyzer.sh

To run this with my massive amount of flows, I edited the flow-analyze.sh and upped java memory to below:

${JAVA_OPTS:--Xms2G -Xmx2G}

The rest of this article is a big command line dump, seems a huge text list is the way to go:

➜ nifi-toolkit-1.7.0 bin/flow-analyzer.sh /Volumes/seagate/apps/nifi-1.7.0/conf/flow.xml.gz

Using flow=/Volumes/seagate/apps/nifi-1.7.0/conf/flow.xml.gz

Total Bytes Utilized by System=519 GB

Max Back Pressure Size=1 GB

Min Back Pressure Size=1 GB

Average Back Pressure Size=0.990458015 GB

Max Flowfile Queue Size=10000

Min Flowfile Queue Size=10000

Avg Flowfile Queue Size=9904.580152672

bin/file-manager.sh

usage: org.apache.nifi.toolkit.admin.filemanager.FileManagerTool [-b <arg>] [-c <arg>] [-d <arg>] [-h] [-i <arg>] [-m] [-o <arg>] [-r <arg>] [-t <arg>] [-v] [-x]

This tool is used to perform backup, install and restore activities for a NiFi node.

-b,--backupDir <arg> Backup NiFi Directory (used with backup or restore operation)

-c,--nifiCurrentDir <arg> Current NiFi Installation Directory (used optionally with install or restore operation)

-d,--nifiInstallDir <arg> NiFi Installation Directory (used with install or restore operation)

-h,--help Print help info (optional)

-i,--installFile <arg> NiFi Install File

-m,--moveRepositories Allow repositories to be moved to new/restored nifi directory from existing installation, if available (used optionally with

install or restore operation)

-o,--operation <arg> File operation (install | backup | restore)

-r,--nifiRollbackDir <arg> NiFi Installation Directory (used with install or restore operation)

-t,--bootstrapConf <arg> Current NiFi Bootstrap Configuration File (optional)

-v,--verbose Set mode to verbose (optional, default is false)

-x,--overwriteConfigs Overwrite existing configuration directory with upgrade changes (used optionally with install or restore operation)

Java home: /Library/Java/Home

NiFi Toolkit home: /Volumes/seagate/apps/nifi-toolkit-1.7.0

 

Backups

nifi-toolkit-1.7.0 bin/file-manager.sh -o backup -b /Volumes/seagate/backupsNIFI/ -c /Volumes/seagate/apps/nifi-1.7.0 -v

➜ nifi-toolkit-1.7.0 bin/notify.sh

usage: org.apache.nifi.toolkit.admin.notify.NotificationTool [-b <arg>] [-d <arg>] [-h] [-l <arg>] [-m <arg>] [-p <arg>] [-v]

This tool is used to send notifications (bulletins) to a NiFi cluster.

-b,--bootstrapConf <arg> Existing Bootstrap Configuration file

-d,--nifiInstallDir <arg> NiFi Installation Directory

-h,--help Print help info

-l,--level <arg> Level for notification bulletin INFO,WARN,ERROR

-m,--message <arg> Notification message for nifi instance or cluster

-p,--proxyDn <arg> User or Proxy DN that has permission to send a notification. User must have view and modify privileges to 'access the controller'

in NiFi

-v,--verbose Set mode to verbose (default is false)

Java home: /Library/Java/Home

NiFi Toolkit home: /Volumes/seagate/apps/nifi-toolkit-1.7.0

nifi-toolkit-1.7.0 bin/s2s.sh

Must specify either Port Name or Port Identifier to build Site-to-Site client

s2s is a command line tool that can either read a list of DataPackets from stdin to send over site-to-site or write the received DataPackets to stdout

The s2s cli input/output format is a JSON list of DataPackets. They can have the following formats:

[{"attributes":{"key":"value"},"data":"aGVsbG8gbmlmaQ=="}]

Where data is the base64 encoded value of the FlowFile content (always used for received data) or

[{"attributes":{"key":"value"},"dataFile":"/Volumes/seagate/apps/nifi-toolkit-1.7.0/EXAMPLE"}]

Where dataFile is a file to read the FlowFile content from

Example usage to send a FlowFile with the contents of "hey nifi" to a local unsecured NiFi over http with an input port named input:

echo '[{"data":"aGV5IG5pZmk="}]' | bin/s2s.sh -n input -p http

usage: s2s

--batchCount <arg> Number of flow files in a batch

--batchDuration <arg> Duration of a batch

--batchSize <arg> Size of flow files in a batch

-c,--compression Use compression

-d,--direction <arg> Direction (valid directions: SEND, RECEIVE) (default: SEND)

-h,--help Show help message and exit

-i,--portIdentifier <arg> Port id

--keyStore <arg> Keystore

--keyStorePassword <arg> Keystore password

--keyStoreType <arg> Keystore type (default: JKS)

-n,--portName <arg> Port name

--needClientAuth Need client auth

-p,--transportProtocol <arg> Site to site transport protocol (default: RAW)

--peerPersistenceFile <arg> File to write peer information to so it can be recovered on restart

--penalization <arg> Penalization period

--proxyHost <arg> Proxy hostname

--proxyPassword <arg> Proxy password

--proxyPort <arg> Proxy port

--proxyUsername <arg> Proxy username

--timeout <arg> Timeout

--trustStore <arg> Truststore

--trustStorePassword <arg> Truststore password

--trustStoreType <arg> Truststore type (default: JKS)

-u,--url <arg> NiFI URL to connect to (default: http://localhost:8080/nifi)


I can see this being used for integration testing.

Notify

Send a bulletin to A Nifi Server

https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.1.2/bk_administration/content/notify.html

notify.sh 

 

References:


nifipglistlocal.png

4,941 Views
Comments
avatar
Master Guru

if you add -ot json to the end your output will be in JSON format. Which you can parse with your favorite tool, again I was thinking to call it from NiFi and process the output. Perhaps this is Nifinception all over again, perhaps this is Nifinception all over again.

By default it's nice, easy to read text which is perfect if a person is watching it.

It's really cool to be able to start and stop processor groups remotely via a command line command.