Objective This tutorial walks you through how to install and secure a NiFi Registry using client certificates. A quick example of modifying user privileges in the Registry is also included. A video version of this tutorial can be seen here: https://youtu.be/qD03ao3R-a4 Note: To learn the basics of setting up an unsecured Registry and integrating with Apache NiFi see the HCC article Versioned DataFlows with Apache NiFi 1.5 and Apache NiFi Registry 0.1.0. Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi Registry 0.1.0 Apache NiFi Toolkit 1.5.0 Secure NiFi Registry Configuration Download & Extract Tarballs Download the tarball for the 0.1.0 Registry release: nifi-registry-0.1.0-bin.tar.gz and the tarball for the 1.5.0 NiFi Toolkit: nifi-toolkit-1.5.0-bin.tar.gz Extract the tars: tar xzvf nifi-registry-0.1.0-bin.tar.gz tar xzvf nifi-toolkit-1.5.0-bin.tar.gz Generate Configuration and Certificate Files We will use the Apache NiFi TLS Toolkit to generate the necessary keystore, truststore, and client certificates. In this tutorial, we will create certs for two users: "sys_admin" and "test_user". The user “sys_admin” will have full access to the registry while “test_user” will be configured to have targeted access in the registry. In the directory of your NiFi Toolkit install, run the following command: ./bin/tls-toolkit.sh standalone -n "localhost" -C "CN=sys_admin, OU=NIFI" -o target Note: To see the usage information for the TLS Toolkit, run: ./bin/tls-toolkit.sh standalone -h . TLS Toolkit generates the following in the target directory: CN=sys_admin_OU=NIFI.p12 CN=sys_admin_OU=NIFI.p12.password localhost nifi-cert.pem nifi-key.key The localhost directory contains: keystore.jks nifi.properties truststore.jks Registry Configuration Copy the keystore and trustore to the conf directory of your Registry install. Copy the values of the keystore and truststore properties from the nifi.properties file: nifi.security.keystore=./conf/keystore.jks nifi.security.keystoreType=jks nifi.security.keystorePasswd=taceJshGdkyBRy4B7mwaSnM3AkbN7ffewjn3nVIGidw nifi.security.keyPasswd=taceJshGdkyBRy4B7mwaSnM3AkbN7ffewjn3nVIGidw nifi.security.truststore=./conf/truststore.jks nifi.security.truststoreType=jks nifi.security.truststorePasswd=WJwg6F2jmUcvpxRHDiseNRc/VV59WOS+SdrZ5amtnsE into the values for the equivalent properties in the nifi-registry.properties file: nifi.registry.security.keystore=./conf/keystore.jks nifi.registry.security.keystoreType=jks nifi.registry.security.keystorePasswd=taceJshGdkyBRy4B7mwaSnM3AkbN7ffewjn3nVIGidw nifi.registry.security.keyPasswd=taceJshGdkyBRy4B7mwaSnM3AkbN7ffewjn3nVIGidw nifi.registry.security.truststore=./conf/truststore.jks nifi.registry.security.truststoreType=jks nifi.registry.security.truststorePasswd=WJwg6F2jmUcvpxRHDiseNRc/VV59WOS+SdrZ5amtnsE While you are in nifi-registry.properties , modify the HTTP and HTTPS web properties as follows: nifi.registry.web.http.host= nifi.registry.web.http.port= nifi.registry.web.https.host=localhost nifi.registry.web.https.port=18443 In the same Registry conf directory, modify authorizers.xml in two places. First in the userGroupProvider section, add the "sys_admin" DN to the "Initial Admin Identity 1" property: <property name="Initial User Identity 1">CN=sys_admin, OU=NIFI</property> Then in the accessPolicyProvider section, add the "sys_admin" DN to the "Initial Admin Identity" property: <property name="Initial Admin Identity">CN=sys_admin, OU=NIFI</property> Note: During this step, it is crucial that you specify the exact DN string used when the TLS Toolkit was invoked. A common error is entering "CN=sys_admin,OU=NIFI" which will not work as it has a missing space. Add Certificate to Keychain Double-click on the .p12 file that was generated by the TLS Toolkit. When prompted, provide the password from the .password file. Start the Registry In a terminal window, navigate to the directory where NiFi Registry was installed and run: ./bin/nifi-registry.sh start Open Registry UI Navigate to the registry UI in your web browser (Chrome used in the following examples): https://localhost:18443/nifi-registry When prompted, select the "sys_admin" cert to add to your browser: When prompted, enter your "login" keychain password: You should now be able to view the Registry UI as the "CN=sys_admin, OU=NIFI" user: Registry Administration The "sys_admin" user has full access to the registry. Here are some examples of administration functions immediately available. Bucket Creation Select the Settings icon ( ) in the top right corner of the screen. In the Buckets window that appears, select the "New Bucket" button. In the dialog that appears, enter the bucket name "ABC" and select the "Create" button. The "ABC" bucket is created: User Administration Select "Users" at the top of the UI to access the user administration area of the Registry: Select the pencil icon ( ) next to the "CN=sys_admin, OU=NIFI" user. This will open a side nav that shows the Special Privileges and group Membership: You can see that the "sys_admin" was given all special privileges as the Initial Admin Identity (IAI). The privileges for the IAI are not editable. Let's create a second user to see how bucket access can be restricted by modifying these privileges. Second User Creation Close the side nav and select the "Add User" button. Enter "CN=test_user, OU=NIFI" in the Identity field and select the "Add" button: "CN=test_user", OU=NIFI" user is created: Second User Certificate Next we need a client certificate for "test_user". Return to the directory of your NiFi Toolkit installation and run: ./bin/tls-toolkit.sh standalone -C "CN=test_user, OU=NIFI" -o target NOTE:The output directory must be set to target in order for the existing CA certificate in that directory to be used. TLS Toolkit generates the following additional files in the target directory: CN=test_user_OU=NIFI.p12 CN=test_user_OU=NIFI.p12.password Add the .p12 cert to the Keychain as described earlier. However, choose a different browser this time to access the UI (Safari in the following examples): https://localhost:18443/nifi-registry Add the client certificate to the browser: You should now be able to view the Registry UI as the "CN=test_user, OU=NIFI" user: You can see that "test_user" has no access to Settings. Return to the Chrome browser where "sys_admin" is the user. Give "test_user" read-only bucket privileges: Return to the Safari browser where "test_user" is the user. Reload the browser. Select the Settings icon which is now available. The ABC bucket is now visible, but note that the Action to delete the bucket is not enabled, which is consistent with the privileges given to this user: Additional Help If you would like to learn more about NiFi Registry functionality and working with versioned flows in NiFi, see the following articles: Versioned DataFlows with Apache NiFi 1.5 and Apache NiFi Registry 0.1.0 Apache NiFi - How do I deploy my flow? Or documentation: Apache NiFi Registry User Guide Apache NiFi Registry System Administrator's Guide Versioning a DataFlow (Apache NiFi User Guide) ... View more

Objective This article highlights some of the latest UI enhancements added in Apache NiFi 1.5.0. Environment The examples shown in the article utilized the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.5.0 "Primary Node" Processors Identification In a NiFi Cluster, processors that have been configured for "Primary node" execution are now identified in the UI by a "P". On the canvas, the "P" is visible next to the processor icon: The "P" is also shown in the Processors tab on the Summary page, specifically in the Name column: Finding Processors Quickly in the Summary Page If your flow has hundreds of processors, it can be difficult differentiating between them in the Summary page (accessible from the top-right Global menu). On the Processors tab, a "Process Group" column has been added to display the name of the parent process group containing the component: Additionally, when hovering over the "Go to location" button the tooltip now includes the path of the component. NiFi Registry Integration NiFi 1.5.0 is the first release to integrate with the Apache NiFi Registry. NiFi dataflows can now be versioned on the process group level and easily deployed across different NiFi instances. More information can be found in the HCC article "Versioned DataFlows with Apache NiFi 1.5 and Apache NiFi Registry 0.1.0" and in the "Versioning a Dataflow" section of the NiFi User Guide. However, here are some related UI changes to highlight. Connecting a Registry Client The NiFi Settings window (accessible from Controller Settings in the top-right Global menu) now has a "Registry Clients" tab where you can connect NiFi to a NiFi Registry: Importing a Flow If your NiFi instance is connected to an active Registry, when adding a process group to the canvas there is also an option to "Import" a versioned flow: Selecting "Import" prompts the user to choose a version of a flow to add to the canvas: Version States There are new icons that show: the version state of an individual process group the count of the statuses of versioned process groups within a process group the count of the statuses of versioned process groups in the root process group Here are the meanings of each icon/state: Up to date Locally modified Stale Locally modified and stale Sync failure Version state information is also shown in the "Process Groups" tab of the Summary Page: As mentioned previously, more information regarding NiFi and NiFi Registry integration can be found in the "Versioning a Dataflow" section of the NiFi User Guide. ... View more

Objective This tutorial walks you through how to install and setup a local Apache NiFi Registry to integrate with Apache NiFi and start using versioned NiFi dataflows. It assumes basic experience with NiFi but little to no experience with NiFi Registry. A video version of this tutorial can be seen here: https://youtu.be/X_qhRVChjZY Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.5.0 Apache NiFi Registry 0.1.0 Note: Apache NiFi 1.5.0 is the first NiFi release to support integration with the NiFi Registry. Nifi Registry 0.1.0 is the first and currently only version of the application. Apache NiFi Registry Configuration Registry Installation Download the tarball of the 0.1.0 Registry release: nifi-registry-0.1.0-bin.tar.gz Extract the tar: tar xzvf nifi-registry-0.1.0-bin.tar.gz Start the Registry In a terminal window, navigate to the directory where NiFi Registry was installed. Run: bin/nifi-registry.sh start Open Registry UI Navigate to the registry UI in your browser: http://localhost:18080/nifi-registry Note:By default the registry is unsecured. The port can be changed by editing the nifi-registry.properties file in the NiFi Registry conf directory (the exact property to change is nifi.registry.web.http.port), but the default port is 18080. Bucket Creation A Bucket is a container that stores and organizes flows in the Registry. The Registry is empty as there are no buckets/flows yet. To create a bucket, select the Settings icon ( ) in the top right corner of the screen. In the Buckets window that appears, select the "New Bucket" button. Enter the bucket name "Test" and select the "Create" button. The "Test" bucket is created: There are no permissions configured by default, so anyone is able to view, create and modify buckets in this instance. For information on securing the Registry, see the NiFi Registry System Administrator’s Guide. Apache NiFi Configuration Connect NiFi to the Registry With the Registry is running, we can tell NiFi about it. In NiFi, select "Controller Settings" from the top-right Global menu: Select the Registry Clients tab and the "+" button to add a new Registry Client. Enter a name and the URL of the Registry instance (http://localhost:18080): Versioned DataFlows Start Version Control on a Process Group NiFi can now place a process group under version control which saves it as a flow resource in the Registry. Right-click on a process group and select "Version→Start version control" from the context menu: The local registry instance and "Test" bucket are chosen by default to store your flow since they are the only registry connected and bucket available. Enter a flow name, flow description, comments and select "Save": As indicated by the Version State icon ( ) in the top left corner of the component, the process group is now saved as a versioned flow in the registry. Go back to the Registry UI and return to the main page to see the versioned flow you just saved (a refresh may be required): Save Changes to a Versioned Flow Changes made to the versioned process group can be reviewed, reverted or saved. For example, if changes are made to the ABCD flow, the Version State changes to "Locally modified" ( ). The right-click menu will now show the options "Commit local changes", "Show local changes" or "Revert local changes": Select "Show local changes" to see the details of the changes made: Return to the context menu and select "Commit local changes". Enter comments and select "Save" to save the changes: Version 2 of the flow is saved: Note: Some actions made to the versioned process group are not considered local changes. More information can be found in the Managing Local Changes section of the NiFi User Guide. Import a Versioned Flow With a flow existing in the Registry, we can use it to illustrate how to import a versioned process group. In NiFi, select Process Group from the Components toolbar and drag it onto the canvas: Instead of entering a name, click the Import link: Choose the version of the flow you want imported and select "Import": A second identical PG is now added: Help To learn more about NiFi Registry functionality and working with versioned flows in NiFi, see the following links: Apache NiFi Registry User Guide Apache NiFi Registry System Administrator's Guide Versioning a DataFlow (Apache NiFi User Guide) Apache NiFi - How do I deploy my flow? ... View more

Objective This is the second of a two article series on the ValidateRecord processor. The first walks you through a NiFI flow that converts a CVS file into JSON format and validates the data against a given schema. This article discusses the effects of enabling/disabling the "Strict Type Checking" property of the ValidateRecord processor. Note: The ValidateRecord processor was introduced in NiFi 1.4.0. Environment This tutorial was tested using the following environment and components: Mac OS X 10.11.6 Apache NiFi 1.4.0 Strict Type Checking Property A useful property of the ValidateRecord processor is "Strict Type Checking". If the incoming data has a Record where a field is not of the correct type, this property determines how to handle the Record. If set to "true", the Record will be considered invalid. If set to "false", the Record will be considered valid. To demonstrate both cases, we need to ingest data that can distinguish between different types (which our CSV data from the first article could not). Let's grab a snippet of the JSON candy data and make some changes. Specifically let's put a string value for the "chocolate" field (which is of type int) and let's put a decimal value for the "competitorname" field (which is of type string😞 [ { "competitorname" : "One dime", "chocolate" : "0", "fruity" : 0, "caramel" : 0, "peanutyalmondy" : 0, "nougat" : 0, "crispedricewafer" : 0, "hard" : 0, "bar" : 0, "pluribus" : 0, "sugarpercent" : 0.011, "pricepercent" : 0.116, "winpercent" : 32.261086 }, { "competitorname" : 3.14159, "chocolate" : 1, "fruity" : 0, "caramel" : 0, "peanutyalmondy" : 0, "nougat" : 0, "crispedricewafer" : 1, "hard" : 0, "bar" : 0, "pluribus" : 1, "sugarpercent" : 0.87199998, "pricepercent" : 0.84799999, "winpercent" : 49.524113 } ] Here is the JSON file: type-checking.txt (Change the extension from .txt to .json after downloading) Place the type-checking.json file in your input directory: In order to process the JSON file, the ValidateRecord processor needs to use a JSON Record Reader. Go to the configuration window for the processor and select "Create new service..." for the Record Reader: Select JSONTreeReader, then "Create": and then select the Arrow icon next to the reader: Save the changes made before going to the Controller Service. Go to the configuration window of the JsonTreeReader controller service, select "AvroSchemaRegistry" for the Schema Registy and then select Apply: Enable the JsonTreeReader service. The flow is ready to run. Start the GetFile, UpdateAtttribute and ValidateRecord processors. With "Strict Type Checking" set to "true", the 2 records are considered invalid and are routed to that connection: Start the LogAttribute processor to clear the queue. Stop all processors. Place the type-checking.json file in your input directory again. Now let's change the Strict Type Checking property to "false": Running the flow this time, the 2 records are considered valid and are routed to that connection: Note: The documentation for the Strict Type Checking property states that when set to false, the relevant record fields will be coerced into the correct type. This functionality is currently broken (see NIFI-4579). ... View more