# Ruby Scripting in NiFi Today I was asked about an issue that I didn't know how to solve using NiFi. On the surface it sounded simple; just map an attribute to another value. The attribute to map was 'port' and based on the port number just add an attribute to more easily identify the system downstream. E.g. for port 10003; syslog:local, for 10004; syslog:db1, etc. After a bit of digging I found a few different options to solve this. ## Many UpdateAttribute Processors The first is to create a new UpdateAttribute processor for each incoming stream. This labels (places an attribute) on all files that come in from that listener. It looked like this:  This looks a bit confusing and tedious but is very precise and arguably easier to read, especially when we label the processors. It also has the added advantage of not having to change the port in more than one place. If for example, the local logs coming in over port 10002 need to change to port 10003, then we just make that change in the ListenSyslog processor and the rest remains unchanged. ## One UpdateAttribute Using the Advanced The advanced option allowed me to keep all the configuration in one place, easily mapping port numbers to tags. The two disadvantages I ran into were: 1. A fairly tedious process to get each mapping. It involved: * Create new rule * Add name * Search for existing rule to import * Change port number and associated label 2. Must now change the port in two different places if it were to change I would look like:  ## ExecuteScript Processor This again allows me to keep all the configuration in one place and makes it much easier to make changes. I created a processor that stores the mappings in a hash and adds the correct attribute appropriately. It looks like so:  From the UI perspective it looks very similar to the single UpdateAttribute solution. This requires the addition of the script: {% highlight ruby %} map = { 10 => "system1", 9 => "system2", 8 => "system3", } map.default = "unknown" flowFile = session.get() return unless flowFile label = map[flowFile.getAttribute("port")] flowFile = session.putAttribute(flowFile, "system", label) session.transfer(flowFile, REL_SUCCESS) session.commit() {% endhighlight %} It is more complex by adding the need to understand a scripting language and also doesn't remove the requirement of changing the port number in more than one place. The script can add more complexity if it becomes necessary to reference as a file rather than using the 'Script Body' option in the processor. The main advantage is that it makes it easier to change the mapping - just copy and paste one of the lines of the hash and make changes all in one place. Given NiFi's goal of minimising the need for data flow managers to know how to code, it's unlikely this is the best approach. # Conclusion The first option is quite foreign to programmers who feel like it isn't generic. This is understandable given that it does feel a bit like copy and paste. I would say it is the most NiFi way of achieving the mapping as it is the solution which is most self-describing and resistant to change. ... View more

Step 1: Download Firstly download and unzip the package (I used: nifi-toolkit-1.0.0-bin.tar.gz) Inside there are a number of folders and files but I am mostly interested in bin/tls-toolkit.sh. This can be run in either standalone or server/client mode: Standalone is for a one-off generation of certificates and keys Client/Server allows you to run the tls toolkit as a server to sign Certificate Signing Requests from clients Step 2: Generate Keys and Certificates Initially I ran the help: ./bin/tls-toolkit.sh standalone -h Then I created three bundles for each of my servers: ./bin/tls-toolkit.sh standalone -n 'localhost(3)' -C 'CN=scarroll,OU=NIFI' -O -o ../security_output Which generated my truststore, keystore and nifi.properties file for my three hosts and my client certificates as well. Here is the output: $ tree security_output/ security_output/ ├── CN=scarroll_OU=NIFI.p12 ├── CN=scarroll_OU=NIFI.password ├── localhost │ ├── keystore.jks │ ├── nifi.properties │ └── truststore.jks ├── localhost_2 │ ├── keystore.jks │ ├── nifi.properties │ └── truststore.jks ├── localhost_3 │ ├── keystore.jks │ ├── nifi.properties │ └── truststore.jks ├── nifi-cert.pem └── nifi-key.key Then I copied the truststore, keystore and nifi.properties files out to each host. I was using vagrant so I just ran cp /vagrant/security_output/localhost/* /opt/nifi-1.0.0-BETA/conf/ WARNING: This will overwrite your old nifi.properties file which is not a problem for a clean system like mine. To maintain your old configuration you can manually copy the relevant security settings over or pass your existing nifi.properties file into the tls-toolkit which will modify the correct values. Issue #1: HTTPS listening only on localhost Initially NiFi wouldn't start on my external interface, but it is easy to resolve. Whatever value is in -n switch for the toolkit will be set to the nifi.web.https.host option. I just set nifi.web.https.host to empty and restarted NiFi. This will mean NiFi is now listening on all interfaces and may not be the most secure way of running NiFI however is fine for a demo Issue #2: ERR_CONNECTION_CLOSED If you try to access the UI now you should see an error: I was expecting to see a connection but get a permission denied/forbidden error so this stumped me for a while but seems to be standard behaviour for NiFi. The solution is to offer up some way to authenticate yourself. Step 3: Importing certificates to Chrome Since I am not integrating NiFi with any user management system I'll need to import the client certificates into Chrome to get access to the NiFi UI. I'm on a mac and use the Keychain access program for this so can open this directly. Alternatively, settings → Manage Certificates will get you to the same place. Once there pick a keychain that is unlocked (Initially I used System which prompted for a password every time I connected) so instead I created a new keychain called nifi-certs. Next import the CN=scarroll_OU=NIFI.p12 file and enter the password in the CN=scarroll_OU=NIFI.password file. Unfortunately on a mac this text box cannot be pasted into, so the super long secure password that the toolkit generated needs to be typed by hand (or some other workaround). Now you can restart chrome and you will be asked to chose a certificate to present: Issue #3: Forbidden Now even through NiFI knows who you are, you shouldn't be allowed to do anything. If open, NiFi gives power to all users by default however when secured, NiFi gives no permissions as default. The path to allow yourself access to the UI varies depending on whether this instance is a new instance or an upgrade. For a new instance you need to add yourself (or another admin) as the Initial Admin Identity and for an upgrade you can use the legacy existing authorized-users.xml file. Both settings can be found in the authorizers.xml. Since this is a new instance I added myself as an initial admin and restarted NiFi. Restarting is important so the Initial Admin credentials gets populated to the users.xml and authorizations.xml files. Now I can log in to my newly secured instance. ... View more