Running NiFi on Raspberry Pi. Best Practices.

[#1]

Overview

Raspberry Pi is an interesting little machine. Even more so with the release of the Raspberry Pi 3. While it may not technically be the most powerful device in this form-factor, it definitely wins on overall community, breadth of support and stability. Naturally, there is sizable interest in running something like NiFi on this little edge device. And that's exactly what we have been doing for the past 8 months (as of May 2016). Today I'd like to share a set of best practices for running NiFi on the Pi that came out of this exercise.

In general, recommendations fall into 2 categories:

1. Raspberry Pi configuration – OS, system-level settings for running an always-on NiFi instance.
2. NiFi configuration – tweaks and changes within NiFi itself

Let’s start with some prep work.

Raspberry Pi Configuration

I’m assuming the Raspberry Pi is fully under one’s control, including physical access to SD card and USB slots, together with a root account.

SD Card

If you bought a starter kit, it probably came with the NOOBS image pre-installed on the SD card. It’s fine for all kinds of projects, but running things 24x7 changes things. Double-check the manufacturer and use a brand-name SD card like SanDisk. Unfortunately, those other brands in starter kits tend to corrupt much more often, and this is not what you want for an always-on service. Go for a faster SD card. E.g. look for those cards having the 4K video designation and 90 MB/s sustained write speed or better. Your Pi will thank you.

Use Raspbian OS Lite

Raspberry Pi default OS image has lots of stuff. All great, but absolutely useless and a waste if you never intend to run it in a desktop mode. Instead, take your SD card and re-image it with the lightweight version of the OS (Raspbian Jessie Lite at the time of writing). It drops all desktop software, giving you more space to do what’s important (i.e. running NiFi 24x7!). This is a standard procedure which is well documented at https://www.raspberrypi.org/documentation/installation/installing-images/

Upgrade OS Packages

One is encouraged to upgrade the OS to pick up any maintenance releases since the OS image was published:

sudo apt-get update && sudo apt-get dist-upgrade

Go have a coffee or make a sandwich.

Note on Initial Setup

Of course, there is this little chicken-and-egg problem of having to connect your typical desktop monitor and keyboard first to go through an initial boot and configuration. One has 3 options:

1. Plug in an Ethernet cable and access over SSH. You may need to have access to a router or basic network scanning utility to find Pi's address. @Jim Heaton has more tips in the comment section below. Thanks!
2. Connect a monitor and keyboard and configure WiFi, start up the SSH daemon and set your Pi free.
3. Hardcore, but lots of fun - connect directly to a serial console over TX/RX/GND wire combination. Finally a good use for that array of GPIO connectors!. You can either buy one of those USB-to-TTL cables or leverage e.g. a BusPirate device if you have one. If above makes sense, you probably can find your way from here, but let me know if you’re interested in knowing more. I can only mention that it saved me so many times even in a disk full, SSH daemon down, SD card corrupted situation (or a combination of all).

Expand the Root Filesystem

Once the lightweight OS has been put on the SD card and Raspberry Pi boots up, there’s one thing often overlooked. The default root partition is around 1.2GB and it will get full eventually. It’s very tedious then to clean up the space, as there is no obvious single large file one could delete to reclaim the space. Instead, use all that SD card (remember NiFi will be running on another mount anyway).

Login into your Pi and run:

sudo raspi-config

Select option #1 - Expand Filesystem. The system will expand the root partition to use available SD card space and reboot.

Configure OS Locale and Timezone

This is optional, but really not if you plan to collect any interesting data and still make sense of it during analysis. While in the same raspi-config screen, select option #5 Internationalization and update your Pi’s Locale and Timezone.

External Storage

If you can, consider external storage for hosting everything NiFi (I.e. not on the same SD card where your OS lives). One can go as far as plugging in a huge external drive (hey, no problem, just ensure it has its own power supply!) into a USB port. But we found a common USB flash drive (or multiple) to be a suitable medium, too. They seem to be less prone to corruption than the SD card, just go with the brand name and opt for higher-speed models whenever you can.

Configure USB Mounts

I will not repeat the internet, there are plenty of guides on how to do it for the Pi. One super-useful tip is to mount by a UUID of the flash drive, which guarantees the mount bindings will persist no matter which USB port one plugs it in. Gives the warm and fuzzy, which we all love. Here’s my favorite guide: http://www.raspberrypi-spy.co.uk/2014/05/how-to-mount-a-usb-flash-disk-on-the-raspberry-pi/ . Use either UUDI or a drive label, just don't use the actual device name, as it can change when you reconfigure the disks.

Disable Access Times Recording

Additionally, modify the mounts for the SD card and USB flash drives to disable access times recording for files and directories. This minimizes unnecessary writes to the SD card and flash drives:

sudo vi /etc/fstab
# modify the options for a mount to add ‘noatime,nodiratime’
# save changes

# repeat for every mount point which you updated above, example for root below
sudo mount -o remount /


# to verify changes - look for your settings in the output
mount

E.g. here’s how my fstab looks like with an SD card and 2 USB flash drives:

proc  /proc  proc  defaults  0  0
/dev/mmcblk0p1  /boot  vfat  defaults  0  2
/dev/mmcblk0p2  /  ext4  defaults,noatime 0  1


LABEL=USBFLASH1 /mnt/flash1     ext4    defaults,noatime,nodiratime,nofail 0
LABEL=USBFLASH2 /mnt/flash2     ext4    defaults,noatime,nodiratime,nofail 0

Update: a more robust fstab options string. If your Pi is hanging on boot with auto-mounted drives, try add the nofail option.

Disable WiFi Power Saving

This is kinda critical and will bite you every time if you forget. By default, Raspberry Pi shuts down wifi after some inactivity period only to re-enable it when there’s an incoming connection. Great in theory. In practice, one is facing a 30-40 seconds delay when trying to access the Pi over SSH or hit a NiFi UI. The procedure is slightly different for Pi 2 & 3, I’m listing both here:

Raspberry Pi 2

Follow instructions at http://www.raspberrypi-spy.co.uk/2015/06/how-to-disable-wifi-power-saving-on-the-raspberry-pi/

Raspberry Pi 3

sudo iw dev wlan0 set power_save off

Create a NiFi User Account

We will not be running NiFi from the SD card. Let’s ensure the home directory isn’t there either. Create a nifi user account and configure its home location to be on the new USB mount (e.g. if there is a /mnt/flash1):

sudo useradd -m -d /mnt/flash1/nifi nifi
sudo chmod 750 /mnt/flash1/nifi

# and here’s how one would log into the account
# sudo su - nifi

Configure OS Kernel

Follow the https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices I, however, configure the limits for the nifi user alone, not for any account on the host. E.g. replace those star symbols with ‘nifi':

nifi  hard  nofile  50000
nifi  soft  nofile  50000
nifi  hard  nproc   10000
nifi  soft  nproc   10000

Install JDK

Java 8 is highly recommended. Largely because the PermGen space doesn’t require any special treatment, great for systems like NiFi.

Important: we had some reports of OpenJDK behaving less stable than Oracle JDK in the ARM build (this is what Pi is using). Save yourself some trouble, install Oracle JDK and move on:

sudo apt-get update && sudo apt-get install oracle-java8-jdk

NiFi Configuration

NiFi installation is trivial, download from https://nifi.apache.org/download.html and unpack. Make sure to unpack in the nifi user home directory (e.g. /mnt/flash1/nifi)

Let’s Get Lean!

Before anything else, it’s important to remove modules which don’t make sense on Raspberry Pi. This will improve startup times considerably and reduce strain on the system. You have a final say of what goes and stays, but here’s e.g. the contents of my $NIFI_HOME/lib directory (you can delete all other NAR files):

bootstrap
jcl-over-slf4j-1.7.12.jar
jul-to-slf4j-1.7.12.jar
log4j-over-slf4j-1.7.12.jar
logback-classic-1.1.3.jar
logback-core-1.1.3.jar
nifi-api-0.6.1.jar
nifi-documentation-0.6.1.jar
nifi-framework-nar-0.6.1.nar
nifi-html-nar-0.6.1.nar
nifi-http-context-map-nar-0.6.1.nar
nifi-jetty-bundle-0.6.1.nar
nifi-kerberos-iaa-providers-nar-0.6.1.nar
nifi-ldap-iaa-providers-nar-0.6.1.nar
nifi-nar-utils-0.6.1.jar
nifi-properties-0.6.1.jar
nifi-provenance-repository-nar-0.6.1.nar
nifi-runtime-0.6.1.jar
nifi-scripting-nar-0.6.1.nar
nifi-ssl-context-service-nar-0.6.1.nar
nifi-standard-nar-0.6.1.nar
nifi-standard-services-api-nar-0.6.1.nar
nifi-update-attribute-nar-0.6.1.nar
slf4j-api-1.7.12.jar

Repositories Configuration

If you have multiple USB drives you can leverage striping for content and provenance repositories. This will improve overall throughput of NiFi.

For a great background and tuning tips see:

https://community.hortonworks.com/content/kbentry/7882/hdfnifi-best-practices-for-setting-up-a-high-...

Reference documentation for all properties is available at https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties

Tame Those Logs

NiFi can get pretty chatty about, well, everything. In an environment that is sensitive to random writes (like the Raspberry Pi) we are better off changing a few things. Edit the $NIFI_HOME/conf/logback.xml and introduce the following changes.

Only ERROR and above messages by default

<root level="ERROR">
  <appender-ref ref="APP_FILE"/>
</root>

NiFi packages at WARN and above (add one if missing)

<logger name="org.apache.nifi" level="WARN"/>

Compress and roll daily logs for nifi-app.log

Find the appender section for nifi-app.log and modify the file pattern to read as below (note the .gz extension):

<fileNamePattern>./logs/nifi-app_%d{yyyy-MM-dd_HH}.%i.log.gz</fileNamePattern>

Bonus tip: there’s no need to restart NiFi to pick up changes in the logging configuration. The file is checked for changes every 30 seconds and NiFi reconfigures logging on the fly.

Closing Thoughts

Once again, these are only a few tips which came through very useful while running NiFi on a Raspberry Pi. There are more tuning steps available based on the kind of data flowing through it, but hope this gets you started and provides sufficient guard rails. Let us know your thoughts!

[#2]

I use the Ethernet connection on a new PI install frequently. Clean and fast. You can install it right next to your router, wherever than happens to be. Updates from Github are much quicker. You just need a network scanning tool to find the new IP that appears, or a router you can configure to give the same IP to a given MAC address every time. Alternatively, get to the point where you configure the image using a keyboard and screen, and then save that boot image to a laptop. Start from there each time. Many routers will reassign the same IP going forward.

[#3]

Thank you Jim, I've updated the Ethernet section to give you credit.

[#4]

You don't even need to scan the network. If you're using a default Pi image, it does multicast DNS. As soon as it boots, you can just do ssh raspberrypi.local from a machine on the same subnet.

[#5]

Build your MicroSD card with www.pibakery.org it lets you pre configure boot, wifi, ...