Created on 05-11-2016 12:26 AM
Raspberry Pi is an interesting little machine. Even more so with the release of the Raspberry Pi 3. While it may not technically be the most powerful device in this form-factor, it definitely wins on overall community, breadth of support and stability. Naturally, there is sizable interest in running something like NiFi on this little edge device. And that's exactly what we have been doing for the past 8 months (as of May 2016). Today I'd like to share a set of best practices for running NiFi on the Pi that came out of this exercise.
In general, recommendations fall into 2 categories:
Let’s start with some prep work.
I’m assuming the Raspberry Pi is fully under one’s control, including physical access to SD card and USB slots, together with a root account.
If you bought a starter kit, it probably came with the NOOBS image pre-installed on the SD card. It’s fine for all kinds of projects, but running things 24x7 changes things. Double-check the manufacturer and use a brand-name SD card like SanDisk. Unfortunately, those other brands in starter kits tend to corrupt much more often, and this is not what you want for an always-on service. Go for a faster SD card. E.g. look for those cards having the 4K video designation and 90 MB/s sustained write speed or better. Your Pi will thank you.
Raspberry Pi default OS image has lots of stuff. All great, but absolutely useless and a waste if you never intend to run it in a desktop mode. Instead, take your SD card and re-image it with the lightweight version of the OS (Raspbian Jessie Lite at the time of writing). It drops all desktop software, giving you more space to do what’s important (i.e. running NiFi 24x7!). This is a standard procedure which is well documented at https://www.raspberrypi.org/documentation/installation/installing-images/
One is encouraged to upgrade the OS to pick up any maintenance releases since the OS image was published:
sudo apt-get update && sudo apt-get dist-upgrade
Go have a coffee or make a sandwich.
Of course, there is this little chicken-and-egg problem of having to connect your typical desktop monitor and keyboard first to go through an initial boot and configuration. One has 3 options:
Once the lightweight OS has been put on the SD card and Raspberry Pi boots up, there’s one thing often overlooked. The default root partition is around 1.2GB and it will get full eventually. It’s very tedious then to clean up the space, as there is no obvious single large file one could delete to reclaim the space. Instead, use all that SD card (remember NiFi will be running on another mount anyway).
Login into your Pi and run:
Select option #1 - Expand Filesystem. The system will expand the root partition to use available SD card space and reboot.
This is optional, but really not if you plan to collect any interesting data and still make sense of it during analysis. While in the same raspi-config screen, select option #5 Internationalization and update your Pi’s Locale and Timezone.
If you can, consider external storage for hosting everything NiFi (I.e. not on the same SD card where your OS lives). One can go as far as plugging in a huge external drive (hey, no problem, just ensure it has its own power supply!) into a USB port. But we found a common USB flash drive (or multiple) to be a suitable medium, too. They seem to be less prone to corruption than the SD card, just go with the brand name and opt for higher-speed models whenever you can.
I will not repeat the internet, there are plenty of guides on how to do it for the Pi. One super-useful tip is to mount by a UUID of the flash drive, which guarantees the mount bindings will persist no matter which USB port one plugs it in. Gives the warm and fuzzy, which we all love. Here’s my favorite guide: http://www.raspberrypi-spy.co.uk/2014/05/how-to-mount-a-usb-flash-disk-on-the-raspberry-pi/ . Use either UUDI or a drive label, just don't use the actual device name, as it can change when you reconfigure the disks.
Additionally, modify the mounts for the SD card and USB flash drives to disable access times recording for files and directories. This minimizes unnecessary writes to the SD card and flash drives:
sudo vi /etc/fstab # modify the options for a mount to add ‘noatime,nodiratime’ # save changes # repeat for every mount point which you updated above, example for root below sudo mount -o remount / # to verify changes - look for your settings in the output mount
E.g. here’s how my fstab looks like with an SD card and 2 USB flash drives:
proc /proc proc defaults 0 0 /dev/mmcblk0p1 /boot vfat defaults 0 2 /dev/mmcblk0p2 / ext4 defaults,noatime 0 1 LABEL=USBFLASH1 /mnt/flash1 ext4 defaults,noatime,nodiratime,nofail 0 LABEL=USBFLASH2 /mnt/flash2 ext4 defaults,noatime,nodiratime,nofail 0
Update: a more robust fstab options string. If your Pi is hanging on boot with auto-mounted drives, try add the nofail option.
This is kinda critical and will bite you every time if you forget. By default, Raspberry Pi shuts down wifi after some inactivity period only to re-enable it when there’s an incoming connection. Great in theory. In practice, one is facing a 30-40 seconds delay when trying to access the Pi over SSH or hit a NiFi UI. The procedure is slightly different for Pi 2 & 3, I’m listing both here:
sudo iw dev wlan0 set power_save off
We will not be running NiFi from the SD card. Let’s ensure the home directory isn’t there either. Create a nifi user account and configure its home location to be on the new USB mount (e.g. if there is a /mnt/flash1):
sudo useradd -m -d /mnt/flash1/nifi nifi sudo chmod 750 /mnt/flash1/nifi # and here’s how one would log into the account # sudo su - nifi
Follow the https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#configuration-best-practices I, however, configure the limits for the nifi user alone, not for any account on the host. E.g. replace those star symbols with ‘nifi':
nifi hard nofile 50000 nifi soft nofile 50000 nifi hard nproc 10000 nifi soft nproc 10000
Java 8 is highly recommended. Largely because the PermGen space doesn’t require any special treatment, great for systems like NiFi.
Important: we had some reports of OpenJDK behaving less stable than Oracle JDK in the ARM build (this is what Pi is using). Save yourself some trouble, install Oracle JDK and move on:
sudo apt-get update && sudo apt-get install oracle-java8-jdk
NiFi installation is trivial, download from https://nifi.apache.org/download.html and unpack. Make sure to unpack in the nifi user home directory (e.g. /mnt/flash1/nifi)
Before anything else, it’s important to remove modules which don’t make sense on Raspberry Pi. This will improve startup times considerably and reduce strain on the system. You have a final say of what goes and stays, but here’s e.g. the contents of my $NIFI_HOME/lib directory (you can delete all other NAR files):
bootstrap jcl-over-slf4j-1.7.12.jar jul-to-slf4j-1.7.12.jar log4j-over-slf4j-1.7.12.jar logback-classic-1.1.3.jar logback-core-1.1.3.jar nifi-api-0.6.1.jar nifi-documentation-0.6.1.jar nifi-framework-nar-0.6.1.nar nifi-html-nar-0.6.1.nar nifi-http-context-map-nar-0.6.1.nar nifi-jetty-bundle-0.6.1.nar nifi-kerberos-iaa-providers-nar-0.6.1.nar nifi-ldap-iaa-providers-nar-0.6.1.nar nifi-nar-utils-0.6.1.jar nifi-properties-0.6.1.jar nifi-provenance-repository-nar-0.6.1.nar nifi-runtime-0.6.1.jar nifi-scripting-nar-0.6.1.nar nifi-ssl-context-service-nar-0.6.1.nar nifi-standard-nar-0.6.1.nar nifi-standard-services-api-nar-0.6.1.nar nifi-update-attribute-nar-0.6.1.nar slf4j-api-1.7.12.jar
If you have multiple USB drives you can leverage striping for content and provenance repositories. This will improve overall throughput of NiFi.
For a great background and tuning tips see:
Reference documentation for all properties is available at https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
NiFi can get pretty chatty about, well, everything. In an environment that is sensitive to random writes (like the Raspberry Pi) we are better off changing a few things. Edit the $NIFI_HOME/conf/logback.xml and introduce the following changes.
<root level="ERROR"> <appender-ref ref="APP_FILE"/> </root>
<logger name="org.apache.nifi" level="WARN"/>
Find the appender section for nifi-app.log and modify the file pattern to read as below (note the .gz extension):
Bonus tip: there’s no need to restart NiFi to pick up changes in the logging configuration. The file is checked for changes every 30 seconds and NiFi reconfigures logging on the fly.
Once again, these are only a few tips which came through very useful while running NiFi on a Raspberry Pi. There are more tuning steps available based on the kind of data flowing through it, but hope this gets you started and provides sufficient guard rails. Let us know your thoughts!