Why does my MiNiFi Flow fail to run when turning off my Raspberry Pi?

[#1]

Hello.

I am new to MiNiFi. I created MiNiFi flow that executes a python script to ingest raw sensor data using the ExecuteProcess processor and sends that data to my nifi core using a Remote Process Group. The flow runs fine the first time I install MiNiFi on the Raspberry Pi while it is powered and run the flow. I am able to the data transfered to the NiFi core running on my HDP sandbox. The "input port" name on NiFi core matches the particular connection name for the Remote Process Group on MiNiFi. However, when the Raspberry Pi is turned off and then back on, the flow no longer works. I look in the logs of MiNiFi and receive the following message:

java.lang.Exception: Unable to load flow due to: java.io.IOException: Expected to read a Sentinel Byte of '1' but got a value of '0' instead

at org.apache.nifi.minifi.MiNiFiServer.start(MiNiFiServer.java:125)

at org.apache.nifi.minifi.MiNiFi.<init>(MiNiFi.java:134)

at org.apache.nifi.minifi.MiNiFi.main(MiNiFi.java:233)

Caused by: java.io.IOException: Expected to read a Sentinel Byte of '1' but got a value of '0' instead

at org.apache.nifi.repository.schema.SchemaRecordReader.readRecord(SchemaRecordReader.java:65)

at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeRecord(SchemaRepositoryRecordSerde.java:111)

at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:105)

at org.apache.nifi.controller.repository.SchemaRepositoryRecordSerde.deserializeEdit(SchemaRepositoryRecordSerde.java:46)

at org.wali.MinimalLockingWriteAheadLog$Partition.recoverNextTransaction(MinimalLockingWriteAheadLog.java:1072)

at org.wali.MinimalLockingWriteAheadLog.recoverFromEdits(MinimalLockingWriteAheadLog.java:442)

at org.wali.MinimalLockingWriteAheadLog.recoverRecords(MinimalLockingWriteAheadLog.java:301)

at org.apache.nifi.controller.repository.WriteAheadFlowFileRepository.loadFlowFiles(WriteAheadFlowFileRepository.java:346)

at org.apache.nifi.controller.FlowController.initializeFlow(FlowController.java:700)

at org.apache.nifi.controller.StandardFlowService.loadFromBytes(StandardFlowService.java:701)

at org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:508)

at org.apache.nifi.minifi.MiNiFiServer.start(MiNiFiServer.java:113)

... 2 common frames omitted

2016-12-29 18:00:34,221 INFO [Thread-1] org.apache.nifi.minifi.MiNiFi Initiating shutdown of MiNiFi server...

2016-12-29 18:00:34,224 ERROR [Thread-1] org.apache.nifi.BootstrapListener Failed to communicate with Bootstrap. Bootstrap may be unable to issue or receive commands from MiNiFi

2016-12-29 18:00:34,224 INFO [Thread-1] org.apache.nifi.minifi.MiNiFi MiNiFi server shutdown completed (nicely or otherwise).

Any thoughts on why this is happening?

Here is my config.yml for the MiNiFi flow.configyml.txt

Thank you.

Regards,

James

[#2]

It looks like the repository got corrupted.

http://stackoverflow.com/questions/9670341/copying-part-of-a-file-to-a-byte-array-in-java

Did you shutdown properly or just pull the plug?

[#3]

The RPi works fine. I unplugged RPi. Yes there are more logs in my "messages" system logs.

Stack Trace:

Dec 29 19:17:09 raspberrypi kernel: [ 5.677122] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready

Dec 29 19:17:09 raspberrypi kernel: [ 5.677131] brcmfmac: power management disabled

Dec 29 19:17:09 raspberrypi kernel: [ 5.783063] uart-pl011 3f201000.uart: no DMA platform data

Dec 29 19:17:09 raspberrypi kernel: [ 5.908412] random: nonblocking pool is initialized

Dec 29 19:17:09 raspberrypi kernel: [ 6.077762] Adding 102396k swap on /var/swap. Priority:-1 extents:2 across:217084k SSFS

Dec 29 19:17:09 raspberrypi kernel: [ 6.130497] cfg80211: Regulatory domain changed to country: GB

Dec 29 19:17:09 raspberrypi kernel: [ 6.130512] cfg80211: DFS Master region: ETSI

Dec 29 19:17:09 raspberrypi kernel: [ 6.130518] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp), (dfs_cac_time)

Dec 29 19:17:09 raspberrypi kernel: [ 6.130526] cfg80211: (2402000 KHz - 2482000 KHz @ 40000 KHz), (N/A, 2000 mBm), (N/A)

Dec 29 19:17:09 raspberrypi kernel: [ 6.130534] cfg80211: (5170000 KHz - 5250000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (N/A)

Dec 29 19:17:09 raspberrypi kernel: [ 6.130541] cfg80211: (5250000 KHz - 5330000 KHz @ 80000 KHz, 160000 KHz AUTO), (N/A, 2000 mBm), (0 s)

Dec 29 19:17:09 raspberrypi kernel: [ 6.130548] cfg80211: (5490000 KHz - 5710000 KHz @ 160000 KHz), (N/A, 2700 mBm), (0 s)

Dec 29 19:17:09 raspberrypi kernel: [ 6.130554] cfg80211: (57000000 KHz - 66000000 KHz @ 2160000 KHz), (N/A, 4000 mBm), (N/A)

Dec 29 19:17:09 raspberrypi kernel: [ 6.419695] smsc95xx 1-1.1:1.0 eth0: hardware isn't capable of remote wakeup

Dec 29 19:17:09 raspberrypi kernel: [ 6.419983] IPv6: ADDRCONF(NETDEV_UP): eth0: link is not ready

Dec 29 19:17:13 raspberrypi kernel: [ 10.197345] Bluetooth: Core ver 2.21

Dec 29 19:17:13 raspberrypi kernel: [ 10.197431] NET: Registered protocol family 31

Dec 29 19:17:13 raspberrypi kernel: [ 10.197442] Bluetooth: HCI device and connection manager initialized

Dec 29 19:17:13 raspberrypi kernel: [ 10.197464] Bluetooth: HCI socket layer initialized

Dec 29 19:17:13 raspberrypi kernel: [ 10.197481] Bluetooth: L2CAP socket layer initialized

Dec 29 19:17:13 raspberrypi kernel: [ 10.197538] Bluetooth: SCO socket layer initialized

Dec 29 19:17:13 raspberrypi kernel: [ 10.205632] Bluetooth: HCI UART driver ver 2.3

Dec 29 19:17:13 raspberrypi kernel: [ 10.205656] Bluetooth: HCI UART protocol H4 registered

Dec 29 19:17:13 raspberrypi kernel: [ 10.205668] Bluetooth: HCI UART protocol Three-wire (H5) registered

Dec 29 19:17:13 raspberrypi kernel: [ 10.205845] Bluetooth: HCI UART protocol BCM registered

Dec 29 19:17:13 raspberrypi kernel: [ 10.411407] Bluetooth: BNEP (Ethernet Emulation) ver 1.3

Dec 29 19:17:13 raspberrypi kernel: [ 10.411427] Bluetooth: BNEP filters: protocol multicast

Dec 29 19:17:13 raspberrypi kernel: [ 10.411452] Bluetooth: BNEP socket layer initialized

Dec 29 19:17:14 raspberrypi kernel: [ 10.694791] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

Dec 29 19:17:14 raspberrypi kernel: [ 10.696931] smsc95xx 1-1.1:1.0 eth0: link up, 100Mbps, full-duplex, lpa 0xCDE1

[#4]

are there any more logs? anything in /var/log/messages or other system logs.

Seems like there may be an OS issue or drive issue. Is the Rpi working otherwise?

[#5]

Logs look okay

can you post the nifi and minifi logs

see any GC files or other error files?

do a proper shutdown of minifi.

then minifi stop

bin/minifi.sh stop

then shutdown rpi

sudo shutdown -h now

when you restart save if it's working.

do you have the latest minifi

http://hortonworks.com/downloads/#dataflow

clean up your config.yml

[#6]

This is due to the underlying NiFi framework not properly handling when journals get corrupted due to system level failures (power loss) in the middle of writing. I created a ticket for this issue here: https://issues.apache.org/jira/browse/NIFI-3273

[#7]

Any chance you have a workaround to remove the corrupted journal or to reset nifi? We had a power outage and now i cant get nifi to even start anymore, and dont want to lose all the dev work i put into the flow processes

[#8]

try the new version. it's been working for me.

Upgrade. Use your backups of templates.

You can also read the flow xml backup