Community Articles

Find and share helpful community-sourced technical articles.
avatar
Master Guru

Tips

Read each processor directions carefully, inputs and outputs vary greatly from attributes, JSON, JSONPath statements and other entries. JSONPath supports a radically different syntax than NIFI Expression Language.

Know your Expression language, there are a lot of useful functions like UUID() and nextint().

6243-hdfsdisk.png

When you are doing a lot of streaming and testing data, you will find that logs grow huge on your Centos sandbox. I built up gigabytes of logs in Hive, Hadoop and NiFi in just a few days of operations. Check your /var/log every few days and watch the Ambari screens for data usage. When your data gets too large, things can start failing or slowing down due to a lack of temporary space, swap space and space for logs.

6242-logslogslogs.png

Delete Test Files For Ever!

hdfs dfs -rm -f -skipTrash /twitter/*.json

Clean up junk files.

cd /
du -hsx * | sort -rh | head -10 ) 

PutHiveQL requires hiveql.args.1.value and hiveql.args.1.type for all fields. If you are using EvaluateJSONPath, you cannot set your types there. Do those in an UpdateAttribute processor after that. Then you need to turn your code in SQL for Hive to execute.

6241-hiveargs.png

503 Views