Overview - ExecuteProcess   When developing custom logic as part of our dataflow, having the ability to execute a custom script is a boom especially when it comes to compensating for certain capabilities which are not present out of the box or may not be the need of the hour now for the larger audience. :package:   NiFi makes this possible by providing a processor called ExecuteProcess which enables you to execute custom code. It leverages the underlying operating system to execute the command/process.   Background Over the course of using this processor I often had trouble with managing different modules that were required by different ExecuteProcess processors within my flow. At times I was looking at having different versions of the same modules for different set of scripts. Virtual environments in python came to my rescue here. Hence as a best practise it is always advised to use virtual environments to setup additional modules while using Python code within the ExecuteProcess processor.   Let me now show you how can this be achieved. For the subsequent steps, these are the different versions I am using:   Python Version : 3.6.8 pip Version : 21.3.1 NiFi Version : 1.18.0.2.2.6.0-260 Python Module : Faker   Process The whole process is divided into two stages: Changes on the system where NiFi is installed(Creation of the virtual env) Configuring the ExecuteProcess processor  Stage 1 : Creation of the virtual env         Step 1 : Execute the following command to create a virtual env                   virtualenv <env-name-without-brackets>                   Eg: virtualenv env           *On executing this a directory with the environment name gets created(the name in our case is env) in the current directory                            Step 2 : Activate the Environment and install the package using the following commands                     source env/bin/activate                Note : Once the virtual environment is activated you should see you environment name at the start of the command prompt.           Step 3 : Install the module using the following command                            pip install faker           Step 4 : Once the installation completes, create a shell script(test.sh) with the following content                                  #!/bin/bash                                   source /path/to/your-environment/env/bin/activate                                   python /path/to/your/script/test.py             My script folder looks something like this:                       Important : Make sure the script and the virtual environment folder are accessible by the user which is running NiFi.   Stage 2 : Configuring the ExecuteProcess processor             Step 1 : In NiFi, configure the processor in the following way Your processor will now be able to import the module that you have setup within your python virtual environment     ... View more