How to import Numpy and other libraries when using Zeppelin with Livy3?

[#1]

Hi everyone,

Currently trying to create a pySpark application that requires Numpy.
I have tried to install it on every worker and master nodes but without success since the notebook always returns the error that it does not exist.

Had anyone done this? I believe this should be something rather easy, but my experience is probably not enough to get it at this point.

Thanks

[#2]

If you're using Conda

Create the environment
conda create -n pyspark_env python=3.9 numpy

Activate it
conda activate pyspark_env

Tell Spark to use it

export PYSPARK_PYTHON=$(which python)
export PYSPARK_DRIVER_PYTHON=$(which python)