I am using the Cloudera Data Science Workbench and I have the following problem with the Jobs API:
I have a homemade package that I want to import in a scheduled script, but the job fails because it cannot find the modules in my package.
Whenever I open a session using the project's engine base image and import my homemade package in a jupyter notebook or python file, the package loads, but a job within the same project cannot load the package.
I can't find a solution to this. Does it have to do with the job not running on the same engine image (it should, because the job and my other files in which the package loads, are in the same project)?