Issues using newer versions of python packages such as mlflow in CDSW runtime images

[#1]

Issue

During the installation of the latest version of mlflow for the custom runtime engine in CDSW 1.10.x, it's still using cmladdon’s version rather than the version installed via pip.

For example, in one case, the error recorded was the following:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflict mlflow-cml-plugin 0.0.1 requires protobuf==3.19,4, but you have protobuf 4.24.0 which is incompatible. 
onnx 1.12.0 requires protobuf<=3.20.1,>=3.12.2, but you have protobuf 4.24.0 which is incompatible.

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflict mlflow-cml-plugin 0.0.1 requires protobuf==3.19,4, but you have protobuf 4.24.0 which is incompatible. 
onnx 1.12.0 requires protobuf<=3.20.1,>=3.12.2, but you have protobuf 4.24.0 which is incompatible.

Cause

The cmladdon feature was introduced in CDSW 1.10.0. And its content has changed in 1.10.1.
This cmladdon feature ships some Python libraries and is mounted in all workloads. We intend to do that for the user's convenience, i.e., to have some packages installed out of the box, such as CML or mlflow.

Within workloads, this folder is mounted as read-only. The same folder is mounted on all workloads.

We know that packages we deliver via this cmladdon feature take precedence over pretty much everything else ( packages installed to the runtime image, packages installed by the user).

Solution:

If the user wants to install and use a newer version of the above-mentioned typing-extensions package, the user can use a workaround. Any of the below steps will remove the cmladdon folder from workloads, which includes all packages coming from cmladdon:

1. Define the PYTHONPATH env variable on a workspace level (Site Administration > Runtime/Engine) or in each project, (Project Settings > Advanced) and give it any value ( empty value is also acceptable). This will override our code that injects the cmladdon folder onto the PYTHONPATH.
2. SSH onto the VFS pod. From there one can delete the content of cmladdon, as the VFS pod has to write access to this folder. This workaround has to be repeated if the cluster is ever upgraded to a newer CDSW version.
3. At the start of every Python workload, modify the list sys.path such the entry corresponding to cmladdon is removed or moved to the end of the list. The latter case will keep the packages in the syspath, but user-installed packages will take precedence.