28726
DISCUSSIONS
101724
MEMBERS
3157
ARTICLES
Created on 05-07-2018 05:36 PM - edited 09-16-2022 06:11 AM
Good day,
got a few issues with cdsw engine.
CDSW is behind corporate proxy. Corporate proxy uses self-signed certificate.
I've added corporate root CA to CDSW master (master has no issues with TLS).
Standard cdsw engine fails with tls erros. These tools aren't working - wget, curl, conda, pip, R (install.packages()), apt-get.
I've created a modified cdsw engine container. This container has config files to ignore TLS:
/etc/pip.conf
/etc/wgetrc
/etc/R/Renviron.site
/etc/curlrc and ~/.curlrc with "insecure" instruction is not honored, curl works only with -k option.
I've also tried to push corporate root CAs to cdsw engine container and issue "update-ca-roots --fresh". I see root CAs listed in /etc/ssl/certs/ca-certificates.crt, inside my custom cdsw engine. But still get TLS errors in tools mentioned above.
Could it be some kind of "clash" between corporate proxy and Traefik?
Or maybe too many redirects which causes requests python lib fails?
Am I correct to assume that cdsw user works in a conda virtual environment by default?
Is there an open repo with CDSW pods/engine configuration?
Conda errors:
1. Without condarc
cdsw@hdztfwk4fj9sm7xe:~$ conda update conda PackageNotInstalledError: Package is not installed in prefix. prefix: /home/cdsw/.conda/envs/python2.7 package name: conda
2. Without condarc
cdsw@hdztfwk4fj9sm7xe:~$ sudo conda update conda sudo: conda: command not found
3.
cdsw@hdztfwk4fj9sm7xe:~$ conda install nltk Fetching package metadata ... CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.continuum.io/pkgs/main/linux-64/repodata.json.bz2> Elapsed: - An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. SSLError(SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),),)
4. With ~/.condarc
#cat ~/.condarc
proxy_servers:
http: http://proxy:8080
https: http://proxy:8080
ssl_verify: False
cdsw@hdztfwk4fj9sm7xe:~$ conda install nltk Fetching package metadata ...An unexpected error has occurred. Please consider posting the following information to the conda GitHub issue tracker at: https://github.com/conda/conda/issues Current conda install: platform : linux-64 conda version : 4.3.30 conda is private : False conda-env version : 4.3.30 conda-build version : not installed python version : 2.7.13.final.0 requests version : 2.12.4 root environment : /opt/conda (read only) default environment : /home/cdsw/.conda/envs/python2.7 envs directories : /home/cdsw/.conda/envs /opt/conda/envs package cache : /opt/conda/pkgs /home/cdsw/.conda/pkgs channel URLs : https://repo.continuum.io/pkgs/main/linux-64 https://repo.continuum.io/pkgs/main/noarch https://repo.continuum.io/pkgs/free/linux-64 https://repo.continuum.io/pkgs/free/noarch https://repo.continuum.io/pkgs/r/linux-64 https://repo.continuum.io/pkgs/r/noarch https://repo.continuum.io/pkgs/pro/linux-64 https://repo.continuum.io/pkgs/pro/noarch config file : /home/cdsw/.condarc netrc file : None offline mode : False user-agent : conda/4.3.30 requests/2.12.4 CPython/2.7.13 Linux/3.10.0-693.el7.x86_64 debian/stretch/sid glibc/2.23 UID:GID : 8536:8536 `$ /opt/conda/bin/conda install nltk` Traceback (most recent call last): File "/opt/conda/lib/python2.7/site-packages/conda/exceptions.py", line 640, in conda_exception_handler return_value = func(*args, **kwargs) File "/opt/conda/lib/python2.7/site-packages/conda/cli/main.py", line 140, in _main exit_code = args.func(args, p) File "/opt/conda/lib/python2.7/site-packages/conda/cli/main_install.py", line 80, in execute install(args, parser, 'install') File "/opt/conda/lib/python2.7/site-packages/conda/cli/install.py", line 231, in install unknown=index_args['unknown'], prefix=prefix) File "/opt/conda/lib/python2.7/site-packages/conda/core/index.py", line 101, in get_index index = fetch_index(channel_priority_map, use_cache=use_cache) File "/opt/conda/lib/python2.7/site-packages/conda/core/index.py", line 120, in fetch_index repodatas = collect_all_repodata(use_cache, tasks) File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 75, in collect_all_repodata repodatas = _collect_repodatas_serial(use_cache, tasks) File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 485, in _collect_repodatas_serial for url, schan, pri in tasks] File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 115, in func res = f(*args, **kwargs) File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 464, in fetch_repodata mod_etag_headers.get('_mod')) File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 156, in fetch_repodata_remote_request json_str = maybe_decompress(filename, resp.content) File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 154, in maybe_decompress if filename.endswith('.bz2') IOError: invalid data stream
5. With /etc/condarc
#cat /etc/condarc
proxy_servers:
http: http://proxy:8080
https: http://proxy:8080
ssl_verify: False
cdsw@hdztfwk4fj9sm7xe:~$ conda install nltk Fetching package metadata ... CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.continuum.io/pkgs/main/linux-64/repodata.json.bz2> Elapsed: - An HTTP error occurred when trying to retrieve this URL. HTTP errors are often intermittent, and a simple retry will get you on your way. SSLError(SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),),)
P.S. Feature request - sudo by default in a default engine.