Welcome to the Cloudera Community

Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Who agreed with this topic

engine tls error

avatar
Explorer

Good day,

got a few issues with cdsw engine.

CDSW is behind corporate proxy. Corporate proxy uses self-signed certificate.

 

I've added corporate root CA to CDSW master (master has no issues with TLS).

 

Standard cdsw engine fails with tls erros. These tools aren't working - wget, curl, conda, pip, R (install.packages()), apt-get.

 

I've created a modified cdsw engine container. This container has config files to ignore TLS:

/etc/pip.conf

/etc/wgetrc

/etc/R/Renviron.site

 

/etc/curlrc and ~/.curlrc with "insecure" instruction is not honored, curl works only with -k option.

 

I've also tried to push corporate root CAs to cdsw engine container and issue "update-ca-roots --fresh". I see root CAs listed in /etc/ssl/certs/ca-certificates.crt, inside my custom cdsw engine. But still get TLS errors in tools mentioned above.

 

Could it be some kind of "clash" between corporate proxy and Traefik?

Or maybe too many redirects which causes requests python lib fails?

Am I correct to assume that cdsw user works in a conda virtual environment by default?

Is there an open repo with CDSW pods/engine configuration?

 

 

 

Conda errors:

1. Without condarc

cdsw@hdztfwk4fj9sm7xe:~$ conda update conda
PackageNotInstalledError: Package is not installed in prefix.
prefix: /home/cdsw/.conda/envs/python2.7
package name: conda

2. Without condarc

cdsw@hdztfwk4fj9sm7xe:~$ sudo conda update conda
sudo: conda: command not found

 

3.

cdsw@hdztfwk4fj9sm7xe:~$ conda install nltk
Fetching package metadata ...
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.continuum.io/pkgs/main/linux-64/repodata.json.bz2>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
SSLError(SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),),)


4. With ~/.condarc
#cat ~/.condarc
proxy_servers:
http: http://proxy:8080
https: http://proxy:8080
ssl_verify: False

cdsw@hdztfwk4fj9sm7xe:~$ conda install nltk
Fetching package metadata ...An unexpected error has occurred.
Please consider posting the following information to the
conda GitHub issue tracker at:
https://github.com/conda/conda/issues
Current conda install:
platform : linux-64
conda version : 4.3.30
conda is private : False
conda-env version : 4.3.30
conda-build version : not installed
python version : 2.7.13.final.0
requests version : 2.12.4
root environment : /opt/conda (read only)
default environment : /home/cdsw/.conda/envs/python2.7
envs directories : /home/cdsw/.conda/envs
/opt/conda/envs
package cache : /opt/conda/pkgs
/home/cdsw/.conda/pkgs
channel URLs : https://repo.continuum.io/pkgs/main/linux-64
https://repo.continuum.io/pkgs/main/noarch
https://repo.continuum.io/pkgs/free/linux-64
https://repo.continuum.io/pkgs/free/noarch
https://repo.continuum.io/pkgs/r/linux-64
https://repo.continuum.io/pkgs/r/noarch
https://repo.continuum.io/pkgs/pro/linux-64
https://repo.continuum.io/pkgs/pro/noarch
config file : /home/cdsw/.condarc
netrc file : None
offline mode : False
user-agent : conda/4.3.30 requests/2.12.4 CPython/2.7.13 Linux/3.10.0-693.el7.x86_64 debian/stretch/sid glibc/2.23 
UID:GID : 8536:8536
`$ /opt/conda/bin/conda install nltk`
Traceback (most recent call last):
File "/opt/conda/lib/python2.7/site-packages/conda/exceptions.py", line 640, in conda_exception_handler
return_value = func(*args, **kwargs)
File "/opt/conda/lib/python2.7/site-packages/conda/cli/main.py", line 140, in _main
exit_code = args.func(args, p)
File "/opt/conda/lib/python2.7/site-packages/conda/cli/main_install.py", line 80, in execute
install(args, parser, 'install')
File "/opt/conda/lib/python2.7/site-packages/conda/cli/install.py", line 231, in install
unknown=index_args['unknown'], prefix=prefix)
File "/opt/conda/lib/python2.7/site-packages/conda/core/index.py", line 101, in get_index
index = fetch_index(channel_priority_map, use_cache=use_cache)
File "/opt/conda/lib/python2.7/site-packages/conda/core/index.py", line 120, in fetch_index
repodatas = collect_all_repodata(use_cache, tasks)
File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 75, in collect_all_repodata
repodatas = _collect_repodatas_serial(use_cache, tasks)
File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 485, in _collect_repodatas_serial
for url, schan, pri in tasks]
File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 115, in func
res = f(*args, **kwargs)
File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 464, in fetch_repodata
mod_etag_headers.get('_mod'))
File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 156, in fetch_repodata_remote_request
json_str = maybe_decompress(filename, resp.content)
File "/opt/conda/lib/python2.7/site-packages/conda/core/repodata.py", line 154, in maybe_decompress
if filename.endswith('.bz2')
IOError: invalid data stream

 

5. With /etc/condarc
#cat /etc/condarc
proxy_servers:
http: http://proxy:8080
https: http://proxy:8080
ssl_verify: False

 

cdsw@hdztfwk4fj9sm7xe:~$ conda install nltk
Fetching package metadata ...
CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.continuum.io/pkgs/main/linux-64/repodata.json.bz2>
Elapsed: -
An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
SSLError(SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),),)

 

 

P.S. Feature request - sudo by default in a default engine.

Who agreed with this topic