Error setting Cloudera Data Science Workbench on AWS EC2

[#1]

I have been trying to set up Cloudera data science workbench on AWS EC2 instances. I can successfully set up Cloudera Manager with services HDFS, Spark and YARN. But when I tried to install and run Cloudera data science workbench, I'm getting some errors. 

I'm sharing the details here: 

Cloudera Manager OS: CentOS 7.4

Other hosts OS: RHEL 7.6

CDSW gateway host: RHEL 7.5

The installation went well, but it shows bad health. It is an issue with the application. Its the result: 

I think its an issue with the DNS. As I'm not very good dealing with DNS, I doubt it may be an issue. But it also shows error in running Kubernetes. 

These are the errors that I noticed in the logs: 

WARNING:: Verification of iptables rules failed: 1

WARNING:: DNS doesn't resolve 10.0.0.87 to cdsw.company.io; DNS is not configured properly: 1

WARNING:: Kubernetes server is not running, version couldn't be checked.: 1

Is there anything that I'm doing wrong? Is there any other documentation that explains setting up CDSW on AWS? 

[#2]

Hello @neron ,

Based on the error messages, it looks like you have not flushed the iptables which is requirement for setting up CDSW. You can find more information here: 

https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_requirements_suppor...

Flush IPTables, Stop Firewalld, Reset weave, Restart host

1) Stop all the instances CM > CDSW > Instances > All > Stop
2) SSH to the master node and clear all iptable rules.

iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
iptables -t nat -F
iptables -t mangle -F
iptables -F
iptables -X

3) Stop firewalld service and disable it at boot time.
systemctl stop firewalld
systemctl disable firewalld

4) From CM selectively start only the Docker instance (this is important to carry out next step)

5) Submit the following command to reset weave:
# /opt/cloudera/parcels/CDSW/cni/bin/weave reset --force

6) Stop Docker role which we started in step 4.

7). Restart Host
# init 6

😎 Start Docker and Master roles to ensure if all POD's comes up fine. If yes, start the Application role.

I hope this will be helpful. 

[#3]

The CDSW showed good health after following these steps. But I think we have an issue with the DNS. After sometimes it is again going to bad health. 

This is the logs:

Pods not ready in cluster kube-system ['k8s-app/kube-dns', 'k8s-app/kube-proxy', 'component/kube-scheduler', 'name/weave-net', 'name/tiller'].

Pods not ready in cluster default ['role/cron', 'role/db', 'role/ds-operator', 'role/ds-reconciler', 'role/ds-vfs', 'role/ingress-controller', 'role/livelog', 'role/s2i-builder', 'role/s2i-client', 'role/s2i-git-server', 'role/s2i-queue', 'role/s2i-registry-auth', 'role/s2i-registry', 'role/s2i-server', 'role/secret-generator', 'role/tcp-ingress-controller', 'role/web', 'role/spark-port-forwarder', 'role/ds-cdh-client'].
Application services are incomplete. [web, db, livelog] not found.
Secrets are incomplete. [internal-secrets, external-secrets] not found.
Persistent volumes are incomplete.
Persistent volume claims are incomplete.
Ingresses are incomplete.
Checking web at url: http://cdsw.company.io
Web is not yet up.
Cloudera Data Science Workbench is not ready yet

For setting up DNS wildcards, I have configured the domain to CDSW's public IP (Elastic IP in AWS). Is that correct? Because we can't use private IPs in the subdomain, right? 

What is this line in the Cloudera document means: The wildcard DNS hostname configured for Cloudera Data Science Workbench must now be resolvable from both, the CDSW cluster, and your browser. 

[#4]

@neron Yes, Public IP should be fine. 

The wildcard DNS hostname configured for Cloudera Data Science Workbench must now be resolvable from both, the CDSW cluster, and your browser. 

This means The wildcard DNS resolution needs to work on both the CDSW nodes and on your end user machine. If you configured it correctly from CDSW master/worker, but fails from your laptop/PC/end user machine, this could be an issue.

[#5]

@neron If that resolves your issue, please spare some time in accepting the solution. Thanks.