question Re: Connecting to hdfs/creating the sparksession from cdsw in Support Questions

Connecting to hdfs/creating the sparksession from cdsw

t5 — Fri, 16 Sep 2022 13:53:52 GMT

Hi Team,

I have installed the cdsw successfully but when I was trying to run the hdfs command/trying create the sparksession from the cdsw terminal then I am getting the below error. Any idea/suggestion what exactly i am missing here from set up of point of view? Thanks in Advance!!!

Error:

hdfs dfs -put data/sample_text_file.txt /tmp clouderamaster.<domain>.com
-put: java.net.UnknownHostException:

clouderamaster.<domain>.com: Is my cdh master server.

cdsw.<domain>.com: is the cdsw master.(from where I am running the hdfs /sparksession from the interactive command prompt/terminal)

Re: Connecting to hdfs/creating the sparksession from cdsw

t5 — Fri, 16 Nov 2018 04:01:53 GMT

Also, just to add one more point i am unable to ping to clouderamaster from the cdsw master host in the cdsw terminal. Any suggestion guys what exactly i am missing here?

Re: Connecting to hdfs/creating the sparksession from cdsw

t5 — Sat, 17 Nov 2018 17:07:49 GMT

@cjervis Sorry to tag you on the above query. Will you able to guide me where i am missing on the above issue?

Re: Connecting to hdfs/creating the sparksession from cdsw

cjervis — Sun, 18 Nov 2018 13:12:52 GMT

Unfortunately I do not have experience with CDSW but perhaps @tristanzajonc or @peter_ableda can be of assistance.

Re: Connecting to hdfs/creating the sparksession from cdsw

t5 — Sun, 18 Nov 2018 15:06:40 GMT

@cjervis Thanks for your quick response.

Re: Connecting to hdfs/creating the sparksession from cdsw

peter_ableda — Mon, 19 Nov 2018 08:45:02 GMT

Hi,

You seem to have some network connectivity issue. I have seen many variations for this, please check

- that you don't have a firewall between the machines,

- you resolve the hosts with DNS and not /etc/hosts

https://www.cloudera.com/documentation/data-science-workbench/1-3-x/topics/cdsw_known_issues.html#networking

- DNS can do forward/reverse resolution on your master hostname/ip

I think if you are ok on the above, this should work.

Regards,

Peter

Re: Connecting to hdfs/creating the sparksession from cdsw

t5 — Mon, 19 Nov 2018 11:54:54 GMT

@peter_ableda Thanks for your response.

Step 1: I have checked and could find we dont have any firewall between my machines.

Step 2: DNS can do forward/reverse resolution on your master hostname/ip> Yes its working for cdswmaster host where I have installed cdsw(added to the dns entry as below which resolves forward as well as backword resolution)

Step 3: I have added 2 DNS entries as below for the cdswmaster host(cdswmaster.lab.test.com)

*.cdswmaster.lab.test.com IN A IP

cdswmaster.lab.test.com IN A IP

Little confused here do i need to add the dns names as above or do i need to change as cdsw(cdsw.lab.test.com) from cdswmaster as per the documentation.

Step 4: Checked the terminal from cdsw web ui and i am unable to ping to clouderamaster.lab.test.com(where cloudera manager was install). But I am able to ping to the terminal ip from the clouderamaster.lab.test.com.

I believe here some set up should be corrected inorder to ping the clouderamaster host correctly. Please advise

Also, do you see anyother set up i need to correct from my end to get going in cdsw. Sorry for little more demanding on the same as i find cdsw is little more complex interms of the set up.

Re: Connecting to hdfs/creating the sparksession from cdsw

peter_ableda — Mon, 19 Nov 2018 11:59:47 GMT

You need to make sure that forward/reverse DNS resolution works from the CDSW terminal to host where you have the YARN ResourceManager and HDFS NameNode services. You referred to this as clouderamaster.<domain>.com before.

This issue is not related to the CDSW master DNS resolution, you mentioned that you are using the session terminal, as it works, the CDSW master DNS is configured properly.

Regards,

Peter

Re: Connecting to hdfs/creating the sparksession from cdsw

t5 — Mon, 19 Nov 2018 16:49:41 GMT

@peter_ableda Thanks Peter for your detailed explanation and your valuable time. Will check how I can access the CDH master node(YARN/HDFS Name Node) from my cdsw terminal.Not sure if the HTTP proxy set up will help or not.

Re: Connecting to hdfs/creating the sparksession from cdsw

t5 — Tue, 20 Nov 2018 10:12:49 GMT

@peter_ableda One Final Question. In my cdsw terminal the Ip addess was different(pod ip not the cdsw host ip) and that was the reason i am unable to connect to the master hostname(clouderamaster.<domain>.com) but i am able to ping using the master host ip from cdsw terminal.

Also, I have not done hadoop Authentication in the cdsw admin web ui. Do you think its must to have access to the cluster and from where I can find the Principal/username and pwd/keytab to get access to the cluster.

Re: Connecting to hdfs/creating the sparksession from cdsw

peter_ableda — Tue, 20 Nov 2018 10:09:04 GMT

Hi,

We have an overlay network on top of your CDSW hosts where the pods are getting their IPs from (100.66.x.x).

Based on your description it seems that DNS resolution is not working from inside the container while it works on the host. This can happen when multiple nameservers are configured in /etc/resolv.conf but some of them can't resolve your clouderamaster. You could figure out what nameserver can resolve your host and drop the rest of them or make sure that all nameservers can resolve the clouderamaster.

I like to use `dig @nameserver clouderamaster.com` command to test these.

Regards,

Peter

Re: Connecting to hdfs/creating the sparksession from cdsw

t5 — Fri, 23 Nov 2018 05:32:27 GMT

@peter_ableda Hi Peter, When we say we need to add the dns entry details of the master host. Are we trying to say we need to add the dns entry of the clouderamaster host dns entry or the cdswmaster dns entry?

As of now I have added the dns entry of the cdsw master host. Also, we need to add the xtra dot(.) after the hostname as per the documentation(*.cdsw.lab.test.com./cdsw.lab.test.com.)? Sorry I am little confused with the docs.

Re: Connecting to hdfs/creating the sparksession from cdsw

peter_ableda — Fri, 23 Nov 2018 11:13:44 GMT

The original issue you reported was an UnknownHostException on the clouderamaster.

hdfs dfs -put data/sample_text_file.txt /tmp clouderamaster.<domain>.com
-put: java.net.UnknownHostException:

You need to make sure that this host can be resolved (both forward/reverse) from inside a CDSW session via DNS.

As you can start a CDSW session and interact with it, you already configured the DNS entry for the CDSW master properly.

Regards,

Peter

Re: Connecting to hdfs/creating the sparksession from cdsw

t5 — Fri, 23 Nov 2018 11:36:39 GMT

@peter_ableda You need to make sure that this host can be resolved (both forward/reverse) from inside a CDSW session via DNS. Is that means we need add another dns entry for the CDH master host(clouderamaster.lab.test.com) so that it can be accessiable from cdsw master host?

Re: Connecting to hdfs/creating the sparksession from cdsw

peter_ableda — Fri, 23 Nov 2018 11:38:00 GMT

Yes.

Re: Connecting to hdfs/creating the sparksession from cdsw

t5 — Mon, 26 Nov 2018 11:53:44 GMT

@peter_ableda Thanks Peter. Now I am able to submit the spark job from cdsw master. Does cloudera provide the user level isoloation when they access to the cdsw project/content as different user can distrub /edit the same content?

Re: Connecting to hdfs/creating the sparksession from cdsw

peter_ableda — Mon, 26 Nov 2018 12:08:52 GMT

We have a collaboration page in the documentation:

https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_collaborate.html

We also have a page about Kerberos authentication:

https://www.cloudera.com/documentation/data-science-workbench/latest/topics/cdsw_kerberos.html

I hope this answers your question.

Regards,

Peter

Re: Connecting to hdfs/creating the sparksession from cdsw

t5 — Mon, 26 Nov 2018 16:50:19 GMT

@peter_ableda Thanks Peter for your time and have a grt day!