Support Questions
Find answers, ask questions, and share your expertise

How to change repo file template to skip proxy for local repository?

Highlighted

How to change repo file template to skip proxy for local repository?

Explorer

I am trying to install HDP 2.6 using Ambari 2.5.3.

I have setup local repository. I have to access internet through proxy server. However, I want to skip proxy for accessing local repository. I have done "no_proxy" setting in shell, -Dhttp.nonProxyHosts setting for ambari-server. Similarly, for yum repository, i have to include "proxy=_none". However, doing so manually isn't working since repos are overwritten by ambari.

I tried firing REST APIs to setup "desired_config" for "cluster-env" with modified "repo_suse_rhel_template" property. However, that does not seem to be working.

Logs:

2017-05-27 15:16:30,235 - Initializing 2 repositories

2017-05-27 15:16:30,236 - Repository['HDP-2.5'] {'base_url': 'http://xxx.net/hdp/centos7/HDP-2.5.3.0/', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP', 'mirror_list': None} 2017-05-27 15:16:30,244 - File['/etc/yum.repos.d/HDP.repo'] {'content': '[HDP-2.5]\nname=HDP-2.5\nbaseurl=http://xxx.net/hdp/centos7/HDP-2.5.3.0/\n\npath=/\nenabled=1\ngpgcheck=0'}

2017-05-27 15:16:30,244 - Writing File['/etc/yum.repos.d/HDP.repo'] because contents don't match

2017-05-27 15:16:30,245 - Repository['HDP-UTILS-1.1.0.21'] {'base_url': 'http://xxx.net/hdp/centos7/HDP-UTILS-1.1.0.21/', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}

2017-05-27 15:16:30,248 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': '[HDP-UTILS-1.1.0.21]\nname=HDP-UTILS-1.1.0.21\nbaseurl=http://xxx.net/hdp/centos7/HDP-UTILS-1.1.0.21/\n\npath=/\nenabled=1\ngpgcheck=0'}

2017-05-27 15:16:30,248 - Writing File['/etc/yum.repos.d/HDP-UTILS.repo'] because contents don't match

11 REPLIES 11
Highlighted

Re: How to change repo file template to skip proxy for local repository?

Super Mentor

If you want to specify 'proxy=_none_' (see there is a trailing _ underscore there) then try the following:

1. First take a backup of the cluster-env and then perform this option.

# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -s get <ambari-server hostname> <clustername> cluster-env > /tmp/cluster-env.bkp 

.

2. Now set the 'proxy=_none_' as following:

# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -s set <ambari-server hostname> <clustername> cluster-env repo_suse_rhel_template "[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0\nproxy=_none_" 

.

3. Once this done, you may do a "yum clean all" on all the hosts.

# yum clean all

.

- Additionally you can check the following file which also provides option to specify no proxy. 'proxy=_none_'

 # grep proxy /etc/yum.conf

.

Highlighted

Re: How to change repo file template to skip proxy for local repository?

Explorer

Thanks !

Is there a particular point I have to get the cluster configuration?

I mean should I do this right after ambari-server installation OR after a particular step in Cluster installation through Ambari ?

I ask this because I didn't get properties at all when i fired curl get request.

Highlighted

Re: How to change repo file template to skip proxy for local repository?

Super Mentor

- The previously shared "config.sh" script command is used to make configuration changes to an existing Ambari Server Cluster.

- Regarding your query "I didn't get properties at all when i fired curl get request."

>>> Can you please share the exact curl command that you made so that we can tell if it is supposed to return any output or not?

Highlighted

Re: How to change repo file template to skip proxy for local repository?

Explorer

Here is what I got :

curl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET http://xxx.net:8080/api/v1/clusters/MY_HDP/configurations?type=cluster-env

HTTP/1.1 200 OK

X-Frame-Options: DENY

X-XSS-Protection: 1; mode=block

X-Content-Type-Options: nosniff

Cache-Control: no-store

Pragma: no-cache

Set-Cookie: AMBARISESSIONID=1933cf5xbbrqwikunjhcc8otv;Path=/;HttpOnly

Expires: Thu, 01 Jan 1970 00:00:00 GMT

User: admin

Content-Type: text/plain Vary: Accept-Encoding, User-Agent Content-Length: 444

Server: Jetty(8.1.19.v20160209)

{ "href" : "http://xxx.net:8080/api/v1/clusters/AT_HDP/configurations?type=cluster-env", "items" : [ { "href" : "http://xxx.net:8080/api/v1/clusters/MY_HDP/configurations?type=cluster-env&tag=version1", "tag" : "version1", "type" : "cluster-env", "version" : 1, "Config" : { "cluster_name" : "MY_HDP", "stack_id" : "HDP-2.6" } } ]

Highlighted

Re: How to change repo file template to skip proxy for local repository?

Explorer

This command gives me properties :

/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin get xxx.net MY_HDP cluster-env

USERID=admin

PASSWORD=admin

########## Performing 'GET' on (Site:cluster-env, Tag:version1)

"properties" : { "agent_mounts_ignore_list" : "", "alerts_repeat_tolerance" : "1", "enable_external_ranger" : "false", "fetch_nonlocal_groups" : "true",

...

...

"repo_suse_rhel_template" : "[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0",

....

...

Highlighted

Re: How to change repo file template to skip proxy for local repository?

Super Mentor

In the config.sh command output we see that you have the repo details as following:

"repo_suse_rhel_template" : "[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0",

.

As previously mentioned you can use the same config.sh to push the 'proxy=_none_' changes back to the "cluster-env" as following:

# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -s set <ambari-server hostname> <clustername> cluster-env repo_suse_rhel_template "[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0\nproxy=_none_" 

.

Highlighted

Re: How to change repo file template to skip proxy for local repository?

Explorer

OK. I did it. I verified that settings were set to desired at the server. However, as I ran through Ambari UI after step 8 (Review), the settings were overwritten and original template was reapplied.

So in all this setting through REST was useless since it was overwritten again.

Back to square one :(

Any ideas ?

Highlighted

Re: How to change repo file template to skip proxy for local repository?

Explorer

I tried a workaround which seems to work.

As I mentioned, after step 8, installation failed.

I corrected the configuration again using CURL and did NOT restart services.

Then I hit the "retry failed" in Ambari-UI. This time modified repos got transfered and installation proceeded further.

This definitely appears to be a bug in Ambari-server.

Highlighted

Re: How to change repo file template to skip proxy for local repository?

Super Mentor

Is your installation complete now? Or are you still not able to start the cluster?