Support Questions

Find answers, ask questions, and share your expertise
Announcements
Check out our newest addition to the community, the Cloudera Data Analytics (CDA) group hub.

How to change repo file template to skip proxy for local repository?

Explorer

I am trying to install HDP 2.6 using Ambari 2.5.3.

I have setup local repository. I have to access internet through proxy server. However, I want to skip proxy for accessing local repository. I have done "no_proxy" setting in shell, -Dhttp.nonProxyHosts setting for ambari-server. Similarly, for yum repository, i have to include "proxy=_none". However, doing so manually isn't working since repos are overwritten by ambari.

I tried firing REST APIs to setup "desired_config" for "cluster-env" with modified "repo_suse_rhel_template" property. However, that does not seem to be working.

Logs:

2017-05-27 15:16:30,235 - Initializing 2 repositories

2017-05-27 15:16:30,236 - Repository['HDP-2.5'] {'base_url': 'http://xxx.net/hdp/centos7/HDP-2.5.3.0/', 'action': ['create'], 'components': [u'HDP', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP', 'mirror_list': None} 2017-05-27 15:16:30,244 - File['/etc/yum.repos.d/HDP.repo'] {'content': '[HDP-2.5]\nname=HDP-2.5\nbaseurl=http://xxx.net/hdp/centos7/HDP-2.5.3.0/\n\npath=/\nenabled=1\ngpgcheck=0'}

2017-05-27 15:16:30,244 - Writing File['/etc/yum.repos.d/HDP.repo'] because contents don't match

2017-05-27 15:16:30,245 - Repository['HDP-UTILS-1.1.0.21'] {'base_url': 'http://xxx.net/hdp/centos7/HDP-UTILS-1.1.0.21/', 'action': ['create'], 'components': [u'HDP-UTILS', 'main'], 'repo_template': '[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0', 'repo_file_name': 'HDP-UTILS', 'mirror_list': None}

2017-05-27 15:16:30,248 - File['/etc/yum.repos.d/HDP-UTILS.repo'] {'content': '[HDP-UTILS-1.1.0.21]\nname=HDP-UTILS-1.1.0.21\nbaseurl=http://xxx.net/hdp/centos7/HDP-UTILS-1.1.0.21/\n\npath=/\nenabled=1\ngpgcheck=0'}

2017-05-27 15:16:30,248 - Writing File['/etc/yum.repos.d/HDP-UTILS.repo'] because contents don't match

11 REPLIES 11

Super Mentor

If you want to specify 'proxy=_none_' (see there is a trailing _ underscore there) then try the following:

1. First take a backup of the cluster-env and then perform this option.

# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -s get <ambari-server hostname> <clustername> cluster-env > /tmp/cluster-env.bkp 

.

2. Now set the 'proxy=_none_' as following:

# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -s set <ambari-server hostname> <clustername> cluster-env repo_suse_rhel_template "[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0\nproxy=_none_" 

.

3. Once this done, you may do a "yum clean all" on all the hosts.

# yum clean all

.

- Additionally you can check the following file which also provides option to specify no proxy. 'proxy=_none_'

 # grep proxy /etc/yum.conf

.

Explorer

Thanks !

Is there a particular point I have to get the cluster configuration?

I mean should I do this right after ambari-server installation OR after a particular step in Cluster installation through Ambari ?

I ask this because I didn't get properties at all when i fired curl get request.

Super Mentor

- The previously shared "config.sh" script command is used to make configuration changes to an existing Ambari Server Cluster.

- Regarding your query "I didn't get properties at all when i fired curl get request."

>>> Can you please share the exact curl command that you made so that we can tell if it is supposed to return any output or not?

Explorer

Here is what I got :

curl --user admin:admin -i -H 'X-Requested-By: ambari' -X GET http://xxx.net:8080/api/v1/clusters/MY_HDP/configurations?type=cluster-env

HTTP/1.1 200 OK

X-Frame-Options: DENY

X-XSS-Protection: 1; mode=block

X-Content-Type-Options: nosniff

Cache-Control: no-store

Pragma: no-cache

Set-Cookie: AMBARISESSIONID=1933cf5xbbrqwikunjhcc8otv;Path=/;HttpOnly

Expires: Thu, 01 Jan 1970 00:00:00 GMT

User: admin

Content-Type: text/plain Vary: Accept-Encoding, User-Agent Content-Length: 444

Server: Jetty(8.1.19.v20160209)

{ "href" : "http://xxx.net:8080/api/v1/clusters/AT_HDP/configurations?type=cluster-env", "items" : [ { "href" : "http://xxx.net:8080/api/v1/clusters/MY_HDP/configurations?type=cluster-env&tag=version1", "tag" : "version1", "type" : "cluster-env", "version" : 1, "Config" : { "cluster_name" : "MY_HDP", "stack_id" : "HDP-2.6" } } ]

Explorer

This command gives me properties :

/var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin get xxx.net MY_HDP cluster-env

USERID=admin

PASSWORD=admin

########## Performing 'GET' on (Site:cluster-env, Tag:version1)

"properties" : { "agent_mounts_ignore_list" : "", "alerts_repeat_tolerance" : "1", "enable_external_ranger" : "false", "fetch_nonlocal_groups" : "true",

...

...

"repo_suse_rhel_template" : "[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0",

....

...

Super Mentor

In the config.sh command output we see that you have the repo details as following:

"repo_suse_rhel_template" : "[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0",

.

As previously mentioned you can use the same config.sh to push the 'proxy=_none_' changes back to the "cluster-env" as following:

# /var/lib/ambari-server/resources/scripts/configs.sh -u admin -p admin -s set <ambari-server hostname> <clustername> cluster-env repo_suse_rhel_template "[{{repo_id}}]\nname={{repo_id}}\n{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}\n\npath=/\nenabled=1\ngpgcheck=0\nproxy=_none_" 

.

Explorer

OK. I did it. I verified that settings were set to desired at the server. However, as I ran through Ambari UI after step 8 (Review), the settings were overwritten and original template was reapplied.

So in all this setting through REST was useless since it was overwritten again.

Back to square one 😞

Any ideas ?

Explorer

I tried a workaround which seems to work.

As I mentioned, after step 8, installation failed.

I corrected the configuration again using CURL and did NOT restart services.

Then I hit the "retry failed" in Ambari-UI. This time modified repos got transfered and installation proceeded further.

This definitely appears to be a bug in Ambari-server.

Super Mentor

Is your installation complete now? Or are you still not able to start the cluster?

Explorer

It failed. (I think it timed out because its taking too much time for each step.)

However, install is progressing on retry. Earlier it used to fail on first component install. Now its passing the steps. I verified the transferred repos on slaves are having correct proxy setting.

Explorer

FYI, for those who want to use the (non-deprecated) python:

/var/lib/ambari-server/resources/scripts/configs.py -u admin -p admin -a set -l <ambari-server hostname> -n <clustername> -c cluster-env -k repo_suse_rhel_template -v "[{{repo_id}}]
name={{repo_id}}
{% if mirror_list %}mirrorlist={{mirror_list}}{% else %}baseurl={{base_url}}{% endif %}
path=/
enabled=1
gpgcheck=0
proxy=_none_"
Take a Tour of the Community
Don't have an account?
Your experience may be limited. Sign in to explore more.