Installing HDP using ambari blueprints

[#1]

Hi,

I installed 2 servers with HDP 2.4. One server I installed with the wizard, and the other with blueprint.

in the cluster_configuration.json file, I've set: "stack_version":"2.4".

After the installation completed successfully, I noticed that the two servers have a slight different HDP version.

In the server that was installed with the wizard, I see:

# ls -l /etc/hadoop/
total 8
drwxr-xr-x 3 root root 4096 Jan  7 19:08 2.4.3.0-227
lrwxrwxrwx 1 root root   35 Jan  7 19:08 conf -> /usr/hdp/current/hadoop-client/conf
drwxr-xr-x 2 root root 4096 Jan  7 19:08 conf.backup

while in he server that installed with the blueprint I see:

# ls -l /etc/hadoop/
total 4
drwxr-xr-x. 3 root root   15 Feb 15 10:59 2.4.0.0-169
lrwxrwxrwx. 1 root root   35 Feb 15 10:59 conf -> /usr/hdp/current/hadoop-client/conf
drwxr-xr-x. 2 root root 4096 Feb 15 10:58 conf.backup

What is the reason for it?

Does it makes any difference?

How can I install HDP 2.4.3 instead of 2.4.0 using the blueprints method ?

[#2]

When you install blueprint you will need to register a vdf file https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-release-notes/content/ambari_reln... so whatever version u have in there for the repos will be the one which will be installed.

[#3]

@Yuval Smp

If you want to register any specific version of HDP version using blueprint then please download the repo file of that version and then register it to Ambari before running the hostmapping blueprint action.

Example:

# curl -H "X-Requested-By: ambari" -X PUT -u admin:admin http://$AMBARI_HOST:8080/api/v1/stacks/HDP/versions/2.4/operating_systems/redhat7/repositories/HDP-2... -d @repo.json

# curl -H "X-Requested-By: ambari" -X PUT -u admin:admin http://$AMBARI_HOST:8080/api/v1/stacks/HDP/versions/2.4/operating_systems/redhat7/repositories/HDP-U... -d @utils.json

.

Here in your case if you want to install HDP 2.4.3.0 then the file will look like following:

repo.json

{  
   "Repositories":{  
      "base_url":"http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.4.3.0",
      "verify_base_url":true
   }
}

.

utils.json

{
  "Repositories": {
    "base_url": "http://public-repo-1.hortonworks.com/HDP-UTILS-1.1.0.20/repos/centos7",
    "verify_base_url": true
  }
}

.

You can find the HDP 2.4.3 repository here: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.3.0/bk_ambari-installation/content/hdp_24_repos...

Is there any specific reason you are planning to install HDP 2.4 ? (HDP 2.6.4 is the latst and has many new features and enhancements). For later version of ambari you can also use the VDF (Version Definition File) concept to register a new version. For that you will have to obtain the VDF.

For example if you are planning to install HDP 2.5 then you casn get the VDF downloaded from here:
https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-installation/content/hdp_25_repos...

For HDP 2.4 VDF is not available here:
https://docs.hortonworks.com/HDPDocuments/Ambari-2.5.0.3/bk_ambari-installation/content/hdp_24_repos...

.

[#4]

Thanks @Jay Kumar SenSharma.

I'm using HDP 2.4.3 as I'm trying to compare benchmark results using my environment to some other environment.

I downloaded the repo file:

wget -nv http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.4.3.0/hdp.repo -O /etc/yum.repos.d/hdp.repo 

When I tried the CURLs above. I received:

# curl -H "X-Requested-By: ambari" -X PUT -u admin:admin http://localhost:8080/api/v1/stacks/HDP/versions/2.4.3/operating_systems/redhat7/repositories/HDP-2.... -d @repo.json
{
  "status" : 400,
  "message" : "java.lang.IllegalArgumentException: repo_name is required to verify repository"
}[root@hdptst56 ~]#

I tried adding "repo_name" with various names (including "HDP Version - HDP-2.4.3.0" which is the exact name I see on "yum repolist". In all my tries, HDP2.4.0 was installed.

What should I fix ?

[#5]

@Yuval Smp

My Bad in the previously mentioned repo.json file i forgot to add the "repo_name" attribute. Please add the "repo_name" in both the repo files.

Example: for hdp.repo

{
  "Repositories": {
    "repo_name": "HDP-2.4.3.0",
    "base_url": "http://public-repo-1.hortonworks.com/HDP/centos7/2.x/updates/2.4.3.0",
    "verify_base_url": true
  }
}

.

Then try:

# curl -H "X-Requested-By: ambari" -X PUT -u admin:admin http://localhost:8080/api/v1/stacks/HDP/versions/2.4.3/operating_systems/redhat7/repositories/HDP-2.... -d @repo.jso

.

Apply the same changes for "utils.json" as well.

[#6]

Thanks a lot for your time and patience. Tried it few times the exact way you proposed and some other options.

Didn't work for me.

I'm giving up as it it is not crucial for me.

BTW - the manual say that if I don't specify the explicit sub version, ambari would take the latest of the version. That of cource didn't work either.

[#7]

Bump to same 'repo_name' issue. But I got an error when trying to register the repo...
https://community.hortonworks.com/questions/182868/blueprint-registration-error.html

[#8]

Download the vdf file

wget http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.3.0/HDP-2.6.3.0-235.xml -O /tmp/HDP-2.6.3.0-235.xml 

Post it

curl -v -k -u admin:admin -H "X-Requested-By:ambari" -X POST \
http://ambari-server:8080/api/v1/version_definitions \
-d '{
"VersionDefinition": {
"version_url": "file:/tmp/HDP-2.6.3.0-235.xml"
}
}'

the response would be

{
"href" : "http://ambari-server:8080/api/v1/version_definitions/1",
"VersionDefinition" : {
"id" : 1,
"stack_name" : "HDP",
"stack_version" : "2.6"
}
} 

version definitions

http://ambari-server:8080/api/v1/version_definitions/1 

# can use id or version in the blueprint 
## id "repository_version_id": "1" 
## version "repository_version" : "2.6.3.0-235",