Member since
03-27-2018
4
Posts
0
Kudos Received
0
Solutions
04-23-2018
07:53 AM
cc: @Steve
Pat
as @Shu accidentally mentioned me by mistake 🙂
... View more
03-29-2018
08:53 PM
Hi Andrew, I have it working but haven't tried upgrading - just fresh installs with pre-baked docker containers and a definitive version using a definitive blueprint definition. This is a snippet out of the ambari-setup.sh # register stack for given version
echoInfo "Getting software stack..."
##############################
# We will download the file if we need it and bake it into the image
# EDIT: no longer using this since 2.6.4 broke the process
##############################
#myUrl="http://public-repo-1.hortonworks.com/HDP/centos6/$(getHdpVersionX)/updates/$(getHdpVersionFourDigit)/HDP-${myHdpFullVersion}.xml"
#myResponse=$(curl -s -w "HTTPSTATUS:%{http_code}" $myUrl)
#myBody=$(echo $myResponse | sed -e 's/HTTPSTATUS\:.*//g')
#myStatus=$(echo $myResponse | tr -d '\n' | sed -e 's/.*HTTPSTATUS://')
#echoInfo "Code from stack version request: $myStatus"
#[ $myStatus -eq 200 ] || die "HTTP status incorrect: $myStatus. Using myUrl '${myUrl}'"
# EDIT: now keeping our own files since 2.6.4 broke with the <tags> problem...
myUrl="file://${scriptDir}/HDP-${myHdpFullVersion}.xml"
myVDF="$myUrl"
echoInfo "Setting software stack..."
myUrl="http://${myHostname}:8080/api/v1/version_definitions"
myResponse=$(curl -d '{ "VersionDefinition": { "version_url": "'$myVDF'" } }' -s -w "HTTPSTATUS:%{http_code}" -u xxxx:xxxx -H "X-Requested-By: ambari" -X POST $myUrl)
myBody=$(echo $myResponse | sed -e 's/HTTPSTATUS\:.*//g')
myStatus=$(echo $myResponse | tr -d '\n' | sed -e 's/.*HTTPSTATUS://')
echoInfo "Code from stack post request: $myStatus"
echoInfo "Body from stack post request: $myBody"
[ $myStatus -eq 201 ] || die "HTTP status incorrect: $myStatus."
myDefinitionId=$(echo "$myBody" | grep -oE "\"id\" : [0-9]*" | awk '{ print $3 }')
[ -n "$myDefinitionId" ] && echoInfo "Determined versionDefinitionId '$myDefinitionId'" || die "Could not determine version definition id from response."
blueprintFile=$(mktemp)
cp $scriptDir/"template-blueprint-${myBlueprintType}.json" $blueprintFile
sed -i -e 's/placeholder.cluster.name/'"${CLUSTERNAME}"'/g' $blueprintFile
sed -i -e 's/placeholder.hdp.version/'"$(getHdpVersion)"'/g' $blueprintFile
sed -i -e 's/placeholder.cpu.cores/'"${myProcessorCnt}"'/g' $blueprintFile
sed -i -e 's/placeholder.memory.max/'"${myMemoryFree}"'/g' $blueprintFile
checkUnreplaced $blueprintFile
hostmappingFile=$(mktemp)
cp $scriptDir/template-hostmapping-multi.json $hostmappingFile
sed -i -e 's/placeholder.cluster.name/'"${CLUSTERNAME}"'/g' $hostmappingFile
sed -i -e 's/placeholder.master.1/'"${myMaster}"'/g' $hostmappingFile
sed -i -e 's/placeholder.data.cnt/'"${myDataNodes}"'/g' $hostmappingFile
sed -i -e 's/placeholder.repository.version.id/'"${myDefinitionId}"'/g' $hostmappingFile
checkUnreplaced $hostmappingFile
echoInfo "Setting blueprint..."
myUrl="http://${myHostname}:8080/api/v1/blueprints/${CLUSTERNAME}"
myResponse=$(curl -d @${blueprintFile} -s -w "HTTPSTATUS:%{http_code}" -u xxxx:xxxx -H "X-Requested-By: ambari" -X POST $myUrl)
myBody=$(echo $myResponse | sed -e 's/HTTPSTATUS\:.*//g')
myStatus=$(echo $myResponse | tr -d '\n' | sed -e 's/.*HTTPSTATUS://')
echoInfo "Code from blueprint post request: $myStatus"
echoInfo "Body from blueprint post request: $myBody"
[ $myStatus -eq 201 ] || die "HTTP status incorrect: $myStatus."
echoInfo "Setting up hostmapping"
myUrl="http://${myHostname}:8080/api/v1/clusters/${CLUSTERNAME}"
myResponse=$(curl -d @${hostmappingFile} -s -w "HTTPSTATUS:%{http_code}" -u xxxx:xxxx -H "X-Requested-By: ambari" -X POST $myUrl)
myBody=$(echo $myResponse | sed -e 's/HTTPSTATUS\:.*//g')
myStatus=$(echo $myResponse | tr -d '\n' | sed -e 's/.*HTTPSTATUS://')
echoInfo "Code from hostmapping post request: $myStatus"
echoInfo "Body from hostmapping post request: $myBody"
[ $myStatus -eq 202 ] || die "HTTP status incorrect: $myStatus."
echo "$myBody" > /tmp/blueprintRequest.log
echoInfo "Blueprint and hostmapping sent successfully" The myDefinitionId is then used in the hostmapping file {
"blueprint" : "placeholder.cluster.name",
"repository_version_id": placeholder.repository.version.id,
"default_password" : "xxxxxxx",
"host_groups" :[
{
"name" : "master",
"hosts" : [
{ "fqdn" : "placeholder.master.1" }
]
},
{
"name" : "slaves",
"host_count" : "placeholder.data.cnt",
"host_predicate" : "Hosts/cpu_count>0"
}
]
} Hope this helps. Steve
... View more
03-29-2018
08:52 PM
You can have the repos in place. They just need to have the matching repo IDs. My use case was that I wanted to create docker containers with the core dependencies already installed (decreased setup time) have the repos use our internal nexus for any other dependencies needed (faster and more reliable downloads) A little off-topic and I haven't tried "upgrading" maintenance versions but I'm guessing any problems have to do with the fact the stack only covers the <major>.<minor> version whereas the repositories actually contain the <major>.<minor>.<maintenance>-<build> information. So for a Blueprint "2.6" stack, if the "2.6.2.0" packages had already been installed under a different repository ID, they are not recognised and therefore not upgraded meaning any "2.6.3.0" packages would be installed "on top" of the current ones. Or something along those lines anyway 😉
... View more
03-27-2018
06:32 PM
Hi, I have the same issue wit 2.6.4. The blueprint uses the xml file for the release in question, in my case found here: http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.6.4.0/HDP-2.6.4.0-91.xml This XML contains the problematic tag elements. I removed them before uploading the blueprint and it worked or at least seemed to. My questions now are:
what are the tag elements used for? was I correct to delete them or is there a better way? Cheers, Steve
... View more