Blueprints dry-run/validation

[#1]

Dear community,

Recently faced a problem with validation of blueprint. Had made a blueprint using simple copy-past, but had not seen that some variables addressed in configuration does not exists. However the blueprint had been submitted successfully. The errors I had seen only in ambari-server log after part of cluster deployment job failed after being pending. Is there a method to make a dry-run/validation of blueprint including this variables?

Best regards,

Vladyslav

[#2]

@Vladislav Falfushinsky

Usually Ambari validates the blueprints when you try and create a cluster. creation would fail if there is any validation errors. can you tell me what error you are seeing in Ambari server?

[#3]

Yes, the validation works when some global constructions are incorrect. However when there is some variable that is not referenced and used somewhere in configuration seems it is not working correctly. I`d recieved following error: "Encountered a host group token in configuration which couldn't be matched to a host group:"

[#4]

Hi @Vladislav Falfushinsky, Apart from the validation routines that you mentioned at the Blueprint and Cluster Creation Template levels, there isn't really much support for a "dry run" deployment, although that would be a good idea for a future release, since it would assist in some types of testing while developing a new Blueprint.

I'm not sure what version of Ambari is being used in your case, but I'll assume Ambari 2.4 for now.

In Ambari 2.1.0, the Blueprints deployment mechanism was refactored to allow for a more dynamic deployment model, such that hosts could join the cluster after the Blueprint deployment started, and the Blueprints processor will react to this and update the configuration accordingly.

I'm not sure how you created the Blueprint, but if you exported a Blueprint from a particular cluster, and are trying to recreate the cluster on a slightly different topology, or using a different set of services, that may be a part of the problem.

Can you post the portion of the ambari-server.log that shows the error you mentioned? The host group involved in the failure should be mentioned in the log message. Can you also post the Blueprint you're trying to deploy?

It may be that your Blueprint references a host group that is not defined in the Blueprint, or perhaps a "%HOSTGROUP%" token is being used for a component that is not included in any host groups in your Blueprint.

Hope this helps.

[#5]

Thanks for explanation. I am using the latest version of Ambari and unfortunately had not saved that log after I had fixed an error. I had not exported the blueprint from cluster, I had written it from scratch. The only thing I had copied the configuration from some config on github, that actually contained different host_groups names. I had really referenced a non-existing "%HOSTGROUP%" token. I was wondering that it had not thrown the validation error on Blueprint import.

[#6]

When trying to deploy blueprint system does not verify dependencies. Is there a possibility to make Ambari verify the service dependencies when deploying blueprint? For instance, the same thing as when I am trying to added services from Ambari UI and it tells that service I am trying to install depends on something other.

[#7 · Accepted solution]

HI @Vladislav Falfushinsky,

The Blueprints processor does validate dependencies in a Blueprint, but at a different level than the UI.

The Ambari Web UI will validate at the service level (HDFS, Yarn, etc), while the Blueprint validates the dependencies between components ("NAMENODE", "YARN_RESOURCEMANAGER", etc).

This is due to the fact that Blueprints operate only on components, and not at the service level. This is why you see slightly different validation behavior in either deployment mode.

The Blueprint processor validates a Blueprint's components for dependencies using the dependencies defined in the stack definition for that service or set of components.

There are two possible reasons why you might not see the dependencies being verified:

1. If you've turned off validation while POST-ing the Blueprint, then no component-level validation will occur.

This validation is on by default, and is controlled by a query parameter that can be set when POST-ing the Blueprint:

https://cwiki.apache.org/confluence/display/AMBARI/Blueprints#Blueprints-BlueprintUsageOverview

2. If the stack definitions for the services in your Blueprint do not correctly specify dependencies between components, then validation may not occur, as the Blueprint validation process is driven by the stack data.

Hope this helps.

[#8]

Thanks @rnettleton. I`ve checked the reasons you mentioned. Everything looks correct.

1. The validation is enabled.

2. The stack definitions are also ok.

Will try to make some manual verification.