am2cm-2.0.4.0-4 % ls -1
am2cm-2.0.0.2.0.4.0-4.jar
am2cm.sh
ambari7_blueprint.json
cm_migration.log
conf
configs_summary.log
kerberos_summary.json
lib
restore_collections.py
AM2CM script uses blueprint from Ambari cluster as input, and based on user input HDP version, makes a decision regarding the config mapping file need to be used. This mapping files consists of mapping rule from Ambari world to CM world.
Config files, such as user-settings.ini and service-config.ini, will be used as a part of mapping logic to generate CM deployment json.
How to use
Command to execute
am2cm-2.0.4.0-4 % sh am2cm.sh -bp conf/blueprint.json -dt cm_deployment.json
INPUT Ambari Blueprint : conf/blueprint.json
OUTPUT CM Template : cm_deployment.json
Starting blueprint to CM Template migration
What is source version 1.HDP2 2.HDP3/HDF352 (1 or 2)? 1
Total number of hosts in blueprint: 6
Your cluster has services (listed below) that are not handled by this migration tool.
AMBARI_METRICS
The tool will skip the above identified service related configs.
Do you want to proceed with migration (Y OR N)? (N):y
Processing: POWERSCALE
Processing: LIVY
Processing: SOLR
Processing: TEZ
Processing: HDFS
Processing: OOZIE
Processing: SQOOP_CLIENT
Processing: NIFIREGISTRY
Processing: ZOOKEEPER
Processing: HBASE
Processing: YARN
Processing: RANGER_KMS
Processing: KNOX
Processing: ATLAS
Processing: HIVE_ON_TEZ
Processing: RANGER
Processing: HIVE
Processing: KAFKA
Processing: NIFI
Processing: SPARK_ON_YARN
Adding: QUEUEMANAGER
CM Template is generated at : am2cm-2.0.4.0-4/cm_deployment.json
Kerberos summary file is generated at : am2cm-2.0.4.0-4/kerberos_summary.json
Successfully completed
The output of the command are:
- CM deployment.json has cluster template for Cloudera Manager
- Kerberos_summary.json gives us list of the keytabs required for CDP cluster
- configs_summary.log gives summary of the each config transformation from Ambari to Cloudera Manager
- cm_migration.log provide AM2CM execution log for each service.
Features
- HDP and HDF services: AM2CM supports blueprints from clusters, HDP 2.6.5*, HDP 3* and HDF3.5*
- Advance/Custom Configs: AM2CM tool can handle Ambari configs in advance and custom sections of service and migrate them to CM safety valves.
- Host/Group Mapping: AM2CM will be able to translate config group mapping into CM role groups.
- Adding New Services: Using AM2CM > config> service-config.ini, we will be able to add new services that are not available in Ambari world. Example: Yarn QM etc.
Hidden Features
- Option to dry run: Getting a blueprint with host is not possible before upgrading ambari to 7.1.* version. Here am2cm dry_run feature will be useful. While running am2cm we can pass argument `--dry_run` using which we can pass blueprint with out host mapping details and get a deployment json for validation purposes.
- Ignore list: AM2CM > config> service-confi.ini comes with config ignore list, which can be extended to ignore configs from blueprint and configs ignore will take default CM configs.
- Override configs in Blueprint: Using AM2CM > config > cm-config-mapping.ini files, we can over write configs in blueprint to new CM standards.
- Install new components: Using AM2CM > config > service-config.ini, we will be able to add new services that are not available in Ambari world. For example, Hue, Phoenix, etc.
- New cluster standards: Using AM2CM > config > user-settings.ini file, we can ingest new CM configs. For example, TLS configuration, Kerberos princ, etc.
- Summary of configs: Output of AM2CM provide you a log file with summary of the each config transformation from Ambari to CM. This will be useful to compare the configs before and after migration.
Limitations
- SSL/TLS configs: does not migrate SSL and TLS configuration to CM world.
- Kerberos (rules, configs): AM2CM does not migrate the kerberos configs, auth_to_local rules to CM.
- Rack Topology: If Host Rack topology is not managed by Ambari then blueprint does not hold rack information so this will not be migrate to CM deployment Json.
- Knox Topologies: Most often Knox topologies are not managed by Ambari, so blueprint does not have them. So, AM2CM will not migrate Knox topologies.
- Ranger Plugins : Ranger plugin configs, for example, plugin name from Ambari will not be migrated to CM deployment template.
- Backup and Restore: AM2CM will not take care of backup and restore of services metadata, for example, Hive DB, Oozie DB, this has to be taken care separately.
- Config Validations: AM2CM will be not be able to validate configs in Ambari, if they are compatible with CM versions. We have to do this separately after deploying the template.
- HA configs (HTTPFS): AM2CM will not be able to migrate the HA configs of new services HTTPFS. Validate them manually on CM side.
- Config groups with correct Names: AM2CM will create convert ambari config groups in to CM role groups. But it comes up with standard names. Once we are on CM side, we have to manually validate and correct the names.
Above-mentioned limitation can be handled separately either after deploying template or configuring AM2CM configs to ingest new configs.
Useful Links & Scripts
- HDP3 to CDP: Transitioning HDP cluster to CDP Private Cloud Base cluster using the AM2CM tool
- HDP2 to CDP: Transitioning HDP 2.6.5 cluster to CDP Private Cloud Base 7.1.x cluster using the AM2CM tool
Full Disclosure & Disclaimer:
I am an Employee of Cloudera, but this is not part of the formal documentation of the Cloudera Data platform. It is purely based on my own experience of advising people in their choice of tooling and customisations.
Announcements
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
What's New @ Cloudera
