Support Questions
Find answers, ask questions, and share your expertise

Cloudera Automated Installation and Configuration

Cloudera Automated Installation and Configuration

New Contributor

 

Using this approach, one can use scripting to automate steps given below for Automated Cloudera Installation & Configuration:

 

  1.        Create passwordless ssh between all node machines
  2.        Install ntp and sync time of all nodes with NTP server
  3.        Remove given directories if exist, /opt/dfs, /etc/cloudera-scm-server, /opt/dfs/snn/current
  4.        Install Cloudera Repos, cloudera-cdh4.repo & cloudera-manager.repo on all nodes
  5.        Install cloudera-manager-daemons, cloudera-manager-server, cloudera-manager-server-db, cloudera-manager-agent & expect using yum on Manager node
  6.        Install cloudera-manager-daemons & cloudera-manager-agent on non-Manager nodes
  7.        Configure config.ini under cloudera-scm-agent on all non-manager nodes to point to Cloudera Manager IP
  8.        GET deployment descriptor from existing Cloudera setup using rest API, /cm/deployment
  9.        The deployment descriptor will contain all the existing custom configurations on the existing setup e.g. JVM Heap Size settings etc. If some more configuration changes are required, GET the deployment descriptor after making the changes.
  10.    Modify deployment descriptor for hosts information as per the requirement
  11.    Add Remote Parcel Repository URL (REMOTE_PARCEL_REPO_URLS) to the deployment descriptor, http://archive-primary.cloudera.com/cdh4/parcels/4.6.0.26/ under "managerSettings"
  12.    Start services cloudera-scm-server-db, cloudera-scm-server & cloudera-scm-agent on Manager node
  13.    Start service cloudera-scm-agent on all non-Manager nodes
  14.    Read db passwords for various Activity Monitor databases from /etc/cloudera-scm-server/db.mgmt.properties and update in the deployment descriptor
  15.    Create Postgresql database and respective role for Hive on Cloudera Postgres service running on Manager node, port 7432
  16.    Update the Hive database details in the deployment descriptor
  17.    Once Cloudera Manager has started, PUT deployment descriptor on Manager node using REST API, /cm/deployment
  18.    Check status of the Parcel using REST API, clusters/ML_CLUSTER_1/parcels/products/CDH/versions/4.6.0-1.cdh4.6.0.p0.26
  19.    Once Parcel is downloaded and distributed, activate Parcel using REST API, clusters/ML_CLUSTER_1/parcels/products/CDH/versions/4.6.0-1.cdh4.6.0.p0.26/commands/activate
  20.    Create supergroup user group on all machines
  21.    Add users hbase, mapred, hive, zookeeper, oozie & hdfs to the supergroup group
  22.    Add ZOO_DATADIR_AUTOCREATE_DISABLE=false to bash profile on all nodes
  23.    Deploy Client Configuration to all nodes using REST API, clusters/ML_CLUSTER_1/commands/deployClientConfig
  24.    Check if Configuration Deployment is complete using REST API, commands/{command_id}
  25.    Format HDFS node using REST API, clusters/ML_CLUSTER_1/services/hdfs1/roleCommands/hdfsFormat
  26.    Check if HDFS Format is complete using REST API, commands/{command_id}
  27.    Start Cluster using REST API, api/v4/clusters/ML_CLUSTER_1/commands/start
  28.    Start Monitoring services using the REST API
2 REPLIES 2

Re: Cloudera Automated Installation and Configuration

Hello, did you have any questions that you wanted answered?

Regards,
Gautam Gopalakrishnan

Re: Cloudera Automated Installation and Configuration

New Contributor

Hello;

 

Is it possible to change CM configuration to use only private ip addresses instead of hostnames?

 

Thanks

 

Guest