To setup an Ambari managed HDP cluster, below services’ components require database for their respective metastore databases. Ambari Server Hive Oozie Ranger While a lab/sandbox environment can be setup with default databases for these components, using the same is strongly not recommended for Dev/QA/UAT/Production clusters. Due diligence and planning must be done to ensure that database selection is appropriate for enterprise standard production cluster. Below are key areas to be taken into consideration while planning to select a database for Ambari and HDP components. Supported Databases Refer below for supported databases for Ambari and different HDP components in current state. https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-installation/content/database_requirements.html High Availability Support It is strongly recommended to setup High Availability for different HDP components, which can be enabled for High Availability in a production cluster. High Availability is supported by different HDP components as below: https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.2.0/bk_ambari-user-guide/content/ch_managing_service_high_availability.html The relational database that backs the Hive Metastore, Ambari Server, Oozie Server etc. itself should also be made highly available using best practices defined for the database system in use for HDP services and Ambari Server to be truly Highly Available and not to have database as single point of failure for the service. Therefore, it is important to select relational database, which supports highly availability, and it should be discussed with in-house DBA when planning for a new database or to use an existing in-house database for HDP deployment. Cost of licensing and support HDP support subscription doesn’t cover any licensing and support for databases being used for Ambari Server and HDP components i.e. Hive Metastore and would incur additional licensing and support cost. Therefore, cost of licensing and support should be considered as an important factor for selecting appropriate database for Ambari Server and HDP stack. Note: Contact in-house database team/database vendor for details on cost for licensing and support for databases. Database maintenance and management Database to be used for Ambari Server and HDP components would need maintenance and management which can be quite frequent/regular for database backup, HA setup and recovery etc. Therefore, while selecting a database for Ambari/HDP, it must be ensured that your organization has in-house skilled people/DBAs available to perform these activities. It is not a good practice to use different relational databases for different components i.e. Postgres for Ambari, MySQL for Hive etc. to avoid complexity for management and maintenance of these different databases. It is recommended to pick relational database of your choice and use the same i.e. MySQL for all components or Postgres for all components and so on. ... View more

To Fix under-replicated blocks in HDFS, below is quick instruction to use: ####Fix under-replicated blocks### su - <$hdfs_user> bash-4.1$ hdfs fsck / | grep 'Under replicated' | awk -F':' '{print $1}' >> /tmp/under_replicated_files -bash-4.1$ for hdfsfile in `cat /tmp/under_replicated_files`; do echo "Fixing $hdfsfile :" ; hadoop fs -setrep 3 $hdfsfile; done ... View more

Below are steps to replace disk in slave nodes or to perform maintenance of slave nodes servers. 1. Decommission the Datanode and all services running on it (i.e. NodeManager, HBase RegionServer, Datanode etc). Refer below docs for the same. https://docs.hortonworks.com/HDPDocuments/Ambari-2.4.1.0/bk_ambari-user-guide/content/decommissioning_masters_and_slaves.html https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.0/bk_administration/content/ch_slave_nodes.html 2. Replace the disks or perform any other tasks for server maintenance. 3. Recommission the node. 4. Start all services components on the node. 5. Run Fsck utility for HDFS to ensure that HDFS is in healthy state. FSCK reports usually show a few over replicated blocks after a datanode is recommissioned which would automatically be fixed over time. ... View more