ERROR 2018-08-09 14:11:58,872 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,873 ComponentStatusExecutor.py:121 - Status command for METRICS_MONITOR failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,873 ComponentStatusExecutor.py:132 - Status for METRICS_MONITOR has changed to INSTALLED ERROR 2018-08-09 14:11:58,873 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,873 ComponentStatusExecutor.py:121 - Status command for HIVE_CLIENT failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,873 ComponentStatusExecutor.py:132 - Status for HIVE_CLIENT has changed to INSTALLED ERROR 2018-08-09 14:11:58,874 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,874 ComponentStatusExecutor.py:121 - Status command for TEZ_CLIENT failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,874 ComponentStatusExecutor.py:132 - Status for TEZ_CLIENT has changed to INSTALLED ERROR 2018-08-09 14:11:58,874 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,875 ComponentStatusExecutor.py:121 - Status command for SPARK2_CLIENT failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,875 ComponentStatusExecutor.py:132 - Status for SPARK2_CLIENT has changed to INSTALLED ERROR 2018-08-09 14:11:58,875 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,875 ComponentStatusExecutor.py:121 - Status command for SECONDARY_NAMENODE failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,875 ComponentStatusExecutor.py:132 - Status for SECONDARY_NAMENODE has changed to INSTALLED ERROR 2018-08-09 14:11:58,876 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,876 ComponentStatusExecutor.py:121 - Status command for HIVE_METASTORE failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,876 ComponentStatusExecutor.py:132 - Status for HIVE_METASTORE has changed to INSTALLED ERROR 2018-08-09 14:11:58,876 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,876 ComponentStatusExecutor.py:121 - Status command for RESOURCEMANAGER failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,876 ComponentStatusExecutor.py:132 - Status for RESOURCEMANAGER has changed to INSTALLED ERROR 2018-08-09 14:11:58,877 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,877 ComponentStatusExecutor.py:121 - Status command for MAPREDUCE2_CLIENT failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,877 ComponentStatusExecutor.py:132 - Status for MAPREDUCE2_CLIENT has changed to INSTALLED ERROR 2018-08-09 14:11:58,877 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,877 ComponentStatusExecutor.py:121 - Status command for LIVY2_SERVER failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,878 ComponentStatusExecutor.py:132 - Status for LIVY2_SERVER has changed to INSTALLED ERROR 2018-08-09 14:11:58,878 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,878 ComponentStatusExecutor.py:121 - Status command for NAMENODE failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,878 ComponentStatusExecutor.py:132 - Status for NAMENODE has changed to INSTALLED ERROR 2018-08-09 14:11:58,878 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,878 ComponentStatusExecutor.py:121 - Status command for HDFS_CLIENT failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,879 ComponentStatusExecutor.py:132 - Status for HDFS_CLIENT has changed to INSTALLED ERROR 2018-08-09 14:11:58,879 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,879 ComponentStatusExecutor.py:121 - Status command for HIVE_SERVER failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,879 ComponentStatusExecutor.py:132 - Status for HIVE_SERVER has changed to INSTALLED ERROR 2018-08-09 14:11:58,879 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,880 ComponentStatusExecutor.py:121 - Status command for METRICS_COLLECTOR failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,880 ComponentStatusExecutor.py:132 - Status for METRICS_COLLECTOR has changed to INSTALLED ERROR 2018-08-09 14:11:58,880 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,880 ComponentStatusExecutor.py:121 - Status command for ZEPPELIN_MASTER failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,880 ComponentStatusExecutor.py:132 - Status for ZEPPELIN_MASTER has changed to INSTALLED ERROR 2018-08-09 14:11:58,881 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,881 ComponentStatusExecutor.py:121 - Status command for APP_TIMELINE_SERVER failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,881 ComponentStatusExecutor.py:132 - Status for APP_TIMELINE_SERVER has changed to INSTALLED ERROR 2018-08-09 14:11:58,881 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,881 ComponentStatusExecutor.py:121 - Status command for YARN_CLIENT failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,881 ComponentStatusExecutor.py:132 - Status for YARN_CLIENT has changed to INSTALLED ERROR 2018-08-09 14:11:58,882 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,882 ComponentStatusExecutor.py:121 - Status command for ZOOKEEPER_SERVER failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,882 ComponentStatusExecutor.py:132 - Status for ZOOKEEPER_SERVER has changed to INSTALLED ERROR 2018-08-09 14:11:58,882 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,882 ComponentStatusExecutor.py:121 - Status command for SPARK2_JOBHISTORYSERVER failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,883 ComponentStatusExecutor.py:132 - Status for SPARK2_JOBHISTORYSERVER has changed to INSTALLED ERROR 2018-08-09 14:11:58,883 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,883 ComponentStatusExecutor.py:121 - Status command for ZOOKEEPER_CLIENT failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,883 ComponentStatusExecutor.py:132 - Status for ZOOKEEPER_CLIENT has changed to INSTALLED ERROR 2018-08-09 14:11:58,883 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,883 ComponentStatusExecutor.py:121 - Status command for JOURNALNODE failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,884 ComponentStatusExecutor.py:132 - Status for JOURNALNODE has changed to INSTALLED ERROR 2018-08-09 14:11:58,884 CustomServiceOrchestrator.py:448 - Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 322, in runCommand command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,884 ComponentStatusExecutor.py:121 - Status command for HISTORYSERVER failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:11:58,884 ComponentStatusExecutor.py:132 - Status for HISTORYSERVER has changed to INSTALLED INFO 2018-08-09 14:12:00,926 NetUtil.py:61 - Connecting to https://ambari-server.XXX-SERVER-FQDN-XXX:8440/connection_info INFO 2018-08-09 14:12:00,947 security.py:61 - Connecting to wss://ambari-server.XXX-SERVER-FQDN-XXX:8441/agent/stomp/v1 INFO 2018-08-09 14:12:00,968 transport.py:329 - Starting receiver loop INFO 2018-08-09 14:12:01,062 HeartbeatThread.py:123 - Sending registration request INFO 2018-08-09 14:12:01,063 security.py:118 - Event to server at /register (correlation_id=0): {'currentPingPort': 8670, 'timestamp': 1533823920970, 'hostname': 'cs-eu-dhp-3-0-0-cluster-m2.XXX-SERVER-FQDN-XXX', 'publicHostname': '104.40.131.184', 'hardwareProfile': {'kernel': 'Linux', 'domain': 'XXX-SERVER-FQDN-XXX', 'kernelrelease': '3.10.0-862.3.3.el7.x86_64', 'uptime_days': '0', 'memorytotal': 28801164, 'swapfree': '0.00 GB', 'processorcount': 4, 'selinux': False, 'timezone': 'UTC', 'hardwareisa': 'x86_64', 'operatingsystem': 'centos', 'hostname': 'cs-eu-dhp-3-0-0-cluster-m2', 'id': 'root', 'memoryfree': 10964488, 'hardwaremodel': 'x86_64', 'uptime_seconds': '11364', 'osfamily': 'redhat', 'physicalprocessorcount': 4, 'interfaces': 'eth0,lo', 'memorysize': 28801164, 'swapsize': '0.00 GB', 'netmask': '255.255.0.0', 'ipaddress': '10.0.0.6', 'kernelmajversion': '3.10', 'kernelversion': '3.10.0', 'macaddress': '00:0D:3A:2E:CB:B4', 'operatingsystemrelease': '7.5.1804', 'uptime_hours': '3', 'fqdn': 'cs-eu-dhp-3-0-0-cluster-m2.XXX-SERVER-FQDN-XXX', 'architecture': 'x86_64', 'mounts': [{'available': '22377356', 'used': '8551792', 'percent': '28%', 'device': '/dev/sda2', 'mountpoint': '/', 'type': 'xfs', 'size': '30929148'}, {'available': '195727412', 'used': '61468', 'percent': '1%', 'device': '/dev/sdb1', 'mountpoint': '/mnt/resource', 'type': 'ext4', 'size': '206290920'}, {'available': '97673124', 'used': '148500', 'percent': '1%', 'device': '/dev/sdc', 'mountpoint': '/hadoopfs/fs1', 'type': 'ext4', 'size': '103080888'}]}, 'agentEnv': {'transparentHugePage': '', 'hostHealth': {'agentTimeStampAtReporting': 1533823921062, 'liveServices': [{'status': 'Healthy', 'name': 'chronyd', 'desc': ''}]}, 'reverseLookup': True, 'umask': '18', 'hasUnlimitedJcePolicy': None, 'alternatives': [], 'firewallName': 'iptables', 'stackFoldersAndFiles': [{'type': 'directory', 'name': '/etc/hadoop'}, {'type': 'directory', 'name': '/etc/hbase'}, {'type': 'directory', 'name': '/etc/hive'}, {'type': 'directory', 'name': '/etc/zookeeper'}, {'type': 'directory', 'name': '/etc/hive-hcatalog'}, {'type': 'directory', 'name': '/etc/tez'}, {'type': 'directory', 'name': '/etc/knox'}, {'type': 'directory', 'name': '/etc/ambari-metrics-collector'}, {'type': 'directory', 'name': '/etc/ambari-metrics-monitor'}, {'type': 'directory', 'name': '/etc/zeppelin'}, {'type': 'directory', 'name': '/var/run/hadoop'}, {'type': 'directory', 'name': '/var/run/hive'}, {'type': 'directory', 'name': '/var/run/zookeeper'}, {'type': 'directory', 'name': '/var/run/hive-hcatalog'}, {'type': 'directory', 'name': '/var/run/hadoop-yarn'}, {'type': 'directory', 'name': '/var/run/hadoop-mapreduce'}, {'type': 'directory', 'name': '/var/run/knox'}, {'type': 'directory', 'name': '/var/run/spark'}, {'type': 'directory', 'name': '/var/run/ambari-metrics-collector'}, {'type': 'directory', 'name': '/var/run/ambari-metrics-monitor'}, {'type': 'directory', 'name': '/var/run/zeppelin'}, {'type': 'directory', 'name': '/var/log/hadoop'}, {'type': 'directory', 'name': '/var/log/hbase'}, {'type': 'directory', 'name': '/var/log/hive'}, {'type': 'directory', 'name': '/var/log/zookeeper'}, {'type': 'directory', 'name': '/var/log/hive-hcatalog'}, {'type': 'directory', 'name': '/var/log/hadoop-yarn'}, {'type': 'directory', 'name': '/var/log/hadoop-mapreduce'}, {'type': 'directory', 'name': '/var/log/knox'}, {'type': 'directory', 'name': '/var/log/spark'}, {'type': 'directory', 'name': '/var/log/ambari-metrics-collector'}, {'type': 'directory', 'name': '/var/log/ambari-metrics-monitor'}, {'type': 'directory', 'name': '/var/log/zeppelin'}, {'type': 'directory', 'name': '/usr/lib/flume'}, {'type': 'directory', 'name': '/usr/lib/storm'}, {'type': 'directory', 'name': '/usr/lib/ambari-metrics-collector'}, {'type': 'directory', 'name': '/var/lib/hive'}, {'type': 'directory', 'name': '/var/lib/hadoop-hdfs'}, {'type': 'directory', 'name': '/var/lib/hadoop-yarn'}, {'type': 'directory', 'name': '/var/lib/hadoop-mapreduce'}, {'type': 'directory', 'name': '/var/lib/ambari-metrics-collector'}, {'type': 'directory', 'name': '/var/lib/ambari-metrics-monitor'}, {'type': 'directory', 'name': '/var/lib/zeppelin'}, {'type': 'directory', 'name': '/tmp/hive'}, {'type': 'directory', 'name': '/tmp/hadoop-hdfs'}, {'type': 'directory', 'name': '/hadoop/yarn'}], 'existingUsers': [{'status': 'Available', 'name': 'hive', 'homeDir': '/home/hive'}, {'status': 'Available', 'name': 'ambari-qa', 'homeDir': '/home/ambari-qa'}, {'status': 'Available', 'name': 'hbase', 'homeDir': '/var/run/hbase'}, {'status': 'Available', 'name': 'mapred', 'homeDir': '/home/mapred'}, {'status': 'Available', 'name': 'hdfs', 'homeDir': '/home/hdfs'}, {'status': 'Available', 'name': 'zookeeper', 'homeDir': '/home/zookeeper'}, {'status': 'Available', 'name': 'yarn', 'homeDir': '/home/yarn'}, {'status': 'Available', 'name': 'tez', 'homeDir': '/home/tez'}, {'status': 'Available', 'name': 'knox', 'homeDir': '/var/lib/knox'}, {'status': 'Available', 'name': 'ams', 'homeDir': '/home/ams'}, {'status': 'Available', 'name': 'spark', 'homeDir': '/home/spark'}, {'status': 'Available', 'name': 'zeppelin', 'homeDir': '/home/zeppelin'}], 'firewallRunning': False}, 'prefix': '/var/lib/ambari-agent/data', 'agentVersion': '2.7.0.0', 'agentStartTime': 1533823778768, 'id': -1} INFO 2018-08-09 14:12:01,077 __init__.py:49 - Event from server at /user/ (correlation_id=0): {u'status': u'OK', u'exitstatus': 1, u'id': -1} INFO 2018-08-09 14:12:01,083 HeartbeatThread.py:128 - Registration response received ERROR 2018-08-09 14:12:01,084 HeartbeatThread.py:104 - Exception in HeartbeatThread. Re-running the registration Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/HeartbeatThread.py", line 91, in run self.register() File "/usr/lib/ambari-agent/lib/ambari_agent/HeartbeatThread.py", line 131, in register self.handle_registration_response(response) File "/usr/lib/ambari-agent/lib/ambari_agent/HeartbeatThread.py", line 189, in handle_registration_response raise Exception(error_message) Exception: Registration failed INFO 2018-08-09 14:12:01,085 transport.py:358 - Receiver loop ended ^X^C [root@cs-eu-dhp-3-0-0-cluster-m2 ~]# nano /var/log/ambari-agent/ambari-agent.log GNU nano 2.3.1 File: /var/log/ambari-agent/ambari-agent.log command = self.generate_command(command_header) File "/usr/lib/ambari-agent/lib/ambari_agent/CustomServiceOrchestrator.py", line 487, in generate_command command_dict = self.configuration_builder.get_configuration(cluster_id, service_name, component_name, required_config_timestamp) File "/usr/lib/ambari-agent/lib/ambari_agent/ConfigurationBuilder.py", line 37, in get_configuration configurations_cache = self.configurations_cache[cluster_id] File "/usr/lib/ambari-agent/lib/ambari_agent/ClusterCache.py", line 155, in __getitem__ raise KeyError("{0} for cluster_id={1} is missing. Check if server sent it.".format(self.get_cache_name().title(), key)) KeyError: 'Configurations for cluster_id=2 is missing. Check if server sent it.' INFO 2018-08-09 14:12:18,895 ComponentStatusExecutor.py:121 - Status command for HISTORYSERVER failed: Caught an exception while executing custom service command: : 'Configurations for cluster_id=2 is missing. Check if server sent it.'; 'Configurations for cluste$ INFO 2018-08-09 14:12:18,896 ComponentStatusExecutor.py:132 - Status for HISTORYSERVER has changed to INSTALLED INFO 2018-08-09 14:12:21,239 NetUtil.py:61 - Connecting to https://ambari-server.XXX-SERVER-FQDN-XXX:8440/connection_info INFO 2018-08-09 14:12:21,257 security.py:61 - Connecting to wss://ambari-server.XXX-SERVER-FQDN-XXX:8441/agent/stomp/v1 INFO 2018-08-09 14:12:21,277 transport.py:329 - Starting receiver loop INFO 2018-08-09 14:12:21,356 HeartbeatThread.py:123 - Sending registration request INFO 2018-08-09 14:12:21,357 security.py:118 - Event to server at /register (correlation_id=0): {'currentPingPort': 8670, 'timestamp': 1533823941279, 'hostname': 'cs-eu-dhp-3-0-0-cluster-m$ INFO 2018-08-09 14:12:21,372 __init__.py:49 - Event from server at /user/ (correlation_id=0): {u'status': u'OK', u'exitstatus': 1, u'id': -1} INFO 2018-08-09 14:12:21,376 HeartbeatThread.py:128 - Registration response received ERROR 2018-08-09 14:12:21,376 HeartbeatThread.py:104 - Exception in HeartbeatThread. Re-running the registration Traceback (most recent call last): File "/usr/lib/ambari-agent/lib/ambari_agent/HeartbeatThread.py", line 91, in run self.register() File "/usr/lib/ambari-agent/lib/ambari_agent/HeartbeatThread.py", line 131, in register self.handle_registration_response(response) File "/usr/lib/ambari-agent/lib/ambari_agent/HeartbeatThread.py", line 189, in handle_registration_response raise Exception(error_message) Exception: Registration failed INFO 2018-08-09 14:12:21,377 transport.py:358 - Receiver loop ended