This will depend on what these other agents are to a large extent, but in the general case there are two options:   1) Run a cron job: one, check to see if the process is already running; two, check to see if HDFS is running; three start the process if neither of the above are true.   2) I believe Ubuntu 16 uses systemd. In systemd start order can be controlled via dependencies in the unit file of the process. You will have to reference it specifically for your version, but I believe there are two relevant settings: wants or requires determines which processes run together; before and after determines the order in which they are run. The most thorough solution is to set these other agents' unit files as requires HDFS and after HDFS. ... View more

You are correct; the drivers are built for each platform. The HDP downloads page is here: https://www.cloudera.com/downloads/hdp.html   It contains the JDBC41 driver for Hive.   ... View more

You are correct; stoping a service is not the same as a service crashing. Alerts generally do not cover intentional administrator activity like starting and stopping of services.   However, you do still have access to this information; starting and stopping of services are covered under Events, of the Audit type:   https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_dg_events.html#cmug_topic_10   The AUDIT_EVENT type covers actions performed. This is also where you will track configuration changes.   Turning to the question of API use, here is the Cloudera Manager documentation's section on the API:   https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cloudera_manager.html#concept_nsg_jq3_mz   Here is the Tutorial linked from that doc, which has a ton of examples, including starting and stopping of services:   https://archive.cloudera.com/cm6/6.3.0/generic/jar/cm_api/apidocs/tutorial.html   While the Alerts don't tell you when services are started and stopped, you can query Events through the API. We have a Knowledge Base Article on the subject:   https://my.cloudera.com/knowledge/Accessing-Critical-Events-Using-the-Cloudera-Manager-API-?id=72521 ... View more

To my knowledge, what the cluster uses to check NPT is the following command:   # ntpdc -np   This is used in the health checks for the hosts. This is included in the docs under the Host Health Tests. For example in CDH 5.16.x:   https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cm_ht_host.html#concept_lxn_zxn_yk   You might also consider Chrony over NTP. From a previous discussion I had with a subject matter expert at Red Hat, these are the t hings chronyd can do better than ntpd: - chronyd can work well in an environment where access to the time reference is intermittent, whereas ntpd needs regular polling of time reference to work well. - chronyd can perform well even when the network is congested for longer periods of time. - chronyd can usually synchronize the clock faster and with better accuracy. - chronyd quickly adapts to sudden changes in the rate of the clock, for example, due to changes in the temperature of the crystal oscillator, whereas ntpd may need a long time to settle down again. - In the default configuration, chronyd never steps the time after the clock has been synchronized at system start, in order not to upset other running programs. ntpd can be configured to never step the time too, but it has to use a different means of adjusting the clock, which has some disadvantages including negative effect on accuracy of the clock. - chronyd can adjust the rate of the clock on a Linux system in a larger range, which allows it to operate even on machines with a broken or unstable clock. For example, on some virtual machines. - chronyd is smaller, it uses less memory and it wakes up the CPU only when necessary, which is better for power saving.   I think the intermittent connection and virtual-machine specific details might apply to your use case.   Their relevant documentation: Differences Between ntpd and chronyd https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/system_administrators_guide/index#sect-differences_between_ntpd_and_chronyd UNDERSTANDING CHRONY AND ITS CONFIGURATION https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sect-understanding_chrony_and-its_configuration   Note that you will have to disable ntpd, because the cluster checks for ntpd first and chronyd second.   Regards, Ryan Blough, COE Cloudera Inc. ... View more