Member since
10-01-2018
81
Posts
4
Kudos Received
4
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1498 | 09-28-2020 08:05 AM | |
| 1402 | 04-16-2020 09:20 AM | |
| 597 | 04-16-2020 08:48 AM | |
| 1523 | 04-16-2020 08:10 AM |
06-01-2022
12:06 PM
@yagoaparecidoti If you notice that the high memory utilization moves from tablet server to tablet server, then another candidate is a problem with the schema of one or more tables. The specific symptom I am thinking of is that the tablet size may be too high due to too few partitions; this can drive high memory utilization because the of the amount of information that must be loaded into memory. The fast way to determine whether this is the case is to look at the Charts for the Tablet Server with high memory utilization, and check the size of data across all tablets on that server. Then, divide this number by the number of replicas on the server, and this gives us an avg value of the replica size. It is perfectly possible for this problem to originate with a single table, so if you have one or more tables you know have few partitions, I recommend checking that table specifically. Beyond this will require log analysis, and is better suited to a support case.
... View more
05-26-2022
09:52 AM
These are all reasonable numbers by themselves, so no huge red flags here. My next question is: are the tablets balanced across the Tablet Servers? If you run a kudu cluster ksck, how many replicas does it show on each server? One scenario where we might see something like this is if the cluster was made with 3 Tablet Servers, and then 2 more were added. The reason this would cause an imbalance is that only data which was added to the cluster after those nodes were added would be on those servers, which means fewer tablets running on them, which means less load on the memory. The kudu rebalancer may be an option: $ sudo -u kudu kudu cluster rebalance <master addresses>
... View more
05-25-2022
10:35 AM
The several possible reasons for this, including but not limited to: a too-low value for hard_memory_limit_bytes; too-large tablet sizes; too-high a workload (which is basically insufficient memory but when there is not any more room to increase it). In order to tell more, we need to know more about the system. Can you tell me the following things about your cluster: 1) How many tablet replicas are there in your cluster, total? 2) How much data is in the kudu cluster? This should be available in the Charts section of Cloudera Manager. 3) How many fs_data_dirs are there for your Tablet Servers? 4) How many maintenance manager threads are there? 5) What is the current value of hard_memory_limit_bytes? With this information I can identify whether you are exceeding any basic scaling limits, or have an inefficient arrangement of workload, or have some kind of basic bottleneck that slows things down.
... View more
08-23-2021
09:24 PM
Context As of CDP 7.1.2, Sentry is deprecated with Kudu, and Ranger becomes the solution for Fine-Grained Authorization in Kudu. The question this article is meant to address is what does it mean for the Impala/Kudu stack to enable Ranger for Kudu, especially if Impala is already using Ranger? This is not obvious up front because the information is spread over three different sets of documents. Fundamentally the answer is nothing. The reason for this is straightforward: enabling the Kudu module in Ranger should automatically configure the --trusted_user_acl flag in Kudu to include Impala (if installed) and Hive (if installed). What this flag does is exempt the listed users from being checked against Kudu's authorization model, which in CDP installations mean Ranger policies. So, the correct way to enforce access to Impala and Hive tables, which are stored as Kudu, is the Hadoop-SQL policies set for Impala and Hive. By default, enabling Ranger for Kudu should have no impact on your Impala or Hive operations; any further changes you want to make to authorization should be done in the Hadoop-SQL policies. What is the motivation for using Ranger with Kudu, if that is the case? These tables are all still accessible at the Kudu level by users, and changes made from the Kudu level can cause inconsistencies or data loss. Enabling Ranger at the Kudu level prevents this. There are two other use-cases where policies are set at the Kudu level in cm_kudu: these are Spark and NiFi. These services are on a by-user basis instead of having a service user communicate with Kudu; so the completely normal logic of writing Ranger policies applies. Please see the documentation of the respective service for more information.
... View more
06-20-2021
09:21 PM
The Qualys tool reports vulnerabilities in ZooKeeper, even when the ZooKeeper security configuration is applied (HDP doc, CDP doc).
There are two kinds of reports Qualys makes that are not addressable by Cloudera:
The security guidance keeps several znodes in the affected services as world-readable, like so: /zookeeper/quota sasl:zookeeper:cdrwa,world:anyone:r Any world-readable znode will appear on the scanner, and will require an exception filed for it. This is the position of Qualys, as reported by our customers who use Qualys.
The security guidance does not cover several services. The following components require no action according to our documentation:
Calcite
Knox
MapReduce
Spark
Tez
Zeppelin
The Qualys tool will still report znodes owned by these services.
Note: it is possible to harden the ACLs beyond the Best Practices recommendation in the documentation, and to harden the ACLs of services not covered in the Best Practices. However, Cloudera cannot provide what the correct ACLs are in that case. Testing on the customer's side is required. It is very easy to set the ACLs such that services that need access to the znode will not have it, and this needs to be handled on a znode by znode basis.
If an attempt at hardening the ACLs is going to be made, these suggestions may help:
Try implementing SASL (this is the same method used in most of the Best Practices recommendations)
Try restricting privileges to the service user
If something breaks, try to identify the user that performed the failed action, and add the necessary privileges only for that user
... View more
06-01-2021
08:19 PM
Note: Cloudera does not support antivirus software of any kind.
This article contains recommendations for excluding HDP components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude which are unique to them. These details will be addressed in individual articles dedicated to the service in question.
The three primary locations you will want to exclude from Antivirus are:
Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes.
Log directories: These are write-heavy.
Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes.
In general, consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/hadoop /usr/hdp /etc/hadoop /etc/<component> /var/lib/<component>
Runtime and Logging
/var/run/<component> /var/log/<component>
Scratch and Temp
/var/tmp/<component> /tmp/<component>
Note: The <component> does not only refer to the service name, as a given service may have multiple daemons with their own directories.
Example: ambari-agent and ambari-server.
Across HDP services there are also many user-configurable locations. Most of these can be found in Ambari properties with names like 'service.scratch.dir' and 'service.data.dir'; go to Ambari > Service > Configs > Advanced and search for any property containing "dir", all of which may be considered for exclusion. Instructions for specific services follow:
Ambari:
Note: Ambari has a special requirement in the form of a user-configurable database. I recommend you exclude this database. However, the details of this database are set on installation; the database may be colocated with ambari-server, or on a remote host. Consult with your database administrators for details on the path where the database information is stored; Ambari does not keep this information anywhere in its configuration. If you need details about which database Ambari is using, search for JDBC in the amber.properties file.
# grep 'jdbc' /etc/ambari-server/conf/ambari.properties
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/usr/hdp
/usr/lib/ambari-agent
/usr/lib/ambari-server
/etc/hadoop
/etc/ambari-agent
/etc/ambari-server
/var/lib/ambari-agent
/var/lib/ambari-server
Runtime and Logging
/var/run/ambari-agent
/var/run/ambari-server
/var/log/ambari-agent
/var/log/ambari-server
HDFS:
Note: The directories in HDFS are user-configurable. I recommend you exclude these, especially the data directory for the DataNode and the meta directories for the NameNode and JournalNode. These details can be found in the 'hdfs-site.xml' file:
# grep -A1 "dir" /etc/hadoop/conf/hdfs-site.xml
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/usr/hdp
/etc/hadoop
/var/lib/hadoop-hdfs
Runtime and Logging
/var/run/hadoop
/var/log/hadoop
Scratch and Temp
/tmp/hadoop-hdfs
Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to exclude the other components.
YARN:
Note: The directories YARN uses are user-configurable. I recommend you exclude them. These properties can be found in Ambari > YARN > Configs > Advanced:
yarn.nodemanager.local-dirs
yarn.nodemanager.log-dirs
yarn.nodemanager.recovery.dir
yarn.timeline-service.leveldb-state-store.path
yarn.timeline-service.leveldb-timeline-store.path
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/usr/hdp
/etc/hadoop
/var/lib/hadoop-yarn
Runtime and Logging
/var/run/hadoop-yarn
/var/log/hadoop-yarn
Note : HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components.
MapReduce:
Note: Some directories in MapReduce are user-configurable. I recommend you exclude them. These properties can be found in Ambari > YARN > Configs > Advanced and this one, in particular, should be excluded:
mapreduce.jobhistory.recovery.store.leveldb.path
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/usr/hdp
/etc/hadoop
/var/lib/hadoop-mapreduce
Runtime and Logging
/var/run/hadoop-mapreduce
/var/log/hadoop-mapreduce
Note : HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components.
ZooKeeper:
Note: ZooKeeper has a user-configurable data directory. I recommend you exclude it. This directory can be found by running the following command:
# grep dataDir /etc/zookeeper/conf/zoo.cfg
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/usr/hdp
/etc/hadoop
Runtime and Logging
/var/run/zookeeper
/var/log/zookeeper
Note : HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components.
... View more
06-01-2021
08:10 PM
Note: Cloudera does not support Antivirus software of any kind.
This article contains recommendations for excluding CDH components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude which are unique to them. These details will be addressed in individual articles dedicated to the service in question.
The three primary locations you will want to exclude from antivirus are:
Data directories: These can be very large, and therefore, take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes.
Log directories: These are write-heavy.
Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes.
In general, consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/opt/cloudera /etc/<component> /var/lib/<component>
Runtime and Logging
/var/run/<component> /var/log/<component>
Scratch and Temp
/var/tmp/<component> /tmp/<component>
Note: The <component> does not only refer to the service name, as a given service may have multiple daemons with their own directories.
Example: cloudera-scm-server and cloudera-scm-agent.
Across CDH services, there are also many user-configurable locations. Most of these can be found in Cloudera Manager properties with names like service.scratch.dir and service.data.dir; go to CM > Service > Configurations and search for any property containing "dir", all of which may be considered for exclusion. Instructions for specific services follow:
Cloudera Manager:
Note: Cloudera Manager has a special requirement in the form of a user-configurable database. I recommend you exclude this database. However, the details of this database are set on installation; the database may be colocated with cloudera-scm-server, or on a remote host. Consult with your database administrators for details on the path where the database information is stored.
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/opt/cloudera/cm /opt/cloudera/cm-agent
/etc/cloudera-scm-agent
/etc/cloudera-scm-server
/var/lib/cloudera-host-monitor /var/lib/cloudera-scm-agent /var/lib/cloudera-scm-eventserver /var/lib/cloudera-scm-server /var/lib/cloudera-scm-server-db /var/lib/cloudera-service-monitor
Runtime and Logging
/var/run/cloudera-scm-agent
/var/run/cloudera-scm-server
/var/log/cloudera-scm-agent /var/log/cloudera-scm-alertpublisher /var/log/cloudera-scm-eventserver /var/log/cloudera-scm-firehose /var/log/cloudera-scm-server
HDFS:
Note: The directories in HDFS are user-configurable. I recommend you exclude these, especially the data directory for the DataNode and the meta directories for the NameNode and JournalNode. These details can be found in the hdfs-site.xml file:
# grep -A1 "dir" /etc/hadoop/conf/hdfs-site.xml
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/opt/cloudera
/var/lib/hadoop-hdfs
Runtime and Logging
/var/log/hadoop-hdfs
Scratch and Temp
/tmp/hadoop-hdfs
Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to exclude the other components.
YARN:
Note: The directories YARN uses are user-configurable. I recommend you exclude them. These properties can be found in Cloudera Manager > YARN > Configuration:
yarn.nodemanager.local-dirs
yarn.nodemanager.log-dirs
yarn.nodemanager.recovery.dir
yarn.timeline-service.leveldb-state-store.path
yarn.timeline-service.leveldb-timeline-store.path
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/opt/cloudera
/var/lib/hadoop-yarn
Runtime and Logging
/var/log/hadoop-yarn
Note : HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components.
MapReduce:
Note: Some directories in MapReduce are user-configurable. I recommend you exclude them. These properties can be found in Cloudera Manager > MapReduce > Configs, and this one, in particular, should be excluded:
mapreduce.jobhistory.recovery.store.leveldb.path
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/opt/cloudera
/var/lib/hadoop-mapreduce
Runtime and Logging
/var/log/hadoop-mapreduce
Note : HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components.
ZooKeeper:
Note: ZooKeeper has a user-configurable data directory. I recommend you exclude it. This directory can be found by running the following command:
# grep dataDir /etc/zookeeper/conf/zoo.cfg
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/opt/cloudera
Runtime and Logging
/var/log/zookeeper
Note : HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components.
... View more
06-01-2021
08:02 PM
Note: Cloudera does not support Antivirus software of any kind. This article contains recommendations for excluding CDP components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude that are unique to them. These details will be addressed in individual articles dedicated to the service in question. The three primary locations you will want to exclude from Antivirus are: Data directories: These can be very large, and therefore, take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes. In general, consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera /etc/<component> /var/lib/<component> Runtime and Logging /var/run/<component> /var/log/<component> Scratch and Temp /var/tmp/<component> /tmp/<component> Note: The <component> does not only refer to the service name, as a given service may have multiple daemons with their own directories. Example: cloudera-scm-agent and cloudera-scm-server Across CDP services, there are also many user-configurable locations. Most of these can be found in Cloudera Manager properties with names like "service.scratch.dir" and "service.data.dir"; go to Cloudera Manager > Service > Configuration and search for any property containing "dir", all of which may be considered for exclusion. Instructions for specific services follow: Cloudera Manager: Note : Cloudera Manager has a special requirement in the form of a user-configurable database. I recommend you to exclude this database. However, the details of this database are set on installation; the database may be co-located with cloudera-scm-server, or on a remote host. Consult with your database administrators for details on the path where the database information is stored. Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera/cm /opt/cloudera/cm-agent
/etc/cloudera-scm-agent
/etc/cloudera-scm-server
/var/lib/cloudera-host-monitor /var/lib/cloudera-scm-agent /var/lib/cloudera-scm-eventserver /var/lib/cloudera-scm-server /var/lib/cloudera-scm-server-db /var/lib/cloudera-service-monitor Runtime and Logging /var/run/cloudera-scm-agent
/var/run/cloudera-scm-server /var/run/cloudera-scm-server-db
/var/log/cloudera-scm-agent /var/log/cloudera-scm-alertpublisher /var/log/cloudera-scm-eventserver /var/log/cloudera-scm-firehose /var/log/cloudera-scm-server HDFS: Note: The directories in HDFS are user-configurable. I recommend you exclude these, especially the data directory for the DataNode and the meta directories for the NameNode and JournalNode. These details can be found in the hdfs-site.xml file: # grep -A1 "dir" /etc/hadoop/conf/hdfs-site.xml Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera
/var/lib/hadoop-hdfs Runtime and Logging /var/log/hadoop-hdfs Scratch and Temp /tmp/hadoop-hdfs Note: HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to exclude the other components. YARN: Note: The directories YARN uses are user-configurable. I recommend you exclude them. These properties can be found in Cloudera Manager > YARN > Configuration: yarn.nodemanager.local-dirs
yarn.nodemanager.log-dirs
yarn.nodemanager.recovery.dir
yarn.timeline-service.leveldb-state-store.path
yarn.timeline-service.leveldb-timeline-store.path Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera
/var/lib/hadoop-yarn Runtime and Logging /var/log/hadoop-yarn Note : HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components. MapReduce: Note: Some directories in MapReduce are user-configurable. I recommend you exclude them. These properties can be found in Cloudera Manager > MapReduce > Configs, and this one, in particular, should be excluded: mapreduce.jobhistory.recovery.store.leveldb.path Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera
/var/lib/hadoop-mapreduce Runtime and Logging /var/log/hadoop-mapreduce Note : HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components. ZooKeeper: Note: ZooKeeper has a user-configurable data directory. I recommend you exclude it. This directory can be found by running the following command: # grep dataDir /etc/zookeeper/conf/zoo.cfg Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /opt/cloudera Runtime and Logging /var/log/zookeeper Note : HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components.
... View more
09-29-2020
07:17 AM
Now that you have made several attempts to start, are there any notifications in Ambari? What we should see is a series of operations try to run, and alerts for errors that are encountered. The alerts icon is shaped like a bell, and the operations icon is shaped like a gear. If there is nothing in either of those locations, than Ambari is not responding at all. The simplest thing would be to restart the Sandbox from scratch. If you can get terminal access to the containers the cluster is running on, you can restart the ambari-server process on the Ambari host.
... View more
09-28-2020
10:25 AM
I'm glad you decided to try it out! From the screenshot, it looks like all the services are currently in the Stopped state. You should see a big button that says Actions; this will produce a drop down menu with different actions listed. What happens when you click on the Action button, and then select Start All Services?
... View more
09-28-2020
08:05 AM
1 Kudo
This will depend on what these other agents are to a large extent, but in the general case there are two options: 1) Run a cron job: one, check to see if the process is already running; two, check to see if HDFS is running; three start the process if neither of the above are true. 2) I believe Ubuntu 16 uses systemd. In systemd start order can be controlled via dependencies in the unit file of the process. You will have to reference it specifically for your version, but I believe there are two relevant settings: wants or requires determines which processes run together; before and after determines the order in which they are run. The most thorough solution is to set these other agents' unit files as requires HDFS and after HDFS.
... View more
09-28-2020
07:32 AM
1 Kudo
@matagyula I suggest we attempt to get more information out of fsck in the PROD environment. This has two parts: 1) Use the options to get more detailed output about which blocks go where, and include snapshots. $ hdfs fsck / -files -blocks -locations -includeSnapshots This will break the results down into files, which blocks belong to which files, and where those files are located. Note: this will be a longer fsck, and induce a heavier load. Not recommended during peak load times. 2) Check the user who is running the fsck. We recommend running as the hdfs user, or another admin-level user. Edit: hdfs fsck also ignores open files by default. Depending on your prod cluster's usage patterns and data structure, it is possible for a very large number of blocks of blocks to be open at once. You can include an option to include these in the count: $ hdfs fsck / -openforwrite I recommend this be done separately, before the heavier multi-option version above.
... View more
09-22-2020
09:20 AM
@kvinod Can you provide the procedure and exact command you are using to restore the snapshot? 1) Are you restoring the snapshot over the top of Y environment, or are you clearing it first? This kind of behavior often happens when a restoration does not overwrite existing content, merely adding to it. 2) Are the versions exactly the same between the two environments? It is sometimes necessary to modify the command and import a different version, due to subtle differences between them. If you could tell us your version information, that would also be useful. For example, there is a default method from CDH5: https://my.cloudera.com/knowledge/Copying-HBase-Table-Between-Clusters--ExportSnapshot?id=72706 But there is a problem with it that appears in some CDH6 versions: https://my.cloudera.com/knowledge/TSB-2020-379-Data-loss-with-restore-snapshot?id=283633 Regards, Ryan Blough, COE Cloudera Inc.
... View more
09-22-2020
09:04 AM
1 Kudo
@matagyula That does appear to be a discrepancy. There are a few things we can check for this. 1) Did you get the block numbers from the NameNode UI in both cases? If the information came from an alert, it may be out of date as old alerts are preserved. 2) In the PROD environment, are all of the DataNodes showing as online? You can get this information from the commandline using the following command: $ hdfs dfsadmin -report This should also include a block count; but the dfsadmin report will include the replicas, and identify incompletely replicated blocks as missing. 3) Is the replication factor the same in PROD as it is in the other environment? The simplest explanation is that one or more DataNodes have been excluded from the count, but if the count came from an alert it may be inaccurate due to timing. Regards, Ryan Blough, COE Cloudera Inc.
... View more
05-11-2020
09:17 AM
Note: Cloudera does not support antivirus software of any kind.
This article contains general recommendations for excluding ZooKeeper components and directories from antivirus scans and monitoring.
The three primary locations you will want to exclude from antivirus are:
Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes.
Log directories: These are write-heavy.
Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes.
Note: ZooKeeper has a user-configurable data directory. I recommend you exclude it. This directory can be found by running the following command:
# grep dataDir /etc/zookeeper/conf/zoo.cfg
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/usr/hdp
/etc/hadoop
Runtime and Logging
/var/run/zookeeper
/var/log/zookeeper
Note : HDFS, YARN, MapReduce and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components.
... View more
05-11-2020
09:17 AM
Note: Cloudera does not support antivirus software of any kind.
This article contains general recommendations for excluding MapReduce components and directories from antivirus scans and monitoring.
The three primary locations you will want to exclude from antivirus are:
Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes.
Log directories: These are write-heavy.
Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes.
Note: Some directories in MapReduce are user-configurable. I recommend you exclude them. These properties can be found in Ambari > YARN > Configs > Advanced, and this one in particular should be excluded:
mapreduce.jobhistory.recovery.store.leveldb.path
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/usr/hdp
/etc/hadoop
/var/lib/hadoop-mapreduce
Runtime and Logging
/var/run/hadoop-mapreduce
/var/log/hadoop-mapreduce
Note : HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components.
... View more
05-11-2020
09:17 AM
1 Kudo
Note: Cloudera does not support antivirus software of any kind.
This article contains general recommendations for excluding YARN components and directories from antivirus scans and monitoring.
The three primary locations you will want to exclude from antivirus are:
Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes.
Log directories: These are write-heavy.
Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes.
Note: The directories YARN uses are user-configurable. I recommend you exclude them. These properties can be found in Ambari > YARN > Configs > Advanced:
yarn.nodemanager.local-dirs
yarn.nodemanager.log-dirs
yarn.nodemanager.recovery.dir
yarn.timeline-service.leveldb-state-store.path
yarn.timeline-service.leveldb-timeline-store.path
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/usr/hdp
/etc/hadoop
/var/lib/hadoop-yarn
Runtime and Logging
/var/run/hadoop-yarn
/var/log/hadoop-yarn
Note : HDFS, YARN, MapReduce, and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to also exclude the other components.
... View more
05-11-2020
09:16 AM
1 Kudo
Note: Cloudera does not support antivirus software of any kind.
This article contains general recommendations for excluding HDFS components and directories from antivirus scans and monitoring.
The three primary locations you will want to exclude from antivirus are:
Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the antivirus holds up writes.
Log directories: These are write-heavy.
Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the antivirus holds up writes.
Note: The directories in HDFS are user-configurable. I recommend you exclude these, especially the data directory for the DataNode and the meta directories for the NameNode and JournalNode. These details can be found in the “hdfs-site.xml” file:
# grep -A1 "dir" /etc/hadoop/conf/hdfs-site.xml
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/usr/hdp
/etc/hadoop
/var/lib/hadoop-hdfs
Runtime and Logging
/var/run/hadoop
/var/log/hadoop
Scratch and Temp
/tmp/hadoop-hdfs
Note: HDFS, YARN, MapReduce and ZooKeeper are mutually interdependent and you are likely to experience unsatisfactory results if you fail to exclude the other components.
... View more
05-11-2020
09:16 AM
1 Kudo
Note: Cloudera does not support antivirus software of any kind.
This article contains general recommendations for excluding Ambari components and directories from antivirus scans and monitoring.
The three primary locations you will want to exclude from antivirus are:
Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes.
Log directories: These are write-heavy.
Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes.
Note: Ambari has a special requirement in the form of a user-configurable database. I recommend you exclude this database. However, the details of this database are set on installation; the database may be colocated with ambari-server, or on a remote host. Consult with your database administrators for details on the path where the database information is stored; Ambari does not keep this information anywhere in its configuration. If you need details about which database Ambari is using, search for JDBC in the “amber.properties” file.
# grep 'jdbc' /etc/ambari-server/conf/ambari.properties
Consider excluding the following directories and all of their subdirectories:
Installation, Configuration, and Libraries
/usr/hdp
/usr/lib/ambari-agent
/usr/lib/ambari-server
/etc/hadoop
/etc/ambari-agent
/etc/ambari-server
/var/lib/ambari-agent
/var/lib/ambari-server
Runtime and Logging
/var/run/ambari-agent
/var/run/ambari-server
/var/log/ambari-agent
/var/log/ambari-server
... View more
04-30-2020
12:21 AM
Note: Cloudera does not support antivirus software of any kind. This article contains generic recommendations for excluding HDP components and directories from AV scans and monitoring. It is important to note that these recommendations do not apply to each service, and further, some services will have additional items to exclude which are unique to them. These details will be addressed in individual articles dedicated to the service in question. The three primary locations you will want to exclude from antivirus are: Data directories: These can be very large, and therefore take a long time to scan; they can also be very write-heavy, and therefore suffer performance impacts or failures if the AV holds up writes. Log directories: These are write-heavy. Scratch directories: These are internal locations used by some services for writing temporary data, and can also cause performance impacts or failures if the AV holds up writes. Consider excluding the following directories and all of their subdirectories: Installation, Configuration, and Libraries /hadoop /usr/hdp /etc/hadoop /etc/<component> /var/lib/<component> Runtime and Logging /var/run/<component> /var/log/<component> Scratch and Temp /var/tmp/<component> /tmp/<component> Note: The <component> does not only refer to the service name, as a given service may have multiple daemons with their own directories. Example: ambari-agent and ambari-server. Across HDP services there are also many user-configurable locations. Most of these can be found in Ambari properties with names like "service.scratch.dir" and "service.data.dir"; go to Ambari > Service > Configs > Advanced and search for any property containing "dir", all of which may be considered for exclusion.
... View more
04-16-2020
09:20 AM
You are correct; the drivers are built for each platform. The HDP downloads page is here: https://www.cloudera.com/downloads/hdp.html It contains the JDBC41 driver for Hive.
... View more
04-16-2020
08:48 AM
You are correct; stoping a service is not the same as a service crashing. Alerts generally do not cover intentional administrator activity like starting and stopping of services. However, you do still have access to this information; starting and stopping of services are covered under Events, of the Audit type: https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cm_dg_events.html#cmug_topic_10 The AUDIT_EVENT type covers actions performed. This is also where you will track configuration changes. Turning to the question of API use, here is the Cloudera Manager documentation's section on the API: https://docs.cloudera.com/documentation/enterprise/6/6.3/topics/cloudera_manager.html#concept_nsg_jq3_mz Here is the Tutorial linked from that doc, which has a ton of examples, including starting and stopping of services: https://archive.cloudera.com/cm6/6.3.0/generic/jar/cm_api/apidocs/tutorial.html While the Alerts don't tell you when services are started and stopped, you can query Events through the API. We have a Knowledge Base Article on the subject: https://my.cloudera.com/knowledge/Accessing-Critical-Events-Using-the-Cloudera-Manager-API-?id=72521
... View more
04-16-2020
08:10 AM
To my knowledge, what the cluster uses to check NPT is the following command: # ntpdc -np This is used in the health checks for the hosts. This is included in the docs under the Host Health Tests. For example in CDH 5.16.x: https://docs.cloudera.com/documentation/enterprise/5-16-x/topics/cm_ht_host.html#concept_lxn_zxn_yk You might also consider Chrony over NTP. From a previous discussion I had with a subject matter expert at Red Hat, these are the t hings chronyd can do better than ntpd: - chronyd can work well in an environment where access to the time reference is intermittent, whereas ntpd needs regular polling of time reference to work well. - chronyd can perform well even when the network is congested for longer periods of time. - chronyd can usually synchronize the clock faster and with better accuracy. - chronyd quickly adapts to sudden changes in the rate of the clock, for example, due to changes in the temperature of the crystal oscillator, whereas ntpd may need a long time to settle down again. - In the default configuration, chronyd never steps the time after the clock has been synchronized at system start, in order not to upset other running programs. ntpd can be configured to never step the time too, but it has to use a different means of adjusting the clock, which has some disadvantages including negative effect on accuracy of the clock. - chronyd can adjust the rate of the clock on a Linux system in a larger range, which allows it to operate even on machines with a broken or unstable clock. For example, on some virtual machines. - chronyd is smaller, it uses less memory and it wakes up the CPU only when necessary, which is better for power saving. I think the intermittent connection and virtual-machine specific details might apply to your use case. Their relevant documentation: Differences Between ntpd and chronyd https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/system_administrators_guide/index#sect-differences_between_ntpd_and_chronyd UNDERSTANDING CHRONY AND ITS CONFIGURATION https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/system_administrators_guide/sect-understanding_chrony_and-its_configuration Note that you will have to disable ntpd, because the cluster checks for ntpd first and chronyd second. Regards, Ryan Blough, COE Cloudera Inc.
... View more
08-23-2019
12:26 PM
Very helpful! It may be implied by " Later as you have already deployed the cluster we need to reset the cluster," but I had to add an additional step of removing the empty baseurl repositories that had been configured on the hosts. If this step isn't done, the error continues to appear because the repo configuration is still left over from the failed attempt. This can be done two ways: # yum remove HDP-3.1-repo-1 # yum clean all And also by manually removing the repo file from `/etc/yum.repos.d/' and then running # yum clean all at least for the RHEL 7 hosts we were on.
... View more
