@mike_bronson7  Always stick to the Cloudera documentation. Yes !!! there is no risk in running that command I can understand your reservation.   ... View more

@bvishal  SmartSense Tool (HST) gives all support subscription customers access to a unique service that analyzes cluster diagnostic data, identifies potential issues, and recommends specific solutions and actions. These analytics proactively identify unseen issues and notify customers of potential problems before they occur.   That is okay as you are just testing and you don't need to buy support which is advised when running a production environment To configure SmartSense you will need to configure the /etc/hst/conf/hst-server.ini the inputs/values   you will get from Hortonworks support if you  have paid for a subscription customer.smartsense.id customer.account.name customer.notification.email customer.enable.flex.subscription The error you are encountering is normal and won't impact your cluster Hope that helps ... View more

@wert_1311  Domain name changes will affect the KDC database. Kerberos is super sensitive to domain changes according to experience you will have to recreate the KDC database and regenerate the keytabs/principals to enable you applications to reconnect. Cluster hostname If the hosts in the cluster were re-named ie host1.old.com to host1.new.com then ensure those changes are also reflected or resolved by the DNS.   This is going a tricky one but fortunately, CM or Ambari will make your work easy now that your domain has changed the earlier generated keytabs have the old domain name . A keytab contains a pair of principals and an encrypted copy of that principal's key it's unique to each host since the principal names include the hostname and may be concatenated with the domain name Delete the old KDC database Usually, as the root user call the Kerberos database utility kdb5_util destroy assuming the old domain was OLD.COM this should delete the keytabs and principals linked to the old REALM,   # kdb5_util -r OLD.COM destroy You will need to manually delete the keytabs liked to the old REALM on the file system /etc/security/keytabs/ [HDP] or /etc/hadoop/conf/[CDH]. You will be prompted to confirm before destroying the database, usually, this is a better option if you have second thought rather than using the kdb5_util destroy -f will naturally not prompt you for a confirmation   Recreate the New KDC database Use the Kerberos database utility kdb5_util create [-s] assuming the new domain was NEW.COM # kdb5_util NEW.COM create # kdb5_util -r NEW.COM create -s With the -s option, kdb5_util will stash a copy of the master key in a stash file this allows a KDC to authenticate itself to the database utilities, such as kadmin, kadmind, krb5kdc, and kdb5_util best option.   Update Kerberos files. Make sure you update the below files to reflect the new REALM assuming your MIT KDC server's domain isn't changed. krb5.conf kdc.conf kadm5.acl Auth-to-local Rules jaas.conf files [if being used by applications] Enable Kerberos Using CM or Ambari the process is straight forward.   Please let me know if you need more help   ... View more

@mahfooz  The property value can be modified only in hive-site.xml cluster configuration file. This will oblige you to restart the stale hive configuration and becomes a cluster-wide change rather than a runtime change. HTH ... View more

@mike_bronson7  Let me try to answer all your 3 questions in a shot [snapshot]   Zookeeper has 2 types of logs the snapshot and transactional log files. As changes are made to the znodes i.e addition or deletion of znodes these changes are appended to a transaction log, occasionally, when a log grows large, a snapshot of the current state of all znodes will be written to the filesystem. This snapshot supersedes all previous logs. To put you in context it's like the edit-logs and the fsimage in Namenode architecture, all changes made in the HDFS is logged in the edits-logs in secondary Namenode when a checkpoint kick in it merges the edits log with the old fsimage to incorporate the changes ever since the last checkpoint.   So zk snapshot is synonym to the fsimage as it contains the current state of the znode entries and ACL's Snapshot policy In the earlier command shared the snapshot count parameter -n <count> if you really want to have sleep then you can increment it to 5 or 7 but I think 3 suffice to use the autopurge feature so I keep only 3 snapshots and 3 transaction logs. When enabled, ZooKeeper auto-purge feature retains the autopurge.snapRetainCount most recent snapshots and the corresponding transaction logs in the dataDir and dataLogDir respectively and deletes the rest. Defaults to 3. The minimum value is 3. Corrupt snapshots The Zookeeper might not be able to read its database and fail to come up because of some file corruption in the transaction logs of the ZooKeeper server.   You will see some IOException on loading the ZooKeeper database. In such a case, make sure all the other servers in your ensemble are up and working. Use the 4 letters command "stat" command on the command port to see if they are in good health. After you have verified that all the other servers of the ensemble are up, you can go ahead and clean the database of the corrupt server. Solution   Delete all the files in datadir/version-2 and datalogdir/version-2/. Restart the server. Hope that helps ... View more

@Shelton,   We were trying to see if database and tables created in MySQL are imported through import-hive.sh. What we saw that import-hive.sh only imports the database and tables created through hive shell and not MySQL. Once we started creating the tables through the hive shell, the import-hive.sh started importing them and the same was seen in Atlas as well. ... View more

@mike_bronson7  log.retention.bytes is a size-based retention policy for logs, i.e the allowed size of the topic. Segments are pruned from the log as long as the remaining segments don't drop below log.retention.bytes. You can also specify retention parameters at the topic level To specify a retention time period per topic, use the following command.   kafka-configs.sh --zookeeper [ZooKeeperConnectionString] --alter --entity-type topics --entity-name [TopicName] --add-config retention.ms=[DesiredRetentionTimePeriod]   To specify a retention log size per topic, use the following command.     kafka-configs.sh --zookeeper [ZooKeeperConnectionString] --alter --entity-type topics --entity-name [TopicName] --add-config retention.bytes=[DesiredRetentionLogSize]     That should resolve your problem   Happy hadooping       ... View more

I was able to restart to the datanode from the Ambari UI after a restart of the ambari-agent on the servers where the datanode run ... View more