For your question: are Ambari Metrics and SmartSense necessary? - No they are not mandatory, and you can ignore the start errors for them if you dont need them for now. Also, can you try restarting Hive, and as Jay mentioned paste what errors you see on trying to restart that ... View more

@SBandaru - Below is an excellent article on HCC explaining distcp with Snapshots: https://community.hortonworks.com/articles/71775/managing-hadoop-dr-with-distcp-and-snapshots.html From the article: Source must support 'snapshots' hdfs dfsadmin -allowSnapshot <path> Target is "read-only" Target, after initial baseline 'distcp' sync needs to support snapshots. Process Identify the source and target 'parent' directory Do not initially create the destination directory, allow the first distcp to do that. For example: If I want to sync source `/data/a` with `/data/a_target`, do *NOT* pre-create the 'a_target' directory. Allow snapshots on the source directory hdfs dfsadmin -allowSnapshot /data/a Create a Snapshot of /data/a hdfs dfs -createSnapshot /data/a s1 Distcp the baseline copy (from the atomic snapshot). Note: /data/a_target does NOT exists prior to the following command. hadoop distcp /data/a/.snapshot/s1 /data/a_target Allow snapshots on the newly create target directory hdfs dfsadmin -allowSnapshot /data/a_target At this point /data/a_target should be considered "read-only". Do NOT make any changes to the content here. Create a matching snapshot in /data/a_target that matches the name of the snapshot used to build the baseline hdfs dfs -createSnapshot /data/a_target s1 Add some content to the source directory /data/a. Make changes, add, deletes, etc. that need to be replicated to /data/a_target. Take a new snapshot of /data/a hdfs dfs -createSnapshot /data/a s2 Just for fun, check on whats changed between the two snapshots hdfs snapshotDiff /data/a s1 s2 Ok, now let's migrate the changes to /data/a_target hadoop distcp -diff s1 s2 -update /data/a /data/a_target When that's completed, finish the cycle by creating a matching snapshot on /data/a_target hdfs dfs -createSnapshot /data/a_target s2 That's it. You've completed the cycle. Rinse and repeat. ... View more

Below post has one example script which deletes files older than certain days: https://community.hortonworks.com/questions/19204/do-we-have-any-script-which-we-can-use-to-clean-tm.html #!/bin/bash usage="Usage: dir_diff.sh [days]" if [!"$1"] then echo$usage exit1 fi now=$(date +%s) hadoop fs -ls /zone_encr2/ | grep "^d" | while read f; do dir_date=`echo $f | awk '{print $6}'` difference=$(( ( $now - $(date -d "$dir_date" +%s) ) / (24 * 60 * 60 ) )) if [$difference-gt$1]; then hadoop fs -ls `echo$f| awk '{ print $8 }'`; fi done ... View more

@khadeer mhmd - You should use the following properties of the RollingFileAppender to efficiently control the Size and the number of backup index of old log files: maxFileSize:This is the critical size of the file above which the file will be rolled. Default value is 10 MB. maxBackupIndex: This property denotes the number of backup files to be created. Default value is 1. More details in : https://community.hortonworks.com/questions/89171/hdfs-audit-log-file-size-issues.html#comment-89639 ... View more