Hi Guys, I am new to hadoop, Cluster is 10 nodes. and Yarn memory is around 64 GB. I have scheduled few jobs that runs hive using TEZ engine data inserts in to partition tables and there is zeppelin dashboard that runs select on select table. I see the, Yarn memory usage goes between 10% to 100%. Container size is 1526 MB. HDP: 1. I see application allocates multiple containers.some occasion i have seen around 24 containers. Is is possible to limit number of container by application ? What's the best practice? 2. This is default ambari setting created 1 queue. How multiple queues helps in managing the load segregation. Is it possible to differentiate load based on the queues. 3. Some times, I have seen Apps Pending more than 0. Is there is any way I can assure it shouldn't go pending. Thanks Deepesh ... View more

Hello experts, I am trying to copy the files being written using tailfile-> putfile. Format of files are below. WD-YYYYMMDD.log At any point of time latest file will be get written(ex. WD-20160908.log) and next it will be switched to new date (ex. WD-20160909.log)and that file start getting written. File to tail : /root/wd/log1/WD-20160831.log Rolling FileName Pattern: WD-*.log State of file: Initial start position: Beginning of File File location : local Results are not looks correct. Please suggested. At source : [root@server1 log1]# cat > WD-20160830.log 1 2 3 ^C [root@server1 log1]# cat >> WD-20160831.log 4 5 6 ^C [root@server1 log1]# ls -ltr total 8 -rw-r--r--. 1 root root 6 Sep 9 00:53 WD-20160830.log -rw-r--r--. 1 root root 6 Sep 9 00:54 WD-20160831.log At target: [root@server1 log2]# ls -tlr total 12 -rw-r--r--. 1 nifi hadoop 2 Sep 9 00:53 WD-20160830.log -rw-r--r--. 1 nifi hadoop 4 Sep 9 00:54 WD-20160831.0-4.log -rw-r--r--. 1 nifi hadoop 2 Sep 9 00:54 WD-20160831.4-6.log [root@*****log2]# cat WD-20160830.log 1 [root@server1 log2]# cat WD-20160831.0-4.log 4 5 [root@server1 log2]# cat WD-20160831.4-6.log 6 ... View more

I am new to nifi. I am copying file the file using nifi getfile(Keep the source true) and putfile (/stage).And Every minute there is cronjob looks for file written by putfile , process them and send it hdfs. Soon files got moved out of (/stage) directory,Nifi re-copies the same files. Is there is any way to avoid re-copy of file in nifi. ... View more