We have external tables on AWS S3 buckets in CSV format not compressed, when we try to query tables with simple Select * from example_table Limit 10 or Where serial = “SomeID” it takes time of minimum of 30 secs and consumes complete resources which are available in RM before the final display of output where table data is less which is approximate of 500 to 1000 records and there are also very large tables with 3 million records which displays even faster. Also one of the table with just 8331 records and 19 columns takes to 5-6 mins to complete Count Clause. Initiating itself takes 2-3 mins and once after initiated it completes quickly, this happens only with this table! I have changed the Execution engine for this table to MR which initiated quickly and completed in 80 Secs. I do not understand the TEZ execution plan if someone would be able to help me out would be appreciated! We have 3 node cluster built on ec2 installed HDP 2.5.2 and Hive 1.2.100. Out of which 2 are Datanodes, RM resources are 24 Vcores and 108 GB RAM. ... View more

Hi All, HDFS Audit logs size have become huge since recent past days, on weekends /root was completely occupied, upon checking largest files in /root it was hdfs.audit.log file which were sizing between 300 MB to 9.9 GB. When I checked file sizes from Oldest to Newest, oldest were 300 MB- 500MB sized, the Newest which are from last 6 days were sizing between 1.9 GB to 9.9 GB! Are there any particular reason for huge space occupancy all of a sudden? Ps: We have not installed Ranger, KNOX, SOLR and ATLAS. ... View more

I wanted to install HDP 2.5 under /opt but hdp-select usually installs under/usr/hdp. Can we symlink /usr/hdp to /opt. I wanted to know if there are disadvantages after setting up with symlink? Due to IT policies, our Linux team is asking me to install under /opt which is a standard procedure for us for any 3rd party software especially for production boxes. Why can't we install under /opt or any other user specified location? Is there any particular reason? ... View more