We are keep getting this kind of alert. Because of too many queries submitted to cluster, we are getting this kind of alerts. Even after restarting name node, these alerts are not going. Is there a way to clean up these alerts. ... View more

Hi, Can any one faced issue while fetching data from external table. We are copying data from upstream system into our storage S3. As part of copy, directories along with Zero bytes files are been copied. Source File Format is in JSON format and Compress (Gz) . Below is Folder Hierarchy Structure DATE --> <Folder> <DAY=201803250> ---> Folder 1.json.gz --> File 2.json.gz <DAY=201803250> ---> Empty Zero Bytes Files. Please find below screenshot We are trying to create external table with JSON Serde. ADD JAR wasb://jsonserde@XYZ.blob.core.windows.net/json/json-serde-1.3.9.jar; SET hive.mapred.supports.subdirectories=TRUE; SET mapred.input.dir.recursive=TRUE; SET hive.merge.mapfiles = true; SET hive.merge.mapredfiles = true; SET hive.merge.tezfiles = true; DROP TABLE IF EXISTS Ext_STG1; CREATE EXTERNAL TABLE Ext_STG1(Col1 String, Col2 String, Col3 String) ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe' WITH SERDEPROPERTIES ("case.insensitive" = "true", "ignore.malformed.json" = "true") STORED AS TEXTFILE LOCATION 'wasb://container1@xyz.blob.core.windows.net/date/day=201803250/' TBLPROPERTIES ('serialization.null.format' = ''); select * from Ext_STG1 limit 100; ... View more

Hi, I hope you're doing good. I'm seeking your help on queue configuration. We have configured our Capacity Scheduler Queues has per link. we need your help on configuring cluster in such a way that both Larger Query and Smaller Query be executed same time. Today whenever user submits large query which deal with GB of data, causes application to consume all the free resources capacity which blocks other smaller query. How does other big data projects handles this scenario. We have setup two queues, but is their way in Templeton to submit query to particular Queue ?. I have scenario like below We have two Queues Q1 and Q2 each with 50% of cluster resources. We submit query to Hive through HiveServer2 and WEBHCAT (Templeton). When i submit my query to HiveServer2, it is making use of Q1 Queue capacity using HiveServer2 config. Is there a way or some setting which makes query submitted through WEBHCAT goes to Q2 queue capacity only or is there any command like CURL which can accept parameter i.e WEBHCAT REST API parameter to which queue the query needs to be submitted. because we are seeing one big query block others.. how to improve concurrency? Thanks in advance ... View more