"How many concurrent queries can be executed on Hive" There is no hard limit. You have two potential bottlenecks a) HiveServer ( I could reach around 20-30 q/s in the newest Hiveserver version ) but you could add more Hiveservers. You should also increase heap to 8GB and you can increase handler threads a bit. b) Tez on Yarn. I reached up to 200 concurrent Hive Sessions (Application Masters ) once on a 50 node cluster But it depends much much more on the complexity of queries you are running. If your queries need 200 containers each on a cluster with 1000 slots you can only run 5 queries in parallel "How to configure Hadoop/Hive for scale of queries hitting it from the API layer" a) The most important part is to configure your queries to use as few resources ( containers ) as possible ( partitioning, predicate pushdown ... ). b) Container reuse Tez is already configured for Session and container reuse but make sure this works properly. You can also tweak the container release settings in tez. ( tez.am.session.min.held-containers tez.am.container.idle.release-timeout-max.millis tez.am.container.idle.release-timeout-min.millis )
Essentially each JDBC connection is one Session. Or you can initialize sessions and Hiveserver will distribute queries among them ( hive.server2.tez.sessions. per.default.queue, hive.server2.tez.initialize.default. sessions) . I prefer to keep the default but then you need to make sure your number of jdbc connections fits to the concurrency of your systemc) Number Sessions Always keep an eye out on the cluster. Also use the newest version of HDP or try disabling the timeline server ( in the hive settings not just switch it off ) "what data access layer then does one use with Other Platforms from the service layer into Hive/Hadoop? Is is just a JDBC connection at that point or something else?" A JDBC pool with x number of open connections that take queries from a pool is best. As said in default mode each jdbc connection will be one Application Master. Also jdbc is faster than beeline/hive shell when it comes to data transfer. "How to configure Hadoop/Hive for scale of queries hitting it from the API layer ? How many concurrent connnections to WebHDFS can be supported?" No ide what you mean. Webhdfs? Tez reads data directly from HDFS and it will not be the limit. Your limit will be either IO or much more likely CPU power in your cluster. Or some configuration bottleneck. Finally it all depends on you queries and your workload so nobody will give you a magical recommendation.
... View more
