Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

How many HIVE concurrent queries can be executed concurrently?

avatar
Expert Contributor

Guys,

- How to configure Hadoop/Hive for scale of queries hitting it from the API layer ? How many concurrent connnections to WebHDFS can be supported?

- How many concurrent queries can be executed on Hive?

- How to configure Hadoop/Hive for scale of queries hitting it from the API layer

- what data access layer then does one use with Other Platforms from the service layer into Hive/Hadoop? Is is just a JDBC connection at that point or something else?

Thanks.

1 ACCEPTED SOLUTION

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login
5 REPLIES 5

avatar
Master Guru
hide-solution

This problem has been solved!

Want to get a detailed solution you have to login/registered on the community

Register/Login

avatar
Expert Contributor

I'm curious what the advantage of suppressing Hive ATS reporting might be. From an esthetic standpoint we really don't want the UI filled up with myriads of successful, short-running queries, but it would be nice to switch it on in a case by case basis for debugging purposes. Beyond that, would turning it off improve query latency?

avatar
Master Guru

@Steven Hirsch

the big issue is that ATS 1.0 often couldn't keep up with 10s of queries per second on large clusters. And in some situations this limited the number of queries running in the cluster. Like really bad. Like cluster being empty because it would wait for ATS bad. There were some tuning options to make that better but by and large the single ATS server and single leveldb backend had limitations. So less aesthetic and more performance. In ATS 1.5 they made it better ATS 2.0 hopefully fixes that problem once and for all.

avatar
Expert Contributor

@Benjamin Leonhardi

Thanks for the explanation. We're having significant scaling issues with our 32-host Hortonworks HDP-2.3.2.0-2950 installation. How do I determine which version of ATS is installed? I do not see it listed in the 'Stacks and Versions' page in Ambari. Assuming we're running one of the troublesome versions, what's the most expedient way to disable reporting to ATS from Hive? Since it is useful for debugging, I'm hoping there's a session parameter we can set at query time to suppress reporting when performance is an issue.

avatar
Master Guru

@Steven Hirsch

I think you can try it for one application, one possibility is to simply switch off ATS for a bit that helped me once but not a second time ( Tez still tries to log to it ) So if you really want to switch it off completely you can add the following settings:

tez.history.logging.service.class
= org.apache.tez.dag.history.logging.impl.SimpleHistoryLoggingService
and to see the logs:
tez.simple.history.logging.dir=/mylogdir

Also removing the following ATSHooks

hive.exec.pre.hooks=
hive.exec.post.hooks=

Also potentially reduce log levels

hive.tez.log.level=ERROR

And see if it makes things faster. Again if you don't see a difference you may have other issues. But its worth to rule out. ATS 1.5 has been enabled in HDP2.4. Also ATS1.0 has some tuning options. If that is really your bottleneck Hortonworks support may be able to help.