Support Questions

Find answers, ask questions, and share your expertise
Announcements
Celebrating as our community reaches 100,000 members! Thank you!

Phoenix - Query Timeout

avatar
Contributor

I have a table in HBase created via Phoenix. The table has approxmiately 20 million records. I'm connecting to Phoenix via:

phoenix-sqlline.py hbasemaster:2181:/hbase-unsecure

I'm trying to run a count as follows:

select count(columnname) from tablename;

When I run that SQL, Phoenix reports a timeout

org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:

java.net.SocketTimeoutException: callTimeout=60000, callDuration=60317: row ....

I've tried changing the hbase.rpc.timeout via Ambari, but that doesn't seem to be the issue. The default timeout in Ambari was set to 1m30s and I changed it to 2m. The timeout reported by Phoenix is 60s before and after the change, so I don't think that's the culprit anyway.

What setting do I need to change to allow for longer running queries? Is there something else that I should be looking at?

1 ACCEPTED SOLUTION

avatar
Super Collaborator

There is an odd environment variable used for the HBase configuration directory. Try setting HBASE_CONF_PATH=/etc/hbase/conf in your shell session and see if the timeout value gets picked up.

View solution in original post

16 REPLIES 16

avatar
Super Guru

Try increasing hbase.rpc.timeout in hbase-site.xml

avatar
Contributor

I tried that already, as I indicated in my original post. It did not work.

avatar
Super Guru

Right, sorry about that, but that's near certainly where the error is coming from. The question on my mind is why sqlline isn't picking up the configuration value. @Terry 's suggestion is a good one. Depending on the version of HDP you're running, HBASE_CONF_DIR would also work.

avatar
Master Guru

There are phoenix parameters as well:

https://phoenix.apache.org/tuning.html

Its weird I don't see an applicable timeout. Query timeout would make sense but that is ten minutes and not 1 minute.

phoenix.query.timeoutMs

avatar
Super Guru

The difference is that Michael showed a SocketTimeoutException in the timeout. That's most likely the HBase RPC framework limiting the lifetime of the socket, not the Phoenix application-level timeout you listed (which is still good to remember!)

avatar
Master Guru

But he said that one was 90s before and changing it didn't help. Ah well lets see what he comes up with.

The issue might be that he changed it on the server side but should have changed on the client. Perhaps the default is 60s?

avatar
Master Guru

Which it is: So most likely the phoenix client does not take the hbase-site.xml from ambari but uses default values.

hbase.rpc.timeout
Description

This is for the RPC layer to define how long HBase client applications take for a remote call to time out. It uses pings to check connections but will eventually throw a TimeoutException.

Default

60000

avatar
Super Guru

Oops. I missed that part 🙂

avatar
Contributor

I discovered several posts via Google that suggested the problem was hbase.rpc.timeout. When I changed the value of this via Ambari, it had no effect. As I responded below, the problem seemed to be that I was missing the HBASE_CONF_DIR environment variable so it was using the default settings instead of the Ambari configured settings.