question Re: Phoenix - Query Timeout in Archives of Support Questions (Read Only)

Phoenix - Query Timeout

Jaraxal — Tue, 02 Feb 2016 01:25:38 GMT

I have a table in HBase created via Phoenix. The table has approxmiately 20 million records. I'm connecting to Phoenix via:

phoenix-sqlline.py hbasemaster:2181:/hbase-unsecure

I'm trying to run a count as follows:

select count(columnname) from tablename;

When I run that SQL, Phoenix reports a timeout

org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:

java.net.SocketTimeoutException: callTimeout=60000, callDuration=60317: row ....

I've tried changing the hbase.rpc.timeout via Ambari, but that doesn't seem to be the issue. The default timeout in Ambari was set to 1m30s and I changed it to 2m. The timeout reported by Phoenix is 60s before and after the change, so I don't think that's the culprit anyway.

What setting do I need to change to allow for longer running queries? Is there something else that I should be looking at?

Re: Phoenix - Query Timeout

elserj — Tue, 02 Feb 2016 01:29:15 GMT

Try increasing hbase.rpc.timeout in hbase-site.xml

Re: Phoenix - Query Timeout

bleonhardi — Tue, 02 Feb 2016 01:42:29 GMT

There are phoenix parameters as well:

https://phoenix.apache.org/tuning.html

Its weird I don't see an applicable timeout. Query timeout would make sense but that is ten minutes and not 1 minute.

phoenix.query.timeoutMs

Re: Phoenix - Query Timeout

elserj — Tue, 02 Feb 2016 01:46:16 GMT

The difference is that Michael showed a SocketTimeoutException in the timeout. That's most likely the HBase RPC framework limiting the lifetime of the socket, not the Phoenix application-level timeout you listed (which is still good to remember!)

Re: Phoenix - Query Timeout

TerryP — Tue, 02 Feb 2016 01:53:57 GMT

There is an odd environment variable used for the HBase configuration directory. Try setting HBASE_CONF_PATH=/etc/hbase/conf in your shell session and see if the timeout value gets picked up.

Re: Phoenix - Query Timeout

bleonhardi — Tue, 02 Feb 2016 01:57:42 GMT

But he said that one was 90s before and changing it didn't help. Ah well lets see what he comes up with.

The issue might be that he changed it on the server side but should have changed on the client. Perhaps the default is 60s?

Re: Phoenix - Query Timeout

bleonhardi — Tue, 02 Feb 2016 01:59:41 GMT

Which it is: So most likely the phoenix client does not take the hbase-site.xml from ambari but uses default values.

hbase.rpc.timeout

Description

This is for the RPC layer to define how long HBase client applications take for a remote call to time out. It uses pings to check connections but will eventually throw a TimeoutException.

Default

60000

Re: Phoenix - Query Timeout

elserj — Tue, 02 Feb 2016 02:08:42 GMT

Oops. I missed that part 🙂

Re: Phoenix - Query Timeout

Jaraxal — Tue, 02 Feb 2016 03:31:21 GMT

I tried that already, as I indicated in my original post. It did not work.

Re: Phoenix - Query Timeout

elserj — Tue, 02 Feb 2016 03:38:45 GMT

Right, sorry about that, but that's near certainly where the error is coming from. The question on my mind is why sqlline isn't picking up the configuration value. @Terry 's suggestion is a good one. Depending on the version of HDP you're running, HBASE_CONF_DIR would also work.

Re: Phoenix - Query Timeout

Jaraxal — Tue, 02 Feb 2016 03:42:50 GMT

This was the issue. Well more specifically HBASE_CONF_DIR. Running phoenix_utils.py showed HBASE_CONF_PATH and HBASE_CONF_DIR as equal to ".". I set my HBASE_CONF_DIR environment variable to to use "/etc/hbase/conf" and now the query seems to work ok.

Re: Phoenix - Query Timeout

Jaraxal — Tue, 02 Feb 2016 03:44:08 GMT

I discovered several posts via Google that suggested the problem was hbase.rpc.timeout. When I changed the value of this via Ambari, it had no effect. As I responded below, the problem seemed to be that I was missing the HBASE_CONF_DIR environment variable so it was using the default settings instead of the Ambari configured settings.

Re: Phoenix - Query Timeout

sparepally1 — Fri, 20 May 2016 23:38:12 GMT

Hi @Terry Padgett @Josh Elser,

I am using DBVisulizer to connect to Phoenix. Mine is a kerberos enabled cluster and getting below error messages. Can suggestions?

An error occurred while establishing the connection: Long Message: callTimeout=600000, callDuration=620131: Details: Type: org.apache.phoenix.exception.PhoenixIOException Error Code: 101 SQL State: 08000

Not sure how I can pass conf directory for DBVisulizer (added all conf files to Resoucres directory of DbViz but didn't helped)

Re: Phoenix - Query Timeout

elserj — Fri, 17 Jun 2016 04:22:43 GMT

FWIW, this should be fixed in recent versions of Apache Phoenix and HDP.

Re: Phoenix - Query Timeout

ankitbeohar90 — Fri, 29 Jul 2016 17:39:24 GMT

Hi All,

I set phoenix timeout is 3hr, rpc 3hr and query alive is also 3hr but for bigger query whenever phoenix calltimeout occurs my hbase region server also goes down, I don not know what is happening.

Please help me out on this. @Terry Padgett

Re: Phoenix - Query Timeout

dgiri_india1989 — Tue, 04 Jul 2017 16:53:56 GMT

Below KB article helped me to resolve the issue.

https://community.hortonworks.com/content/supportkb/49037/phoenix-sqlline-query-on-larger-data-set-fails-wit.html

Re: Phoenix - Query Timeout

Simbosan — Thu, 29 Apr 2021 00:00:50 GMT

Frustrating that that link is hidden behind a 'paywall'. I have an account but I am not allowed to view without contacting sales