Phoenix - Query Timeout

[#1]

I have a table in HBase created via Phoenix. The table has approxmiately 20 million records. I'm connecting to Phoenix via:

phoenix-sqlline.py hbasemaster:2181:/hbase-unsecure

I'm trying to run a count as follows:

select count(columnname) from tablename;

When I run that SQL, Phoenix reports a timeout

org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed after attempts=36, exceptions:

java.net.SocketTimeoutException: callTimeout=60000, callDuration=60317: row ....

I've tried changing the hbase.rpc.timeout via Ambari, but that doesn't seem to be the issue. The default timeout in Ambari was set to 1m30s and I changed it to 2m. The timeout reported by Phoenix is 60s before and after the change, so I don't think that's the culprit anyway.

What setting do I need to change to allow for longer running queries? Is there something else that I should be looking at?

[#2 · Accepted solution]

There is an odd environment variable used for the HBase configuration directory. Try setting HBASE_CONF_PATH=/etc/hbase/conf in your shell session and see if the timeout value gets picked up.

[#3]

This was the issue. Well more specifically HBASE_CONF_DIR. Running phoenix_utils.py showed HBASE_CONF_PATH and HBASE_CONF_DIR as equal to ".". I set my HBASE_CONF_DIR environment variable to to use "/etc/hbase/conf" and now the query seems to work ok.

[#4]

FWIW, this should be fixed in recent versions of Apache Phoenix and HDP.

[#5]

Hi @Terry Padgett @Josh Elser,

I am using DBVisulizer to connect to Phoenix. Mine is a kerberos enabled cluster and getting below error messages. Can suggestions?

An error occurred while establishing the connection: Long Message: callTimeout=600000, callDuration=620131: Details: Type: org.apache.phoenix.exception.PhoenixIOException Error Code: 101 SQL State: 08000

Not sure how I can pass conf directory for DBVisulizer (added all conf files to Resoucres directory of DbViz but didn't helped)

[#6]

Hi All,

I set phoenix timeout is 3hr, rpc 3hr and query alive is also 3hr but for bigger query whenever phoenix calltimeout occurs my hbase region server also goes down, I don not know what is happening.

Please help me out on this. @Terry Padgett

[#7]

Below KB article helped me to resolve the issue.

https://community.hortonworks.com/content/supportkb/49037/phoenix-sqlline-query-on-larger-data-set-f...

[#8]

Frustrating that that link is hidden behind a 'paywall'.  I have an account but I am not allowed to view without contacting sales