Time out in creating table in Phoenix based on table in HBase

[#1]

Hi,

I'm running sandbox 2.6 and I'm trying to create a table in Phoenix based on a HBase table using the following code:

create table "eco_2015_2016_3" ("row" VARCHAR primary key,"year"."year" VARCHAR,"year"."year_title" VARCHAR, "month"."month" VARCHAR,"month"."month_title" VARCHAR, "company"."company" VARCHAR,"company"."company_title" VARCHAR, "account"."account" VARCHAR,"account"."account_title" VARCHAR, "responsibility"."responsibility" VARCHAR,"responsibility"."responsibility_title" VARCHAR, "function"."function" VARCHAR,"function"."function_title" VARCHAR, "activity"."activity" VARCHAR,"activity"."activity_title" VARCHAR, "project_type"."project_type" VARCHAR,"project_type"."project_type_title" VARCHAR, "project"."project" VARCHAR,"project"."project_title" VARCHAR, "object"."object" VARCHAR,"object"."object_title" VARCHAR, "counterpart"."counterpart" VARCHAR,"counterpart"."counterpart_title" VARCHAR, "free"."free" VARCHAR,"free"."free_title" VARCHAR, "amount"."amount" VARCHAR);

and I get the following error:

org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed to get result within timeout, timeout=60000ms

The same code works for a smaller amount of data so I guess that it times out because of the large data set. I have increased the HBase RPC Timeout to 2 minutes and 30 seconds and the Phoenix query timeout to 600 000 ms so neither of these seem to be cause of the time out (since it says 60 000 ms). I found this post about the things that needs to be set to not get a time out https://community.hortonworks.com/content/supportkb/49037/phoenix-sqlline-query-on-larger-data-set-f... but I don't get how to find these properties in Ambari using the sandbox. Would really appreciate it if someone could explain that to me.

Thanks,

//Rebecca

Added the full error:

org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed
 to get result within timeout, timeout=60000ms                                                          
        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)                 
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:771)    
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:714)    
        at org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)   
        at org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
        at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)          
        at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingRe
sultIterator.java:64)                                                                                   
        at org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResult
Iterator.java:39)                                                                                       
        at org.apache.phoenix.compile.PostDDLCompiler$2.execute(PostDDLCompiler.java:285)               
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.updateData(ConnectionQueryServicesImpl.j
ava:2823)                                                                                               
        at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:882)                
        at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:194)       
        at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:343)                   
        at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331)                   
        at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)                                   
        at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:329)          
        at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1440)                 
        at sqlline.Commands.execute(Commands.java:822)                                                  
        at sqlline.Commands.sql(Commands.java:732)                                                      
        at sqlline.SqlLine.dispatch(SqlLine.java:808)                                                   
        at sqlline.SqlLine.begin(SqlLine.java:681)                                                      
        at sqlline.SqlLine.start(SqlLine.java:398)                                                      
        at sqlline.SqlLine.main(SqlLine.java:292)                                                       
Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Fai
led to get result within timeout, timeout=60000ms                                                       
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)                                  
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)                                     
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:766)    
        ... 21 more                                                                                     
Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed to get result within timeout, timeout
=60000ms                                                                                                
        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)                 
        at org.apache.phoenix.iterate.TableResultIterator.initScanner(TableResultIterator.java:203)     
        at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:108)              
        at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:103)              
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)                                     
        at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)         
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)              
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)              
        at java.lang.Thread.run(Thread.java:748)                                                        
Caused by: java.io.IOException: Failed to get result within timeout, timeout=60000ms                    
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.j
ava:206)                                                                                                
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.j
ava:60)                                                                                                 
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:20
0)                                                                                                      
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:326)                    
        at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:301)             
        at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.ja
va:166)                                                                                                 
        at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:161)                  
        at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)                            
        at org.apache.phoenix.iterate.TableResultIterator.initScanner(TableResultIterator.java:199)     
        ... 7 more   

[#2]

Please share the full stacktrace of error which is populated after increasing timeout.

[#3]

@schhabra Here is the full error:

org.apache.phoenix.exception.PhoenixIOException: org.apache.phoenix.exception.PhoenixIOException: Failed
 to get result within timeout, timeout=60000ms                                                          
        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)                 
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:771)    
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:714)    
        at org.apache.phoenix.iterate.ConcatResultIterator.getIterators(ConcatResultIterator.java:50)   
        at org.apache.phoenix.iterate.ConcatResultIterator.currentIterator(ConcatResultIterator.java:97)
        at org.apache.phoenix.iterate.ConcatResultIterator.next(ConcatResultIterator.java:117)          
        at org.apache.phoenix.iterate.BaseGroupedAggregatingResultIterator.next(BaseGroupedAggregatingRe
sultIterator.java:64)                                                                                   
        at org.apache.phoenix.iterate.UngroupedAggregatingResultIterator.next(UngroupedAggregatingResult
Iterator.java:39)                                                                                       
        at org.apache.phoenix.compile.PostDDLCompiler$2.execute(PostDDLCompiler.java:285)               
        at org.apache.phoenix.query.ConnectionQueryServicesImpl.updateData(ConnectionQueryServicesImpl.j
ava:2823)                                                                                               
        at org.apache.phoenix.schema.MetaDataClient.createTable(MetaDataClient.java:882)                
        at org.apache.phoenix.compile.CreateTableCompiler$2.execute(CreateTableCompiler.java:194)       
        at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:343)                   
        at org.apache.phoenix.jdbc.PhoenixStatement$2.call(PhoenixStatement.java:331)                   
        at org.apache.phoenix.call.CallRunner.run(CallRunner.java:53)                                   
        at org.apache.phoenix.jdbc.PhoenixStatement.executeMutation(PhoenixStatement.java:329)          
        at org.apache.phoenix.jdbc.PhoenixStatement.execute(PhoenixStatement.java:1440)                 
        at sqlline.Commands.execute(Commands.java:822)                                                  
        at sqlline.Commands.sql(Commands.java:732)                                                      
        at sqlline.SqlLine.dispatch(SqlLine.java:808)                                                   
        at sqlline.SqlLine.begin(SqlLine.java:681)                                                      
        at sqlline.SqlLine.start(SqlLine.java:398)                                                      
        at sqlline.SqlLine.main(SqlLine.java:292)                                                       
Caused by: java.util.concurrent.ExecutionException: org.apache.phoenix.exception.PhoenixIOException: Fai
led to get result within timeout, timeout=60000ms                                                       
        at java.util.concurrent.FutureTask.report(FutureTask.java:122)                                  
        at java.util.concurrent.FutureTask.get(FutureTask.java:206)                                     
        at org.apache.phoenix.iterate.BaseResultIterators.getIterators(BaseResultIterators.java:766)    
        ... 21 more                                                                                     
Caused by: org.apache.phoenix.exception.PhoenixIOException: Failed to get result within timeout, timeout
=60000ms                                                                                                
        at org.apache.phoenix.util.ServerUtil.parseServerException(ServerUtil.java:111)                 
        at org.apache.phoenix.iterate.TableResultIterator.initScanner(TableResultIterator.java:203)     
        at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:108)              
        at org.apache.phoenix.iterate.ParallelIterators$1.call(ParallelIterators.java:103)              
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)                                     
        at org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run(JobManager.java:183)         
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)              
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)              
        at java.lang.Thread.run(Thread.java:748)                                                        
Caused by: java.io.IOException: Failed to get result within timeout, timeout=60000ms                    
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.j
ava:206)                                                                                                
        at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.j
ava:60)                                                                                                 
        at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:20
0)                                                                                                      
        at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:326)                    
        at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:301)             
        at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.ja
va:166)                                                                                                 
        at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:161)                  
        at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:794)                            
        at org.apache.phoenix.iterate.TableResultIterator.initScanner(TableResultIterator.java:199)     
        ... 7 more   

[#4]

Current error shows query is failing while scan.

Please try increasing values for below properties.

hbase.rpc.timeout

phoenix.query.timeoutMs

hbase.client.scanner.timeout.period

If properties are not present in hbase-site.xml, add them.

Default value of these parameters is 60000ms.