Reply
Contributor
Posts: 49
Registered: ‎04-21-2015

some commands' execution intermittently takes too long time

Hello

 

I am using impalad version 2.2.0-cdh5.4.5 community edition. I have a problem that intermittently occurs. 

Some commands' execution takes too long time. The commands are physical database management commands not select, insert commands. 

 

show create table <table name>  (it takes more than 15 mins today, so I terminated with Ctrl+C)

create table <table name>,

create table <table name> like <other table name>

drop table <table name>    (even empty table's drop operation may takes more than 20 mins today, so I had to terminate it too)

 

There is no error while command executing.

Following lines show today's WARNING and ERROR entries in log files. I am not sure the lines are related or not 

 

 

impala@hdpdnode03:/var/log/impalad$ grep 0725 impalad.ERROR
impala@hdpdnode03:/var/log/impalad$ grep 0725 impalad.WARNING
java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.160.15.142:60725 remote=/10.160.15.142:50010]
W0623 05:35:54.195564 58466 DFSInputStream.java:657] Failed to connect to /10.160.15.142:50010 for block, add to deadNodes and continue. java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.160.15.142:60725 remote=/10.160.15.142:50010]
java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.160.15.142:60725 remote=/10.160.15.142:50010]
java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.160.15.142:60725 remote=/10.160.15.142:50010]
W0624 17:34:47.654734 58470 DFSInputStream.java:657] Failed to connect to /10.160.15.142:50010 for block, add to deadNodes and continue. java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.160.15.142:60725 remote=/10.160.15.142:50010]
java.net.SocketTimeoutException: 60000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/10.160.15.142:60725 remote=/10.160.15.142:50010]
W0725 00:03:12.909853 58553 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 01:03:15.475993 58517 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 02:03:31.348456 58544 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 03:03:24.569105 58506 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 04:03:17.290172 58533 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 05:03:10.648597 58511 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 06:03:37.333330 58555 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 07:03:17.709178 58515 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 08:03:11.393440 58544 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 09:03:14.093744 58506 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 10:03:20.556089 58523 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 11:03:23.019448 58533 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 12:03:12.509655 58519 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 13:03:14.731720 58509 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 14:03:21.690156 58502 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
W0725 15:03:16.251411 58525 HdfsScanNode.java:784] Per-host mem cost 8.25GB exceeded per-host upper bound 7.50GB.
impala@hdpdnode03:/var/log/impalad$

 

 

Do you have any idea why I am waiting at this commands ?

 

Thanks

Highlighted
Posts: 642
Topics: 3
Kudos: 121
Solutions: 67
Registered: ‎08-16-2016

Re: some commands' execution intermittently takes too long time

The log entries have to do with reading from HDFS.

Normally, metadata option like the ones you mentioned go through the Statestore, CatalogD, and HMS. I would check out the threads on each and the ImpalaD you are running the commands from to see what else is running. It is possible that one of these service is slow; it is also possible, based on the log entries, that reading from HDFS slow and the other threads are waiting on the one reading from HDFS.
Announcements

Our community is getting a little larger. And a lot better.


Learn More about the Cloudera and Hortonworks community merger planned for late July and early August.