Support Questions
Find answers, ask questions, and share your expertise
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- Cloudera Community
- Support
- Support Questions
- Who agreed with this topic
Who agreed with this topic
Impala ECONNRESET errors
Labels:
- Labels:
-
Apache Impala
New Contributor
Created on 10-01-2018 10:11 PM - edited 09-16-2022 06:45 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Once in a while, one of the impala daemons in my cluster of 35 nodes (5 coordinators, 30 executors) gets into this strange state where any query that hits that node errors out with a
RPC Error: Client for XXXXXXXX:22000 hit an unexpected exception: ECONNRESET, type: N6apache6thrift9transport19TTransportExceptionE, rpc: N6impala19TTransmitDataResultE, send: done
I've downloaded the impalad logs, here's the relevent logs -
I0927 07:42:49.859812 619158 query-state.cc:384] Instance completed. instance_id=194985c29be5ae3a:bde7c7d800000003 #in-flight=0 status=OK I0927 07:42:49.859828 619158 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=194985c29be5ae3a:bde7c7d800000000 refcnt=1 I0927 10:33:15.154033 269963 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.3 Port: 56052>Connection timed out I0927 10:33:15.154088 269963 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:16.113936 1838028 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.11 Port: 46586>Connection timed out I0927 10:33:16.114027 1838028 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:16.145911 1838240 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.3 Port: 56924>Connection timed out I0927 10:33:16.145941 1838240 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:16.177868 325713 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.7 Port: 33802>Connection timed out I0927 10:33:16.177892 325713 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:17.009917 1903962 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.17 Port: 34498>Connection timed out I0927 10:33:17.009943 1903962 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:17.073900 1453361 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.17 Port: 58190>Connection timed out I0927 10:33:17.073923 1453361 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:17.393921 1814127 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.17 Port: 42020>Connection timed out I0927 10:33:17.394212 1814127 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:17.969949 324535 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.3 Port: 52246>Connection timed out I0927 10:33:17.970016 324535 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.049926 77582 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.9 Port: 46380>Connection timed out I0927 10:33:18.049927 1814129 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.9 Port: 45942>Connection timed out I0927 10:33:18.049986 1814129 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.049989 77582 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.161929 1453362 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.11 Port: 46566>Connection timed out I0927 10:33:18.161967 1837961 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.5 Port: 48634>Connection timed out I0927 10:33:18.161967 1844307 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.5 Port: 49966>Connection timed out I0927 10:33:18.162014 1837961 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.162036 1844307 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.162322 1453362 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.193888 1842105 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.11 Port: 56060>Connection timed out I0927 10:33:18.193913 1842105 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.193914 1453360 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.11 Port: 46568>Connection timed out I0927 10:33:18.193936 1453360 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.225919 76857 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.9 Port: 51012>Connection timed out I0927 10:33:18.225965 76857 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.385915 1867082 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.9 Port: 57594>Connection timed out I0927 10:33:18.385918 1838188 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.9 Port: 38146>Connection timed out I0927 10:33:18.385942 1867082 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.385943 1838188 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.513911 1814132 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.3 Port: 40794>Connection timed out I0927 10:33:18.513934 1814132 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.513934 1842422 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.3 Port: 58036>Connection timed out I0927 10:33:18.514281 1842422 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.673913 324542 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.17 Port: 48202>Connection timed out I0927 10:33:18.673918 324536 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.17 Port: 48200>Connection timed out I0927 10:33:18.673918 326301 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.7 Port: 40638>Connection timed out I0927 10:33:18.673955 324536 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.673956 326301 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.673990 324542 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.705889 1814130 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.5 Port: 42222>Connection timed out I0927 10:33:18.705902 1842135 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.7 Port: 60594>Connection timed out I0927 10:33:18.705911 1814130 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.705910 1838205 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.15 Port: 59348>Connection timed out I0927 10:33:18.705929 1814111 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.15 Port: 51556>Connection timed out I0927 10:33:18.705942 1838205 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.705948 1814111 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.705965 1842135 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.737911 1842244 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.9 Port: 57730>Connection timed out I0927 10:33:18.737918 82908 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.9 Port: 39722>Connection timed out I0927 10:33:18.737931 1842244 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.737943 1831255 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.3 Port: 53228>Connection timed out I0927 10:33:18.737951 82908 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.737996 1831255 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.801892 76852 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.11 Port: 39724>Connection timed out I0927 10:33:18.801913 76852 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.865937 1897535 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.15 Port: 46274>Connection timed out I0927 10:33:18.865957 1897535 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.881888 324529 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.11 Port: 55214>Connection timed out I0927 10:33:18.882067 324529 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.961906 315246 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.9 Port: 38652>Connection timed out I0927 10:33:18.961921 1844273 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.9 Port: 34412>Connection timed out I0927 10:33:18.961928 315246 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.961944 1844273 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:18.993883 315582 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.11 Port: 41828>Connection timed out I0927 10:33:18.993911 315582 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.025907 315583 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.11 Port: 41830>Connection timed out I0927 10:33:19.025918 315584 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.11 Port: 41832>Connection timed out I0927 10:33:19.025929 315583 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.025940 315584 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.057925 324528 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.11 Port: 45430>Connection timed out I0927 10:33:19.057935 315580 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.11 Port: 41824>Connection timed out I0927 10:33:19.058008 324528 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.058010 315580 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.089908 1468713 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.1 Port: 52292>Connection timed out I0927 10:33:19.089932 1468713 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.249917 1436458 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.15 Port: 44558>Connection timed out I0927 10:33:19.250085 1436458 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.377894 1844393 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.3 Port: 59698>Connection timed out I0927 10:33:19.377908 1534739 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.7 Port: 34238>Connection timed out I0927 10:33:19.377916 1844393 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.377934 1534739 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.409929 315245 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.3 Port: 38672>Connection timed out I0927 10:33:19.409936 315244 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.3 Port: 38670>Connection timed out I0927 10:33:19.409946 315245 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.409960 315244 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.537945 365599 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.5 Port: 38118>Connection timed out I0927 10:33:19.537952 1838017 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.5 Port: 33806>Connection timed out I0927 10:33:19.537998 365599 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.538004 1838017 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.601943 269801 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.3 Port: 56050>Connection timed out I0927 10:33:19.602007 269801 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.826009 1838841 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.1 Port: 47194>Connection timed out I0927 10:33:19.826052 1838841 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.841915 1839082 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.1 Port: 47254>Connection timed out I0927 10:33:19.841938 1839082 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.857929 1463527 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.1 Port: 50882>Connection timed out I0927 10:33:19.857945 1814126 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.1 Port: 40394>Connection timed out I0927 10:33:19.857964 315249 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.1 Port: 48220>Connection timed out I0927 10:33:19.858026 315249 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.858031 1814126 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.858037 1463527 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.921974 1816539 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.3 Port: 36218>Connection timed out I0927 10:33:19.922116 1816539 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.953853 1543971 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.15 Port: 46546>Connection timed out I0927 10:33:19.953896 1543971 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.953904 1897541 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.15 Port: 52680>Connection timed out I0927 10:33:19.954236 1897541 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.985857 1842061 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.17 Port: 44728>Connection timed out I0927 10:33:19.985908 1842061 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:19.985913 31061 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.17 Port: 48918>Connection timed out I0927 10:33:19.985937 31061 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.017904 1449036 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.1 Port: 52120>Connection timed out I0927 10:33:20.017925 1449036 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.033918 1844395 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.1 Port: 59088>Connection timed out I0927 10:33:20.033960 1844395 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.049921 1814133 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.1 Port: 39408>Connection timed out I0927 10:33:20.049933 1838178 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.1 Port: 51434>Connection timed out I0927 10:33:20.049939 1468712 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.1 Port: 52288>Connection timed out I0927 10:33:20.049993 1814133 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.049994 1838178 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.050247 1468712 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.305917 324551 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.5 Port: 46290>Connection timed out I0927 10:33:20.305964 324551 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.369925 324534 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.7 Port: 49642>Connection timed out I0927 10:33:20.369968 324534 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.385934 1814140 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.7 Port: 38784>Connection timed out I0927 10:33:20.385936 324531 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.7 Port: 49638>Connection timed out I0927 10:33:20.385963 324531 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.385994 1814140 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.529896 77577 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.7 Port: 56106>Connection timed out I0927 10:33:20.529927 1814131 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.7 Port: 36494>Connection timed out I0927 10:33:20.529938 77577 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.529954 1814131 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.545898 82343 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.5 Port: 47536>Connection timed out I0927 10:33:20.545918 82343 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.561887 1842137 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.7 Port: 41148>Connection timed out I0927 10:33:20.561908 1842137 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.561939 365600 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.5 Port: 38120>Connection timed out I0927 10:33:20.561947 1839519 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.17 Port: 52220>Connection timed out I0927 10:33:20.561949 1839518 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.17 Port: 52154>Connection timed out I0927 10:33:20.561995 365600 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.561949 1534737 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.7 Port: 34236>Connection timed out I0927 10:33:20.562022 1814109 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.17 Port: 44620>Connection timed out I0927 10:33:20.562003 87132 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.7 Port: 36492>Connection timed out I0927 10:33:20.562044 1839518 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.562114 87132 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.562115 1839519 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.562125 1814109 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.562125 1534737 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.562242 1814219 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.5 Port: 55254>Connection timed out I0927 10:33:20.562659 1814219 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.753911 269997 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.7 Port: 34478>Connection timed out I0927 10:33:20.753954 269997 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.785934 324526 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.17 Port: 59720>Connection timed out I0927 10:33:20.785974 324537 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.17 Port: 59722>Connection timed out I0927 10:33:20.785990 324526 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.786031 324537 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.817904 315243 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.9 Port: 38650>Connection timed out I0927 10:33:20.817915 1842233 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.17 Port: 37362>Connection timed out I0927 10:33:20.817927 315243 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.817929 1837967 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.9 Port: 32920>Connection timed out I0927 10:33:20.817937 1842233 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.817936 324525 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.17 Port: 59718>Connection timed out I0927 10:33:20.817958 324525 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.818171 1837967 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.849879 315248 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.9 Port: 38654>Connection timed out I0927 10:33:20.849895 1814117 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.9 Port: 54992>Connection timed out I0927 10:33:20.849901 315248 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.850059 1814117 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:20.850208 1834351 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.5 Port: 34756>Connection timed out I0927 10:33:20.850451 1834351 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.073925 1842104 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.1 Port: 42850>Connection timed out I0927 10:33:21.073956 1842104 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.105901 1814110 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.1 Port: 34574>Connection timed out I0927 10:33:21.105904 324524 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.1 Port: 36752>Connection timed out I0927 10:33:21.105922 1814110 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.106379 324524 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.137897 338780 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.1 Port: 37296>Connection timed out I0927 10:33:21.137907 338779 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.1 Port: 37294>Connection timed out I0927 10:33:21.137933 338779 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.138103 338780 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.233893 31059 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.17 Port: 48916>Connection timed out I0927 10:33:21.233922 31059 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.265902 1814123 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.17 Port: 36382>Connection timed out I0927 10:33:21.265921 31060 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.17 Port: 48914>Connection timed out I0927 10:33:21.265949 1814123 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.265957 31060 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.329864 1899080 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.7 Port: 47528>Connection timed out I0927 10:33:21.329882 1899080 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.362359 1814116 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.7 Port: 49670>Connection timed out I0927 10:33:21.362380 1814116 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.489897 1839789 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.15 Port: 37034>Connection timed out I0927 10:33:21.489900 327533 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.15 Port: 58620>Connection timed out I0927 10:33:21.489924 327533 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.490015 1839789 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.553889 1814113 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.15 Port: 58514>Connection timed out I0927 10:33:21.553961 1814113 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.617873 324549 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.5 Port: 46286>Connection timed out I0927 10:33:21.617905 324549 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.681910 1543972 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.15 Port: 46542>Connection timed out I0927 10:33:21.681915 1543973 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.15 Port: 46540>Connection timed out I0927 10:33:21.681953 1543972 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.681978 1543973 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.697896 1838048 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.15 Port: 38952>Connection timed out I0927 10:33:21.697916 1838048 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.697954 1814115 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.15 Port: 59992>Connection timed out I0927 10:33:21.697980 1814115 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.873911 1814122 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.9 Port: 33124>Connection timed out I0927 10:33:21.873914 1866647 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.9 Port: 45950>Connection timed out I0927 10:33:21.873914 1478041 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.9 Port: 50914>Connection timed out I0927 10:33:21.873947 1866647 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.873956 1478041 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.873956 76856 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.9 Port: 51008>Connection timed out I0927 10:33:21.873962 1814122 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:21.873978 76856 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:22.033855 1814125 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.9 Port: 49382>Connection timed out I0927 10:33:22.033874 1814125 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:22.257892 315581 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.11 Port: 41826>Connection timed out I0927 10:33:22.257918 1814120 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.11 Port: 39200>Connection timed out I0927 10:33:22.257939 315581 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:22.257944 1814120 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:22.417894 76781 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.11 Port: 39702>Connection timed out I0927 10:33:22.417915 76781 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:22.449894 1814112 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.11 Port: 47400>Connection timed out I0927 10:33:22.449918 1814112 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:22.481889 1838513 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.11 Port: 54426>Connection timed out I0927 10:33:22.481909 1838513 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:22.833919 324533 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.3 Port: 52840>Connection timed out I0927 10:33:22.833925 1844101 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.3 Port: 60416>Connection timed out I0927 10:33:22.833950 324533 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:22.833956 1844101 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:22.897928 1814128 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.3 Port: 57320>Connection timed out I0927 10:33:22.897948 1814128 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:23.313886 1814119 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.17 Port: 57914>Connection timed out I0927 10:33:23.313907 1814119 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:23.329917 1837807 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.17 Port: 48320>Connection timed out I0927 10:33:23.329936 1837807 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:23.505882 269917 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.3 Port: 55072>Connection timed out I0927 10:33:23.505901 269917 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:23.537914 1449793 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.1 Port: 51952>Connection timed out I0927 10:33:23.537932 1449793 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:23.602130 1814045 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.7 Port: 33674>Connection timed out I0927 10:33:23.602200 1814045 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:23.665917 326302 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.7 Port: 40640>Connection timed out I0927 10:33:23.665937 326302 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:23.793905 269863 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.7 Port: 34474>Connection timed out I0927 10:33:23.793926 269863 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:23.857887 324527 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.11 Port: 55212>Connection timed out I0927 10:33:23.857911 324527 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:23.889899 1838180 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.11 Port: 57706>Connection timed out I0927 10:33:23.889909 1814114 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.11 Port: 50970>Connection timed out I0927 10:33:23.889927 1838180 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:23.889928 1814114 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.081909 324532 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.1 Port: 46142>Connection timed out I0927 10:33:24.081917 1839533 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.1 Port: 53522>Connection timed out I0927 10:33:24.081941 324532 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.081944 1839533 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.161932 325720 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.4.3 Port: 51484>Connection timed out I0927 10:33:24.161969 325720 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.209920 315247 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.1 Port: 48222>Connection timed out I0927 10:33:24.209954 315247 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.241964 1816268 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.7 Port: 59418>Connection timed out I0927 10:33:24.241991 1816268 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.401888 269714 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.3 Port: 55070>Connection timed out I0927 10:33:24.401908 269714 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.481864 77922 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.7 Port: 47942>Connection timed out I0927 10:33:24.481881 77922 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.481904 1838220 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.3.7 Port: 59964>Connection timed out I0927 10:33:24.481928 1838220 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.593876 324547 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.5 Port: 46284>Connection timed out I0927 10:33:24.593899 324547 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.609899 328110 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.1 Port: 60440>Connection timed out I0927 10:33:24.609921 328110 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.625880 1814118 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.1 Port: 46162>Connection timed out I0927 10:33:24.625898 324550 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.6.5 Port: 46288>Connection timed out I0927 10:33:24.625902 324530 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.1 Port: 46140>Connection timed out I0927 10:33:24.625916 1839532 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.1 Port: 53518>Connection timed out I0927 10:33:24.625924 324550 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.625928 324530 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.625939 1839532 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.626058 1814118 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.801890 1814124 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.7.11 Port: 48106>Connection timed out I0927 10:33:24.801910 1814124 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:24.945883 1812108 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.17 Port: 50668>Connection timed out I0927 10:33:24.945907 1812108 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 10:33:25.081912 1812107 thrift-util.cc:123] TSocket::read() recv() <Host: 10.73.5.17 Port: 50666>Connection timed out I0927 10:33:25.081928 1812107 thrift-util.cc:123] TAcceptQueueServer client died: ETIMEDOUT I0927 13:32:57.825640 1316077 impala-internal-service.cc:44] ExecQueryFInstances(): query_id=c14aa7f9fbbe7466:9066c0f500000000 I0927 13:32:57.825662 1316077 query-exec-mgr.cc:46] StartQueryFInstances() query_id=c14aa7f9fbbe7466:9066c0f500000000 coord=hkccs022.XXX:22000 I0927 13:32:57.826149 1316077 query-state.cc:173] Buffer pool limit for c14aa7f9fbbe7466:9066c0f500000000: 109951162777 I0927 13:32:57.826205 1316077 initial-reservations.cc:60] Successfully claimed initial reservations (70.00 MB) for query c14aa7f9fbbe7466:9066c0f500000000 I0927 13:32:57.826331 1316078 query-state.cc:286] StartFInstances(): query_id=c14aa7f9fbbe7466:9066c0f500000000 #instances=2 I0927 13:32:57.827217 1316078 query-state.cc:299] descriptor table for query=c14aa7f9fbbe7466:9066c0f500000000 tuples: Tuple(id=2 size=93 slots=[Slot(id=16 type=STRING col_path=[] offset=0 null=(offset=92 mask=1) slot_idx=0 field_idx=-1), Slot(id=17 type=STRING col_path=[] offset=16 null=(offset=92 mask=2) slot_idx=1 field_idx=-1), Slot(id=18 type=STRING col_path=[] offset=32 null=(offset=92 mask=4) slot_idx=2 field_idx=-1), Slot(id=19 type=STRING col_path=[] offset=48 null=(offset=92 mask=8) slot_idx=3 field_idx=-1), Slot(id=20 type=INT col_path=[] offset=88 null=(offset=92 mask=80) slot_idx=7 field_idx=-1), Slot(id=21 type=BIGINT col_path=[] offset=64 null=(offset=92 mask=10) slot_idx=4 field_idx=-1), Slot(id=22 type=BIGINT col_path=[] offset=72 null=(offset=92 mask=20) slot_idx=5 field_idx=-1), Slot(id=23 type=BIGINT col_path=[] offset=80 null=(offset=92 mask=40) slot_idx=6 field_idx=-1)] tuple_path=[]) Tuple(id=1 size=101 slots=[Slot(id=8 type=STRING col_path=[] offset=0 null=(offset=100 mask=1) slot_idx=0 field_idx=-1), Slot(id=9 type=STRING col_path=[] offset=16 null=(offset=100 mask=2) slot_idx=1 field_idx=-1), Slot(id=10 type=STRING col_path=[] offset=32 null=(offset=100 mask=4) slot_idx=2 field_idx=-1), Slot(id=11 type=STRING col_path=[] offset=48 null=(offset=100 mask=8) slot_idx=3 field_idx=-1), Slot(id=12 type=INT col_path=[] offset=96 null=(offset=100 mask=80) slot_idx=7 field_idx=-1), Slot(id=13 type=STRING col_path=[] offset=64 null=(offset=100 mask=10) slot_idx=4 field_idx=-1), Slot(id=14 type=BIGINT col_path=[] offset=80 null=(offset=100 mask=20) slot_idx=5 field_idx=-1), Slot(id=15 type=BIGINT col_path=[] offset=88 null=(offset=100 mask=40) slot_idx=6 field_idx=-1)] tuple_path=[]) Tuple(id=0 size=93 slots=[Slot(id=0 type=STRING col_path=[1] offset=0 null=(offset=92 mask=1) slot_idx=0 field_idx=-1), Slot(id=1 type=STRING col_path=[3] offset=16 null=(offset=92 mask=2) slot_idx=1 field_idx=-1), Slot(id=2 type=STRING col_path=[4] offset=32 null=(offset=92 mask=4) slot_idx=2 field_idx=-1), Slot(id=3 type=STRING col_path=[6] offset=48 null=(offset=92 mask=8) slot_idx=3 field_idx=-1), Slot(id=4 type=BIGINT col_path=[11] offset=64 null=(offset=92 mask=10) slot_idx=4 field_idx=-1), Slot(id=5 type=BIGINT col_path=[13] offset=72 null=(offset=92 mask=20) slot_idx=5 field_idx=-1), Slot(id=6 type=BIGINT col_path=[5] offset=80 null=(offset=92 mask=40) slot_idx=6 field_idx=-1), Slot(id=7 type=INT col_path=[0] offset=88 null=(offset=92 mask=80) slot_idx=7 field_idx=-1)] tuple_path=[]) I0927 13:32:57.827322 1316079 query-state.cc:377] Executing instance. instance_id=c14aa7f9fbbe7466:9066c0f50000002a fragment_idx=1 per_fragment_instance_idx=14 coord_state_idx=13 #in-flight=1 I0927 13:32:57.827414 1316080 query-state.cc:377] Executing instance. instance_id=c14aa7f9fbbe7466:9066c0f50000000f fragment_idx=2 per_fragment_instance_idx=14 coord_state_idx=13 #in-flight=2 I0927 13:32:57.827504 1316080 hdfs-scan-node.cc:160] Max row batch queue size for scan node '0' in fragment instance 'c14aa7f9fbbe7466:9066c0f50000000f': 170 I0927 13:32:57.854048 1316078 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=c14aa7f9fbbe7466:9066c0f500000000 refcnt=3 I0927 13:33:03.756124 1316077 impala-internal-service.cc:63] CancelQueryFInstances(): query_id=c14aa7f9fbbe7466:9066c0f500000000 I0927 13:33:03.756161 1316077 query-exec-mgr.cc:92] QueryState: query_id=c14aa7f9fbbe7466:9066c0f500000000 refcnt=3 I0927 13:33:03.756408 1316077 query-state.cc:396] Cancel: query_id=c14aa7f9fbbe7466:9066c0f500000000 I0927 13:33:03.756417 1316077 data-stream-mgr.cc:270] cancelling all streams for fragment=c14aa7f9fbbe7466:9066c0f50000000f I0927 13:33:03.756419 1316077 data-stream-mgr.cc:270] cancelling all streams for fragment=c14aa7f9fbbe7466:9066c0f50000002a I0927 13:33:03.756424 1316077 data-stream-recvr.cc:235] cancelled stream: fragment_instance_id_=c14aa7f9fbbe7466:9066c0f50000002a node_id=2 I0927 13:33:03.756438 1316077 query-exec-mgr.cc:149] ReleaseQueryState(): query_id=c14aa7f9fbbe7466:9066c0f500000000 refcnt=3 I0927 13:33:03.757410 1316079 thrift-util.cc:123] TSocket::read() recv() <Host: hkccs022.XXX Port: 22000>Connection reset by peer I0927 13:33:03.757422 1316080 thrift-util.cc:123] TSocket::read() recv() <Host: hkccs022.XXX Port: 22000>Connection reset by peer I0927 13:33:03.757941 1316079 client-cache.h:304] RPC Error: Client for hkccs022.XXX:22000 hit an unexpected exception: ECONNRESET, type: N6apache6thrift9transport19TTransportExceptionE, rpc: N6impala23TReportExecStatusResultE, send: done I0927 13:33:03.757948 1316080 client-cache.h:304] RPC Error: Client for hkccs022.XXX:22000 hit an unexpected exception: ECONNRESET, type: N6apache6thrift9transport19TTransportExceptionE, rpc: N6impala23TReportExecStatusResultE, send: done
Any ideas what's causing this? What additional information can I provide to help with this?
