When we try to run more complex Impala queries, we often run into the following error:
Couldn't open transport for worker29.ourdomain.com:22000(connect() failed: Connection timed out)
Sometimes there's only one node with that error message, sometimes there are 2-5.
There doesn't seem to be a network related problem - ping works, telnet to that port works, Impala debug ui works.
We tried setting vm.swappiness on the nodes from 60 to 0 - no positive effect. Same with switching vm.overcommit from 0 to 1.
Our setup:
- around 40 nodes, i7 quad core, 2-3TB, 1Gbit NIC, located in 5 different racks
- nodes have around 16-48GB ram, same amount of swap, which they alsmost never use
- OS: Ubuntu Linux 12.04
- CDH 5.1.0
- impalad version 1.4.0-cdh5-INTERNAL RELEASE (build e801bd8c0d134e783c2313c7dd422a5ad06591af)
- ~100TB HDFS storage
- we are using a HA proxy which points to the nodes with >32GB ram
- "workerlogs"-table is around 6-7TB big, partitioned by year > month > day and contains apache log-data
- almost 100% short circuit reads
Query Profile
Maybe you could give us a hint.
