We faced an issue in our Impala environment:
Refresh to any of the impala tables were getting failed while when we query any tables like select count * , it was working.
We found that the catalogd lost its connection to statestored. The catalog logs show:
Connection with statestore lost, entering recovery mode.
Trying to re-register with statestore, attempt: 1
TSocket::write_partial() send() <Host: ipaddress Port: 53164>Broken pipe
While the statestored logs shows:
Missed subscriber (catalog-server@servername:26000) heartbeat deadline by 8971ms, consider increasing --statestore_heartbeat_frequency_ms (currently 1000)
Can someone advise what could be the issue.
Both Statstore and catalog daemons are on the same node.