error processing the impalad catalog update. Requesting a full topic update to recover

[#1]

Version: Cloudera Express 5.8.2 (#17 built by jenkins on 20160916-1426 git: d23c620f3a3bbd85d8511d6ebba49beaaab14b75)

Parcel Name Version Status Actions
CDH 5 5.8.2-1.cdh5.8.2.p0.3 Distributed, Activated

$ uname -a
Linux hostname_redacted 2.6.32-642.6.2.el6.x86_64 #1 SMP Mon Oct 24 10:22:33 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux

We are seeing this error 2 to 10 times per day.  What is this?

Thank you.

Log file created at: 2016/11/03 18:32:33
Running on machine: hostname_redacted
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E1103 18:32:33.125335 3940 logging.cc:118] stderr will be logged to this file.
E1104 03:46:23.122285 4755 impala-server.cc:1339] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
E1106 03:36:54.086156 4755 impala-server.cc:1339] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
E1108 08:16:41.835364 4764 fe-support.cc:308] RPC client failed to connect: Couldn't open transport for hostname_redacted:26000 (connect() failed: Connection refused)

E1108 08:16:42.003527 4764 fe-support.cc:308] RPC client failed to connect: Couldn't open transport for hostname_redacted:26000 (connect() failed: Connection refused)

E1108 08:16:59.126739 4755 impala-server.cc:1339] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
E1108 09:32:24.577546 4766 fe-support.cc:308] RPC client failed to connect: Couldn't open transport for hostname_redacted:26000 (connect() failed: Connection refused)

E1108 09:32:24.595510 4766 fe-support.cc:308] RPC client failed to connect: Couldn't open transport for hostname_redacted:26000 (connect() failed: Connection refused)

E1108 09:32:40.664857 4755 impala-server.cc:1339] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
E1109 03:21:00.335299 4755 impala-server.cc:1339] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
E1109 07:47:05.972446 4755 impala-server.cc:1339] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
E1112 01:14:54.006551 4755 impala-server.cc:1339] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
E1112 03:15:03.835086 4755 impala-server.cc:1339] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()
E1113 06:14:55.153087 4755 impala-server.cc:1339] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()

[#2]

This would typically happen if the catalog daemon was restarted.

[#3]

We are getting stuck 'insert' impala operations, which immediately afterwards log this error.

I1203 03:44:12.598379 25140 impala-beeswax-server.cc:171] query(): query=insert into table AParquetTable
(params) partition select (more params) from AnExternalCsvTable

I1203 03:44:12.599138 25140 impala-beeswax-server.cc:544] TClientRequest.queryOptions: TQueryOptions { params }

I1203 03:44:12.600991 25140 Frontend.java:875] analyze query insert into table AParquetTable
(params) partition select (more params) from AnExternalCsvTable

I1203 03:44:12.607447 25140 Frontend.java:819] Requesting prioritized load of table(s): default.aparquettable

I1203 03:44:43.429272 25124 jni-util.cc:166] com.cloudera.impala.catalog.CatalogException: Detected catalog service ID change. Aborting updateCatalog()
at com.cloudera.impala.catalog.ImpaladCatalog.updateCatalog(ImpaladCatalog.java:120)
at com.cloudera.impala.service.Frontend.updateCatalogCache(Frontend.java:227)
at com.cloudera.impala.service.JniFrontend.updateCatalogCache(JniFrontend.java:180)

I1203 03:44:43.431560 25124 status.cc:111] CatalogException: Detected catalog service ID change. Aborting updateCatalog()
@@@ 0x80f2c9 (unknown)
@ 0xb37c30 (unknown)
@ 0xa4e5cf (unknown)
@ 0xa68ea9 (unknown)
@ 0xb00a02 (unknown)
@ 0xb068f3 (unknown)
@ 0xd2bed8 (unknown)
@ 0xd2b114 (unknown)
@ 0x7dc26c (unknown)
@ 0x1b208bf (unknown)
@ 0x9b0a39 (unknown)
@ 0x9b1492 (unknown)
@ 0xb89327 (unknown)
@@ 0xb89c64 (unknown)
@ 0xdee99a (unknown)
@ 0x3d05e07aa1 (unknown)
@ 0x3d05ae893d (unknown)

E1203 03:44:43.431589 25124 impala-server.cc:1339] There was an error processing the impalad catalog update. Requesting a full topic update to recover: CatalogException: Detected catalog service ID change. Aborting updateCatalog()

I1203 04:47:33.657946 25141 impala-server.cc:1676] Connection from client 151.214.100.169:41748 closed, closing 1 associated session(s)

I1203 04:47:33.658761 25141 status.cc:111] Session closed
@@ 0x80f2c9 (unknown)
@ 0xa727b9 (unknown)
@ 0xa72cb8 (unknown)
@ 0x9aac90 (unknown)
@ 0x1b1ca93 (unknown)
@ 0x1b03b09 (unknown)
@ 0x9b0a39 (unknown)
@ 0x9b1492 (unknown)
@ 0xb89327 (unknown)
@ 0xb89c64 (unknown)
@ 0xdee99a (unknown)
@ 0x3d05e07aa1 (unknown)
@ 0x3d05ae893d (unknown)

[#4 · Accepted solution]

increased catalog server heap resolved this problem.