I added some new nodes to our impala cluster.
On the new nodes, Impala continues to crash with the following error:
F0521 12:34:46.838728 5699 llvm-codegen.cc:112] LLVM hit fatal error: Cannot select: 0x7f62888c84c0: ch = store<ST1[%loc.i]> 0x7f622ebc5850, 0x1068b980, 0x1068b850, undef:i64 0x1068b980: i1,ch = load<LD1[%val_ptr.i.i27]> 0x7f626d3d1830, 0x7f622ebc54c0, undef:i64 0x7f622ebc54c0: i64 = add 0x1068bd10, Constant:i64<8> 0x1068bd10: i64,ch = CopyFromReg 0x7f626d3d1830, Register:i64 %vreg21 0x7f622ebc5ab0: i64 = Register %vreg21 0x7f62888c8d10: i64 = Constant<8> 0x7f61b3668260: i64 = undef 0x1068b850: i64,ch = CopyFromReg 0x7f626d3d1830, Register:i64 %vreg20 0x7f62888c8850: i64 = Register %vreg20 0x7f61b3668260: i64 = undef In function: _ZN6impala26PartitionedAggregationNode12ProcessBatchILb0EEENS_6StatusEPNS_8RowBatchENS_13TPrefetchMode4typeEPNS_12HashTableCtxE.1
The issue seems to be related to an hardware incompatibility with CPUs that have AVX-512 instructions.
Indeed the problem is present only on the new machines and from impala logs I notices that impala uses avx-512 cpu flags for generating at runtime code to obtain fully-optimized functions.
The problem is solved in the new version of impala IMPALA-6291 but at the moment we cannot upgrade the cluster.
Do you know if there is the possibility to modify the compiler options used by impala ?