Impala Crashes on CPU with AVX-512

[#1]

Hi all,

I added some new nodes to our impala cluster.

On the new nodes, Impala continues to crash with the following error:

F0521 12:34:46.838728  5699 llvm-codegen.cc:112] LLVM hit fatal error: Cannot select: 0x7f62888c84c0: ch = store<ST1[%loc.i]> 0x7f622ebc5850, 0x1068b980, 0x1068b850, undef:i64
  0x1068b980: i1,ch = load<LD1[%val_ptr.i.i27]> 0x7f626d3d1830, 0x7f622ebc54c0, undef:i64
    0x7f622ebc54c0: i64 = add 0x1068bd10, Constant:i64<8>
      0x1068bd10: i64,ch = CopyFromReg 0x7f626d3d1830, Register:i64 %vreg21
        0x7f622ebc5ab0: i64 = Register %vreg21
      0x7f62888c8d10: i64 = Constant<8>
    0x7f61b3668260: i64 = undef
  0x1068b850: i64,ch = CopyFromReg 0x7f626d3d1830, Register:i64 %vreg20
    0x7f62888c8850: i64 = Register %vreg20
  0x7f61b3668260: i64 = undef
In function: _ZN6impala26PartitionedAggregationNode12ProcessBatchILb0EEENS_6StatusEPNS_8RowBatchENS_13TPrefetchMode4typeEPNS_12HashTableCtxE.1

The issue seems to be related to an hardware incompatibility with CPUs that have AVX-512 instructions.

Indeed the problem is present only on the new machines and from impala logs I notices that impala uses avx-512 cpu flags for generating at runtime code to obtain fully-optimized functions.

The problem is solved in the new version of impala IMPALA-6291 but at the moment we cannot upgrade the cluster.

Do you know if there is the possibility to modify the compiler options used by impala ? 

Thanks

Alessio

[#2 · Accepted solution]

Solved upgrading Impala to version 2.12