Impala critical bug in CDH 5.14.0

[#1]

Since upgrading from 5.13 to 5.14 we have found that certain types of queries will consistently crash all Impala daemons in the cluster (at least all those which are running fragments of the queries in question).

These queries ran for 6 months+ as scheduled jobs on versions prior to 5.14, so it is not an issue with the query or table.

The queries that trigger the crash are accessing nested fields, of the format:

   SELECT a.topLevelField, b.priceFromNestedField FROM db.table a LEFT JOIN a.nestedField b

The underlying tables are all parquet, written by Spark. It happens on multiple tables, and multiple partitions, so is not a corrupt parquet file. It doesn't affect nested fields in most other tables.

When the query fails, the logs show the following error:

tcmalloc: large alloc 1073741824 bytes == 0x7f62666e8000 @ 0x21c6f07 0x20483f2
Wrote minidump to /var/log/impala-minidumps/impalad/9fcf5df1-13c0-47bd-35ceaa92-72438843.dmp

Does anyone else have the same issue?

Thanks

[#2]

7f62540aa000-7f6254185000 r-xp 00000000 fd:01 162121                     /lib64/libkrb5.so.3.3
7f6254185000-7f6254385000 ---p 000db000 fd:01 162121                     /lib64/libkrb5.so.3.3
7f6254385000-7f625438f000 r--p 000db000 fd:01 162121                     /lib64/libkrb5.so.3.3
7f625438f000-7f6254391000 rw-p 000e5000 fd:01 162121                     /lib64/libkrb5.so.3.3
7f6254391000-7f625454b000 r-xp 00000000 fd:01 136060                     /usr/lib64/libcrypto.so.1.0.1e
7f625454b000-7f625474a000 ---p 001ba000 fd:01 136060                     /usr/lib64/libcrypto.so.1.0.1e
7f625474a000-7f6254765000 r--p 001b9000 fd:01 136060                     /usr/lib64/libcrypto.so.1.0.1e
7f6254765000-7f6254771000 rw-p 001d4000 fd:01 136060                     /usr/lib64/libcrypto.so.1.0.1e
7f6254771000-7f6254775000 rw-p 00000000 00:00 0 
7f6254775000-7f62547d7000 r-xp 00000000 fd:01 142269                     /usr/lib64/libssl.so.1.0.1e
7f62547d7000-7f62549d7000 ---p 00062000 fd:01 142269                     /usr/lib64/libssl.so.1.0.1e
7f62549d7000-7f62549db000 r--p 00062000 fd:01 142269                     /usr/lib64/libssl.so.1.0.1e
7f62549db000-7f62549e2000 rw-p 00066000 fd:01 142269                     /usr/lib64/libssl.so.1.0.1e
7f62549e2000-7f62549fb000 r-xp 00000000 fd:01 135049                     /usr/lib64/libsasl2.so.2.0.23
7f62549fb000-7f6254bfa000 ---p 00019000 fd:01 135049                     /usr/lib64/libsasl2.so.2.0.23
7f6254bfa000-7f6254bfb000 r--p 00018000 fd:01 135049                     /usr/lib64/libsasl2.so.2.0.23
7f6254bfb000-7f6254bfc000 rw-p 00019000 fd:01 135049                     /usr/lib64/libsasl2.so.2.0.23
7f6254bfc000-7f6254bfe000 r-xp 00000000 fd:01 276475                     /usr/java/jdk1.8.0_121/jre/lib/amd64/libjsig.so
7f6254bfe000-7f6254dfd000 ---p 00002000 fd:01 276475                     /usr/java/jdk1.8.0_121/jre/lib/amd64/libjsig.so
7f6254dfd000-7f6254dfe000 rw-p 00001000 fd:01 276475                     /usr/java/jdk1.8.0_121/jre/lib/amd64/libjsig.so
7f6254dfe000-7f6254dff000 rw-p 00000000 00:00 0 
7f6254dff000-7f6254e1f000 r-xp 00000000 fd:01 133789                     /lib64/ld-2.12.so
7f6254e1f000-7f6254e21000 r--s 0000d000 fd:06 939680444                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/commons-logging-1.1.1.jar
7f6254e21000-7f6254e22000 r--s 00005000 fd:06 939679746                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/commons-daemon-1.0.13.jar
7f6254e22000-7f6254e24000 r--s 00006000 fd:06 939680241                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/commons-compiler-2.7.6.jar
7f6254e24000-7f6254e26000 r--s 00009000 fd:06 939679980                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/commons-cli-1.2.jar
7f6254e26000-7f6254e2a000 r--s 0002f000 fd:06 939680580                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/commons-beanutils-core-1.8.0.jar
7f6254e2a000-7f6254e2f000 r--s 00034000 fd:06 939679971                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/commons-beanutils-1.8.3.jar
7f6254e2f000-7f6254e39000 r--s 00060000 fd:06 939679841                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/calcite-linq4j-1.0.0-incubating.jar
7f6254e39000-7f6254e6e000 r--s 002f0000 fd:06 939679795                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/calcite-core-1.0.0-incubating.jar
7f6254e6e000-7f6254e74000 r--s 0002f000 fd:06 939680218                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/calcite-avatica-1.0.0-incubating.jar
7f6254e74000-7f6254e76000 r--s 0001b000 fd:06 939680353                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/bonecp-0.7.1.RELEASE.jar
7f6254e76000-7f6254e78000 r--s 00014000 fd:06 939680027                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/azure-data-lake-store-sdk-2.2.3.jar
7f6254e78000-7f6254e81000 r--s 0006c000 fd:06 939680726                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/avro-1.7.6-cdh5.14.0.jar
7f6254e81000-7f6254e82000 r--s 00004000 fd:06 939680556                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/audience-annotations-0.4.0.jar
7f6254e82000-7f6254e84000 r--s 00003000 fd:06 939680201                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/async-1.3.1.jar
7f6254e84000-7f6254e86000 r--s 00004000 fd:06 939680418                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/asm-tree-3.1.jar
7f6254e86000-7f6254e87000 r--s 00007000 fd:06 939680351                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/asm-commons-3.1.jar
7f6254e87000-7f6254e88000 r--s 0000a000 fd:06 939679832                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/asm-3.1.jar
7f6254e88000-7f6254e8a000 r--s 00012000 fd:06 939680365                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/api-util-1.0.0-M20.jar
7f6254e8a000-7f6254e8c000 r--s 00003000 fd:06 939680366                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/api-asn1-api-1.0.0-M20.jar
7f6254e8c000-7f6254e9e000 r--s 00097000 fd:06 939680136                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/apacheds-kerberos-codec-2.0.0-M15.jar
7f6254e9e000-7f6254ea1000 r--s 00017000 fd:06 939680288                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/apacheds-jdbm1-2.0.0-M2.jar
7f6254ea1000-7f6254ea2000 r--s 0000a000 fd:06 939679855                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/apacheds-i18n-2.0.0-M15.jar
7f6254ea2000-7f6254eaa000 r--s 00066000 fd:06 939679955                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/apache-log4j-extras-1.2.17.jar
7f6254eaa000-7f6254eac000 r--s 00000000 fd:06 939680514                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/aopalliance-1.0.jar
7f6254eac000-7f6254eaf000 r--s 00025000 fd:06 939680059                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/antlr-runtime-3.3.jar
7f6254eaf000-7f6254eb4000 r--s 00068000 fd:06 939680074                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/antlr-2.7.7.jar
7f6254eb4000-7f6254eb5000 r--s 00004000 fd:06 939677753                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/ant-launcher-1.9.1.jar
7f6254eb5000-7f6254eba000 r--s 00032000 fd:06 939680014                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/ant-contrib-1.0b3.jar
7f6254eba000-7f6254ed8000 r--s 001ca000 fd:06 939677754                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/ant-1.9.1.jar
7f6254ed8000-7f6254ee3000 r--s 000a4000 fd:06 939680159                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/ant-1.5.jar
7f6254ee3000-7f6254ee5000 r--s 0000e000 fd:06 939680755                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/activation-1.1.jar
7f6254ee5000-7f6254eec000 r--s 0004e000 fd:06 939680038                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/libthrift-0.9.0.jar
7f6254eec000-7f6254ef2000 r--s 0007e000 fd:01 149166                     /usr/share/cmf/common_jars/postgresql-9.0-801.jdbc4.jar
7f6254ef2000-7f6254efb000 r--s 000e2000 fd:01 159276                     /usr/share/java/mysql-connector-java-5.1.34.jar
7f6254efb000-7f6254efc000 ---p 00000000 00:00 0 
7f6254efc000-7f6254ffc000 rw-p 00000000 00:00 0 
7f6254ffc000-7f6255004000 rw-s 00000000 fd:03 131079                     /tmp/hsperfdata_impala/62476
7f6255004000-7f6255012000 rw-p 00000000 00:00 0 
7f6255012000-7f6255016000 r--s 00036000 fd:06 939677752                  /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/jars/ST4-4.0.4.jar
7f6255016000-7f6255017000 rw-p 00000000 00:00 0 
7f6255017000-7f6255018000 r--p 00000000 00:00 0 
7f6255018000-7f625501e000 rw-p 00000000 00:00 0 
7f625501e000-7f625501f000 r--p 0001f000 fd:01 133789                     /lib64/ld-2.12.so
7f625501f000-7f6255020000 rw-p 00020000 fd:01 133789                     /lib64/ld-2.12.so
7f6255020000-7f6255021000 rw-p 00000000 00:00 0 
7fff086be000-7fff086c1000 ---p 00000000 00:00 0 
7fff086c2000-7fff087be000 rw-p 00000000 00:00 0                          [stack]
7fff087cf000-7fff087d0000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

VM Arguments:
jvm_args: -Djava.library.path=/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/bin/../lib/impala/lib 
java_command: <unknown>
java_class_path (initial): /usr/share/java/mysql-connector-java.jar:/usr/share/cmf/lib/postgresql-9.0-801.jdbc4.jar:/usr/share/java/oracle-connector-java.jar:/var/lib/impala/*.jar:/usr/share/java/mysql-connector-java.jar:/var/run/cloudera-scm-agent/process/6111-impala-IMPALAD/impala-conf:/var/run/cloudera-scm-agent/process/6111-impala-IMPALAD/hadoop-conf:/var/run/cloudera-scm-agent/process/6111-impala-IMPALAD/hive-conf:/var/run/cloudera-scm-agent/process/6111-impala-IMPALAD/hbase-conf:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/libthrift-0.9.0.jar::/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/ST4-4.0.4.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/activation-1.1.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/ant-1.5.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/ant-1.9.1.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/ant-contrib-1.0b3.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/ant-launcher-1.9.1.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/antlr-2.7.7.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/antlr-runtime-3.3.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/aopalliance-1.0.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/apache-log4j-extras-1.2.17.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/apacheds-i18n-2.0.0-M15.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/apacheds-jdbm1-2.0.0-M2.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/api-asn1-api-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/api-util-1.0.0-M20.jar:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib/asm-3.1.jar:/opt/cloudera/pa
Launcher Type: generic

Environment Variables:
JAVA_HOME=/usr/java/jdk1.8.0_121
JAVA_TOOL_OPTIONS=
PATH=/sbin:/usr/sbin:/bin:/usr/bin
LD_LIBRARY_PATH=/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/lib:/opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/impala/sbin-retail:/usr/java/jdk1.8.0_121/jre/lib/amd64:/usr/java/jdk1.8.0_121/jre/lib/amd64/server:/usr/java/jdk1.8.0_121/jre/lib/amd64/server:
SHELL=/bin/bash

Signal Handlers:
SIGSEGV: [libjvm.so+0xac78c0], sa_mask[0]=11111111111111111111111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xac78c0], sa_mask[0]=11111111111111111111111111111110, sa_flags=SA_ONSTACK|SA_SIGINFO
SIGFPE: [impalad+0x1cd2160], sa_mask[0]=00011111001000000000000000000000, sa_flags=SA_ONSTACK|SA_SIGINFO
SIGPIPE: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGXFSZ: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGILL: [impalad+0x1cd2160], sa_mask[0]=00011111001000000000000000000000, sa_flags=SA_ONSTACK|SA_SIGINFO
SIGUSR1: [impalad+0x8c5fb0], sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGUSR2: [libjvm.so+0x922300], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0x923700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: [libjvm.so+0x923700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGTERM: [libjvm.so+0x923700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0x923700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO


---------------  S Y S T E M  ---------------

OS:CentOS release 6.8 (Final)

uname:Linux 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Jan 11 20:56:24 UTC 2017 x86_64
libc:glibc 2.12 NPTL 2.12 
rlimit: STACK 10240k, CORE 0k, NPROC 65536, NOFILE 32768, AS infinity
load average:1.97 2.30 2.73

/proc/meminfo:
MemTotal:       198174948 kB
MemFree:         9252876 kB
Buffers:         5654672 kB
Cached:         123370380 kB
SwapCached:        26224 kB
Active:         60623864 kB
Inactive:       116502324 kB
Active(anon):   42772008 kB
Inactive(anon):  5403356 kB
Active(file):   17851856 kB
Inactive(file): 111098968 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:      33554428 kB
SwapFree:       33311736 kB
Dirty:              2640 kB
Writeback:             0 kB
AnonPages:      48078448 kB
Mapped:            87452 kB
Shmem:             73816 kB
Slab:           10272028 kB
SReclaimable:    9756152 kB
SUnreclaim:       515876 kB
KernelStack:       54416 kB
PageTables:       137012 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    132641900 kB
Committed_AS:   81881340 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      706048 kB
VmallocChunk:   34259060232 kB
HardwareCorrupted:     0 kB
AnonHugePages:      8192 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        7852 kB
DirectMap2M:     2971648 kB
DirectMap1G:    198180864 kB


CPU:total 32 (8 cores per cpu, 2 threads per core) family 6 model 62 stepping 4, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, aes, clmul, erms, ht, tsc, tscinvbit

/proc/cpuinfo:
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 0
cpu cores	: 8
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 1
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 1
cpu cores	: 8
apicid		: 2
initial apicid	: 2
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 2
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 2
cpu cores	: 8
apicid		: 4
initial apicid	: 4
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 3
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 3
cpu cores	: 8
apicid		: 6
initial apicid	: 6
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 4
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 4
cpu cores	: 8
apicid		: 8
initial apicid	: 8
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 5
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 5
cpu cores	: 8
apicid		: 10
initial apicid	: 10
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 6
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 6
cpu cores	: 8
apicid		: 12
initial apicid	: 12
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 7
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 7
cpu cores	: 8
apicid		: 14
initial apicid	: 14
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 8
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 0
cpu cores	: 8
apicid		: 32
initial apicid	: 32
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 9
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 1
cpu cores	: 8
apicid		: 34
initial apicid	: 34
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 10
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 2
cpu cores	: 8
apicid		: 36
initial apicid	: 36
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 11
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 3
cpu cores	: 8
apicid		: 38
initial apicid	: 38
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 12
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 4
cpu cores	: 8
apicid		: 40
initial apicid	: 40
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 13
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 5
cpu cores	: 8
apicid		: 42
initial apicid	: 42
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 14
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 6
cpu cores	: 8
apicid		: 44
initial apicid	: 44
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 15
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 7
cpu cores	: 8
apicid		: 46
initial apicid	: 46
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 16
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 0
cpu cores	: 8
apicid		: 1
initial apicid	: 1
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 17
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 1
cpu cores	: 8
apicid		: 3
initial apicid	: 3
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 18
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 2
cpu cores	: 8
apicid		: 5
initial apicid	: 5
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 19
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 3
cpu cores	: 8
apicid		: 7
initial apicid	: 7
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 20
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 4
cpu cores	: 8
apicid		: 9
initial apicid	: 9
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 21
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 5
cpu cores	: 8
apicid		: 11
initial apicid	: 11
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 22
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 6
cpu cores	: 8
apicid		: 13
initial apicid	: 13
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 23
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 0
siblings	: 16
core id		: 7
cpu cores	: 8
apicid		: 15
initial apicid	: 15
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.67
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 24
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 0
cpu cores	: 8
apicid		: 33
initial apicid	: 33
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 25
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 1
cpu cores	: 8
apicid		: 35
initial apicid	: 35
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 26
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 2
cpu cores	: 8
apicid		: 37
initial apicid	: 37
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 27
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 3
cpu cores	: 8
apicid		: 39
initial apicid	: 39
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 28
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 4
cpu cores	: 8
apicid		: 41
initial apicid	: 41
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 29
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 5
cpu cores	: 8
apicid		: 43
initial apicid	: 43
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 30
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 6
cpu cores	: 8
apicid		: 45
initial apicid	: 45
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:

processor	: 31
vendor_id	: GenuineIntel
cpu family	: 6
model		: 62
model name	: Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping	: 4
microcode	: 1047
cpu MHz		: 2493.839
cache size	: 20480 KB
physical id	: 1
siblings	: 16
core id		: 7
cpu cores	: 8
apicid		: 47
initial apicid	: 47
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips	: 4987.27
clflush size	: 64
cache_alignment	: 64
address sizes	: 46 bits physical, 48 bits virtual
power management:



Memory: 4k page, physical 198174948k(9252876k free), swap 33554428k(33311736k free)

vm_info: Java HotSpot(TM) 64-Bit Server VM (25.121-b13) for linux-amd64 JRE (1.8.0_121-b13), built on Dec 12 2016 16:36:53 by "java_re" with gcc 4.3.0 20080428 (Red Hat 4.3.0-8)

time: Fri Apr 27 00:09:11 2018
elapsed time: 203 seconds (0d 0h 3m 23s)

[#3]

https://issues.apache.org/jira/browse/IMPALA-6882 is easy to rule out since it only occurs on > 5 year old processors.

[#4]

Thank you Chris for providing more information. It looks like it crashed in the code that writes Parquet files (HdfsParquetTableWriter::ColumnWriter<impala::StringValue>::ProcessValue). However, your query should not write any data: "SELECT a.topLevelField, b.priceFromNestedField FROM db.table a LEFT JOIN a.nestedField b"

I also noticed that the stack looks like it has been overwritten by something.

I don't recall any recent issues in that method and will have a look at the code to see if I can spot anything obvious.

In the meantime, can you double check that this query caused the crash and no other query was running?

Thanks, Lars

[#5]

Hi Tim,

I checked cpuinfo and I don't see a popcnt:

vendor_id : GenuineIntel
cpu family : 6
model : 62
model name : Intel(R) Xeon(R) CPU E5-2450 v2 @ 2.50GHz
stepping : 4
microcode : 1047
cpu MHz : 2493.839
cache size : 20480 KB
physical id : 1
siblings : 16
core id : 7
cpu cores : 8
apicid : 47
initial apicid : 47
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good xtopology nonstop_tsc aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm ida arat epb xsaveopt pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase smep erms
bogomips : 4987.27
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:

[#6]

Hi Lars,

Yes it was this query as if it was run many time it would always cause the crash.

I'm upgrading the cluster to 5.14.2 this afternoon to see if that resolves the issue. I'll update you later today (uk time) to let you know how it goes.

[#7]

Lars - Tim,

This issue has been resolved by upgrading to version 5.14.2.  Thanks for all your assistance.

Cheers,
Chris

[#8 · Accepted solution]

The reason you were seeing HdfsParquetTableWriter::ColumnWriter is that I was testing the bug using the syntax:

CREATE TABLE db.newTable STORED AS PARQUET AS

SELECT a.topLevelField, b.priceFromNestedField FROM db.table a LEFT JOIN a.nestedField 

This was purely to force the bug to occur - if you just did the SELECT in Hue it would often succeed because it only brings back the first 100 rows - to consistently trigger the crash I had to make Impala read from both Parquet files. No other query was running at the time.

Anyway, as Chris says, the bug appears to be fixed in 5.14.2. The job which originally consistently triggered the crash has now been running unchanged over the same source data for 20 hours without hitch.

Thanks for your help

Matt