Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Hive shell crashes after running queries

Hive shell crashes after running queries

New Contributor

I'm putting this out there mostly to get it indexed, I don't currently have a problem to fix.

Setting up a new HDP cluster, I ran into a problem where hive shell crashes after running a query.

hive> select * from my_table limit 2;
[2 rows of valid output]
hive> Exception in thread "main" java.io.IOException: Bad file descriptor
[Java stack trace]
[me@n01 import]$ _

Looking in /tmp/me/hive.log, I found this message among the wreckage:

2017-11-16 17:18:40,278 WARN  [main]: hdfs.BlockReaderFactory (BlockReaderFactory.java:createShortCircuitReplicaInfo(506)) - BlockReaderFactory(fileName=/apps/hive/warehouse/some.db/my_table/000000_0, block=BP-1039108363-10.96.99.32-1510818878096:blk_1073741949_1125): I/O error requesting file descriptors.  Disabling domain socket DomainSocket(fd=512,path=/var/lib/hadoop-hdfs/dn_socket)
java.io.IOException: size of shared memory segment was 0, but that is not enough to hold even one slot.

This led me to dig around and find a possibly relevant Jira about HDFS short-circuit reads.

Relevant or not, it did give me the idea to hunt down and disable the option HDFS Short-circuit read in Ambari's HDFS config. Hive shell no longer crashes after this change.

I have no idea if this is a 'correct' fix or not, but I thought I'd put it out there for the next person who has similar problems. If someone has a better idea of what could be wrong or what should be fixed, I'm all ears.

HDP-2.6.3.0 on CentOS 7, using MySQL 5.7 for any task requiring a relational store.