Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Were we able to see data from beeline ? What am I doing incorrectly here ? or does the thrift server need additional configuration ?

Were we able to see data from beeline ? What am I doing incorrectly here ? or does the thrift server need additional configuration ?

New Contributor

pclhdp001 ~]$ beeline

WARNING: Use "yarn jar" to launch YARN applications.

Beeline version 1.2.1000.2.4.0.0-169 by Apache Hive

beeline> !connect jdbc:hive2://localhost:10015

Connecting to jdbc:hive2://localhost:10015

Enter username for jdbc:hive2://localhost:10015: kishore

Enter password for jdbc:hive2://localhost:10015: **********

Connected to: Spark SQL (version 1.6.0)

Driver: Hive JDBC (version 1.2.1000.2.4.0.0-169)

Transaction isolation: TRANSACTION_REPEATABLE_READ

0: jdbc:hive2://localhost:10015> show tables;

+--------------------------------------------+--------------+--+

| tableName | isTemporary |

+--------------------------------------------+--------------+--+

| hold_pos | false |

|hold_pos1 | false |

|hold_pos_new | false |

|

|………………………Clipped………………………………………

|

|tablenew2 | false |

| tablenew_1 | false |

| trial2021916_2| false |

+--------------------------------------------+--------------+--+

64 rows selected (0.173 seconds)

0: jdbc:hive2://localhost:10015> select * from orders limit 10;

Error: java.io.IOException: Failed to create local dir in /tmp/blockmgr-a789ddf4-6b0e-496f-8f29-2b470e2af926/05. (state=,code=0)

0: jdbc:hive2://localhost:10015> select * from stores limit 10;

Error: java.io.IOException: Failed to create local dir in /tmp/blockmgr-a789ddf4-6b0e-496f-8f29-2b470e2af926/1e. (state=,code=0)

0: jdbc:hive2://localhost:10015>

I have a case open with HortonWorks concerning this requesting info on how to connect to SparkSQL through JDBC on a Windows machine. I’m getting the above error as when selecting the data, and researching it now.

please explain how to resolve this issue ..

5 REPLIES 5

Re: Were we able to see data from beeline ? What am I doing incorrectly here ? or does the thrift server need additional configuration ?

Expert Contributor

It looks like the user under which you started the thrift server does not have permission to write in /tmp (local dir). You can set the spark.local.dir property (see Spark docs) to some other directory. Or check the permission for /tmp directory (local, not HDFS) on the nodes.

If that does not help, here are some questions that might help get to the crux of the problem:

  • How are you starting the Spark Thrift Server? (i.e. what command under what user)
  • With the user that is running Spark Thrift Server, can it write to the /tmp dir on the cluster nodes?
  • Is this a kerberized cluster?

I hope this helps!

Re: Were we able to see data from beeline ? What am I doing incorrectly here ? or does the thrift server need additional configuration ?

New Contributor

@clukasik Beeline connectivity to HiveServer2 is possible through localhost but not through IP available for external network i.e. 10.54.251.11 . Can you please check and help in resolution ?

Re: Were we able to see data from beeline ? What am I doing incorrectly here ? or does the thrift server need additional configuration ?

New Contributor

I added a config property called spark.local.dir, and this seemed to resolve this issue below, and I can select from tables when connecting through port 10015 in beeline.

I set it to /tmp, as it needs to be writeable by the spark process. I tried making a sub-directory called /tmp/spark-tmp, and change ownership to spark:hadoop but it didn’t like it for some reason. Maybe because it wasn’t executable.

From /var/log/spark/spark-hive-org.apache.spark.sql.hive.thriftserver.HiveThriftServer2-1-mdzusvpclhdp001.mdz.local.out:

16/08/03 14:58:14 ERROR DiskBlockManager: Failed to create local dir in /tmp/spark-tmp. Ignoring this directory.

java.io.IOException: Failed to create a temp directory (under /tmp/spark-tmp) after 10 attempts!

Re: Were we able to see data from beeline ? What am I doing incorrectly here ? or does the thrift server need additional configuration ?

New Contributor
,

If "show databases" command is working and if you get this error "Failed to create local dir in /tmp/blockmgr-a789ddf4-6b0e-496f-8f29-2b470e2af926/05. (state=,code=0)" only for select sqls, then it's due to stale spark thrift server. Some spark/hive configuration might changed after your spark thrift server started.

Fix: Just restart your spark thrift server, and try select sql again, it will work.

Re: Were we able to see data from beeline ? What am I doing incorrectly here ? or does the thrift server need additional configuration ?

New Contributor

Setting up 0000 umask on /tmp dir helped to solve the issue.

Don't have an account?
Coming from Hortonworks? Activate your account here