Map Reduce error for concurrent user from Java application to access Apache Hive

K_K — Mon, 10 Aug 2020 16:26:12 GMT

We have a user interface based on Java. This application connects Hive as data source. Please find the product version of Apache Hive below. Application is working fine when single user has login with the application to access data from Hive. It gives error while concurrent users login to same application and access the Hive data..

apache-hive-2.3.7

db-derby-10.14.2.0

hadoop-2.10.0

Further investigation –

The authentication mechanism of Hive is NOSASL. We did a small mutli threaded java application for debugging. Please find Java code and log for your analysis.

Thanks in advance for your help!!

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.sql.SQLException;

public class MapReduceThreadClass implements Runnable {
String name;
Thread t;
String query;
MapReduceThreadClass (String thread,String query){
name = thread;
this.query=query;
t = new Thread(this, name);
System.out.println("New thread: " + t);
t.start();
}

@Override
public void run() {
Connection conn = null;
try {
Class.forName("org.apache.hive.jdbc.HiveDriver");
String url = "jdbc:hive2://XX.XX.XX.XX:10000/test1;auth=noSasl";
conn = DriverManager.getConnection(url);

PreparedStatement ps=conn.prepareStatement(query);
System.out.println("thread "+Thread.currentThread().getName()+"before query execution");
ResultSet rs=ps.executeQuery();
if(rs.next()) {
System.out.println("thread "+Thread.currentThread().getName()+"::"+rs.getString(1));
System.out.println("thread "+Thread.currentThread().getName()+"::"+"Got it!");
}
} catch (Exception e) {
e.printStackTrace();
throw new Error("Problem", e);
} finally {
try {
if (conn != null) {
conn.close();
}
} catch (SQLException ex) {
System.out.println(ex.getMessage());
}
}
// TODO Auto-generated method stub

}

----

public class MapReduceMain {

public static void main(String[] args) {
//18
for(int i=0;i<Integer.parseInt(args[0]);i++)
{
new MapReduceThreadClass(i+1+"",args[1]);
}
// TODO Auto-generated method stub

}

Re: Map Reduce error for concurrent user from Java application to access Apache Hive

asish — Tue, 11 Aug 2020 17:11:18 GMT

Derby only allows single connection (process) to access the database at a give time, hence only one user can access the Hive.

Upgrade your hive metastore to either MySQL, PostgreSQL to support multiple concurrent connections to Hive.

Re: Map Reduce error for concurrent user from Java application to access Apache Hive

K_K — Thu, 20 Aug 2020 05:53:03 GMT

Hello Asish,

Thanks for your help.

It's working now with concurrent users.

Thanks,

question Re: Map Reduce error for concurrent user from Java application to access Apache Hive in Support Questions

Map Reduce error for concurrent user from Java application to access Apache Hive

Re: Map Reduce error for concurrent user from Java application to access Apache Hive

Re: Map Reduce error for concurrent user from Java application to access Apache Hive