JDBC stopped working after upgrading to CDH 5.12

[#1]

We're upgrading our cluster from CDH 5.7 to 5.12. We've sentry (service ) authentication enabled and roles created for different users.

#1.

            connectionUrl = "jdbc:impala://cluster.com:21050;AuthMech=2;UID=sparmar";
            jdbcDriverName = "com.cloudera.impala.jdbc41.Driver";

Result :  hangs on this line and times out. No log on server side of an attempt.

           con = DriverManager.getConnection(connectionUrl);

#2.

           connectionUrl = "jdbc:impala://cluster.com:21050;";

Result: Exception;

java.sql.SQLException: [Simba][ImpalaJDBCDriver](500051) ERROR processing query/statement. Error Code: 0, SQL state: TStatus(statusCode:ERROR_STATUS, sqlState:HY000, errorMessage:AuthorizationException: User '' does not have privileges to access: default.*
), Query: show tables;.

I have made sure my user ( group) ( who is running java process remotely) also exists on the daemon and is authorized to access tables.

I see this on server/daemon logs

3:56:28.968 PM 	WARN 	java:62 	

Unable to obtain groups for 
Java exception follows:
java.io.IOException: No groups found for user 
	at org.apache.hadoop.security.Groups.noGroupsForUser(Groups.java:199)
	at org.apache.hadoop.security.Groups.getGroups(Groups.java:222)
	at org.apache.sentry.provider.common.HadoopGroupMappingService.getGroups(HadoopGroupMappingService.java:60)
	at org.apache.sentry.provider.common.ResourceAuthorizationProvider.getGroups(ResourceAuthorizationProvider.java:167)
	at org.apache.sentry.provider.common.ResourceAuthorizationProvider.doHasAccess(ResourceAuthorizationProvider.java:97)
	at org.apache.sentry.provider.common.ResourceAuthorizationProvider.hasAccess(ResourceAuthorizationProvider.java:91)
	at org.apache.impala.authorization.AuthorizationChecker.hasAccess(AuthorizationChecker.java:176)
	at org.apache.impala.authorization.AuthorizationChecker.checkAccess(AuthorizationChecker.java:127)
	at org.apache.impala.analysis.AnalysisContext.authorizePrivilegeRequest(AnalysisContext.java:530)
	at org.apache.impala.analysis.AnalysisContext.authorize(AnalysisContext.java:493)
	at org.apache.impala.service.Frontend.analyzeStmt(Frontend.java:930)
	at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1052)
	at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)

3:56:28.970 PM 	INFO 	cc:176 	

org.apache.impala.catalog.AuthorizationException: User '' does not have privileges to access: default.*
	at org.apache.impala.authorization.AuthorizationChecker.checkAccess(AuthorizationChecker.java:137)
	at org.apache.impala.analysis.AnalysisContext.authorizePrivilegeRequest(AnalysisContext.java:530)
	at org.apache.impala.analysis.AnalysisContext.authorize(AnalysisContext.java:493)
	at org.apache.impala.service.Frontend.analyzeStmt(Frontend.java:930)
	at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:1052)
	at org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:156)

3:56:28.975 PM 	INFO 	cc:122 	

AuthorizationException: User '' does not have privileges to access: default.*
    @           0x83d9f9  impala::Status::Status()
    @           0xb99e10  impala::JniUtil::GetJniExceptionMsg()
    @           0xaa7feb  impala::Frontend::GetExecRequest()
    @           0xac2d22  impala::ImpalaServer::ExecuteInternal()
    @           0xac8e08  impala::ImpalaServer::Execute()
    @           0xaf8f5d  impala::ImpalaServer::ExecuteStatement()
    @           0xd972e6  apache::hive::service::cli::thrift::TCLIServiceProcessor::process_ExecuteStatement()
    @           0xd90f04  apache::hive::service::cli::thrift::TCLIServiceProcessor::dispatchCall()
    @           0x80dc4c  apache::thrift::TDispatchProcessor::process()
    @          0x1b4e57b  apache::thrift::server::TThreadPoolServer::Task::run()
    @          0x1b36139  apache::thrift::concurrency::ThreadManager::Worker::run()
    @           0x9fd219  impala::ThriftThread::RunRunnable()
    @           0x9fdc72  boost::detail::function::void_function_obj_invoker0<>::invoke()
    @           0xbf0ac9  impala::Thread::SuperviseThread()
    @           0xbf1484  boost::detail::thread_data<>::run()
    @           0xe592ea  (unknown)
    @       0x39a00079d1  (unknown)
    @       0x399fce88fd  (unknown)

Our JDBC clients were using connector/driver version 2.5.30 before upgrade and they also tried upgrading to latest 2.5.38 but both had similar failures. I created a sample JAVA project by manually adding all the driver jars in lib and tried runnning it but have similar results.

Is jdbc client really sendint empty user to server / server is not able to determine user ?

Any pointers ?

How can I debug / find out where it is getting stuck ?

Thanks,

Sunil

[#2]

Anyone ?
In a nutshell.
CDH 5.12 Impala with sentry (Service ) enabled.
Impala JDBC Driver 3.5.38 ( latest )
AuthMech=2 hangs the getConnection on client.
Without authmech , server logs complains about empty user.

[#3]

Hi Sunil,

Was the same connection string in #1 also used prior to upgrading CDH?

Let's enable logging level 5 for more detailed information. See page 22 of this doc. 

[#4]

Actually I tried #1 with CDH 5.7 and it didn't work as well but #2 worked with 5.7 and stopped working on 5.12

[#5]

I see. Have you already tried #1 without specifying the UID field?

[#6]

Yeah tried this URL on both versions; it fails with exception.

                           "jdbc:impala:/cluster.threatmetrix.com:21050/dwh;AuthMech=2";

java.sql.SQLNonTransientConnectionException: [Simba][JDBC](10100) Connection Refused: [Simba][JDBC](11640) Required Connection Key(s): UID; [Simba][JDBC](11480) Optional Connection Key(s): AsyncExecPollInterval, CatalogSchemaSwitch, DefaultStringColumnLength, DelegationUID, LowerCaseResultSetColumnName, OptimizedInsert, PreparedMetaLimitZero, RowsFetchedPerBlock, SocketTimeOut, ssl, StripCatalogName, SupportTimeOnlyTimestamp, UseCustomTypeCoercionMap, UseNativeQuery, UseSasl
    at com.cloudera.exceptions.ExceptionConverter.toSQLException(Unknown Source)
    at com.cloudera.jdbc.common.BaseConnectionFactory.checkResponseMap(Unknown Source)
    at com.cloudera.jdbc.common.BaseConnectionFactory.doConnect(Unknown Source)
    at com.cloudera.jdbc.common.AbstractDriver.connect(Unknown Source)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:270)
    at impala_jdbc_test.ImpalaJDBCTestBud.main(ImpalaJDBCTestBud.java:59)
Exception in thread "main" java.lang.NullPointerException

[#7]

Interesting - what information do we receive when logging is enabled?

What is set in HDFS > Hadoop User Group Mapping?