I am getting java.lang.ClassNotFoundException:Class org.apache.hive.hcatalog.data.JsonSerDenot found, when using hive in pyspark program. I have tried using --jars (to define the path for hive-hcatalog-core.jar ) option during spark-submit, but its not working ... View more

Hi, How do we find why all services went down ?We have HDP2.6 all services went down, do we just restart the all services? .We are able to connect to Ambari. ... View more

We have HDP 2.6.4 Ambari URL is not loading sudo ambari-server status Using python /usr/bin/python Ambari-server status Ambari Server running Found Ambari Server PID: 14654 at: /var/ambari-server/ambari-server.pid 01 Aug 2018 19:08:53,866 ERROR [ambari-client-thread-1088676] MetricsRequestHelper:115 - Error getting timeline metrics : Connection refused (Connection refused) 01 Aug 2018 19:08:53,867 ERROR [ambari-client-thread-1088676] MetricsRequestHelper:122 - Cannot connect to collector: SocketTimeoutException for ctcl-hdpedge1.msoit.com 01 Aug 2018 19:08:46,391 ERROR [ambari-client-thread-1088525] MetricsRequestHelper:122 - Cannot connect to collector: SocketTimeoutException for ctcl-hdpedge1.msoit.com 01 Aug 2018 19:08:53,867 ERROR [ambari-client-thread-1088676] MetricsRequestHelper:122 - Cannot connect to collector: SocketTimeoutException for ctcl-hdpedge1.msoit.com ... View more

We have HDP 2.6.4, we have user janu, when she runs the first sql it works fine, 2 sql gives below error we have kerberos enabled and users were created using ldap 1)select * from db.tbl 2) select col1 from db.tbl group by col1 AM Container for appattempt_1523646395030_2519_000002 exited with exitCode: -1000 For more detailed output, check the application tracking page: http://ctcl-hdpmaster3.com:8088/cluster/app/application_1523646395030_2519 Then click on links to logs of each attempt. Diagnostics: Application application_1523646395030_2519 initialization failed (exitCode=255) with output: main : command provided 0 main : run as user is janu main : requested yarn user is janu User janu not found Failing this attempt ... View more

We are getting below error on randomly for few minutes and then goes away, its coming in PUThiveql 2018-05-29 01:01:07,279 INFO [Timer-Driven Process Thread-95] org.apache.hive.jdbc.HiveConnection Will try to open client transport with JDBC Uri: jdbc:hive2://hdpmaster2.msoit.com:10000/default;principal=hive/_HOST@IT.COM 2018-05-29 01:01:07,281 ERROR [Timer-Driven Process Thread-95] o.apache.thrift.transport.TSaslTransport SASL negotiation failure javax.security.sasl.SaslException: GSS initiate failed at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:204) at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:176) at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105) at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:38) at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:582) at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1148) at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106) at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044) at org.apache.nifi.dbcp.hive.HiveConnectionPool.lambda$getConnection$0(HiveConnectionPool.java:355) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656) at org.apache.nifi.dbcp.hive.HiveConnectionPool.getConnection(HiveConnectionPool.java:355) at sun.reflect.GeneratedMethodAccessor393.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:89) at com.sun.proxy.$Proxy97.getConnection(Unknown Source) at org.apache.nifi.processors.hive.PutHiveQL.lambda$new$1(PutHiveQL.java:191) at org.apache.nifi.processor.util.pattern.Put.onTrigger(Put.java:96) at org.apache.nifi.processors.hive.PutHiveQL.lambda$onTrigger$6(PutHiveQL.java:274) at org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger(PartialFunctions.java:114) at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger(RollbackOnFailure.java:184) at org.apache.nifi.processors.hive.PutHiveQL.onTrigger(PutHiveQL.java:274) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1147) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:175) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192 ... View more

spark-submit --master yarn --deploy-mode cluster sqlserver.py --jars sqljdbc42.jar I get Error : java.lang.ClassNotFoundException: com.microsoft.sqlserver.jdbc.SQLServerDriver Everything works when i use --deploy-mode client and copy the jar file to /usr/hdp/current/spark2-client/jars/sqljdbc42.jar Should i copy the sqljdbc42.jar to /usr/hdp/current/hadoop-yarn-client/lib/ on all data nodes? ... View more

I have created a HIVE UDF in scala, i am getting below error when creating function in hive using beeline jdbc:hive2:> list jars; +----------------------------------------------------------------------------+--+ | resource | +----------------------------------------------------------------------------+--+ | /tmp/a22a6181-545f-4826-8d79-eee2609b2cda_resources/cnt_business_days.jar | +----------------------------------------------------------------------------+--+ 1 row selected (0.015 seconds) 0: jdbc:hive2:> CREATE temporary FUNCTION trim AS 'com.hiveudf.ScalaHiveUdf' ; ERROR : FAILED: Class com.hiveudf.ScalaHiveUdf not found Error: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.FunctionTask (state=08S01,code=1) ... View more

I am trying to use python udf in hive, but i get the below error Error: Error while processing statement: FAILED: Hive Internal Error: org.apache.hadoop.hive.ql.security.authorization.plugin.HiveAccessControlException(Query with transform clause is disallowed in current configuration.) (state=08S01,code=12) ... View more

I had killed the few hive query running though Beeline , however the Tez UI shows the status as running? When will this status change ... View more

We have HDP 2.6, when i added table & column comments using alter table.These comments are not reflected in Atlas. I thought this should be automated ... View more

I am trying to create an external table hive , i get below error.I have defined the policy in ranger. Hive policy i have given all permissions to the user for the database(all tables & all columns). I am conecting to hive using beeline Error: Error while compiling statement: FAILED: HiveAccessControlException Permission denied: user [hdp_ingest] does not have [ALL] privilege on [hdfs://hdpmaster1.m.com:8020/datalake/first/landing/tbltest] (state=42000,code=40000) ... View more