Re: Pig on Spark (How to use spark as an execution...

thejas · ‎07-14-2017

I believe newer version of the jdbc driver for mariadb with HiveServer2/Metastore should solve the problem for it as well. Have you tried that ?

thejas · ‎06-02-2017

As pig-on-spark is not currently supported (its not in immediate plans either) in HDP, you might want to raise this question in the apache pig mailing list itself, where some developers who are working on it might be able to respond. As mentioned previously, pig-on-tez is lot more mature as it has been in production use for few years now.

thejas · ‎06-01-2017

Pig on spark is a very new feature, and it still not part of an official apache release. It is likely to be take more time before it would be widely used in production and reccomended for production use. Pig on tez has been around for sometime and has been used in production many large installations. I would reccomend using that over pig on spark.

thejas · ‎05-09-2017

You can also use following command to retrieve the password - sudo java -cp "/usr/hdp/current/ranger-tagsync/lib/*" org.apache.ranger.credentialapi.buildks get javax.jdo.option.connectionpassword -provider jceks://file/etc/hive/conf/conf.server/hive-site.jceks

thejas · ‎05-05-2017

"localhost" doesn't work with kerberos authentication, you need the actual hostname in URL. @Jonas Straub can you please edit ?

thejas · ‎04-20-2017

What is the mysql jdbc driver version in use here ? Have you tried using mysql-connector-java-5.1.35.zip or later JDBC driver as reccomended ? http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_command-line-installation/content/meet-min-system-requirements.html

thejas · ‎03-23-2017

Look for "Hortonworks Sandbox Archive" under https://hortonworks.com/downloads/#sandbox and click on "Expand" to find older versions of sandbox.

thejas · ‎02-25-2017

User.name would be ignored during kerberos mode. The credentials via kinit is what gets used.

thejas · ‎02-16-2017

Running the command with "-verbose" will give more information. Looks like the error is "Specified key was too long; max key length is 1000 bytes" . It seems like you might be using myisam instead of innodb as the engine for some table. You might want to see if suggestions to change type to innodb as mentioned here works for you - http://stackoverflow.com/questions/3489041/mysqlerror-specified-key-was-too-long-max-key-length-is-1000-bytes The tables do explicitly specify innodb engine type, so not sure why you are seeing this issue. Are you on a very old version of mysql ? See supported versions here - https://cwiki.apache.org/confluence/display/Hive/AdminManual+MetastoreAdmin#AdminManualMetastoreAdmin-SupportedBackendDatabasesforMetastore

thejas · ‎02-16-2017

I would not reccomend using derby for any production workload.

thejas · ‎02-08-2017

Can you please include the stack trace you see ?

thejas · ‎08-06-2016

Thanks for sharing. What query is this ? Is this performance of reads or writes ?

thejas · ‎05-05-2016

With HDP 2.3.4 and later releases, you don't need to specify any additional options as it includes the changes in HIVE-11581. Just the first URL is sufficient - jdbc:hive2://<zookeeper quorum>/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2. See the hiveserver2 client doc Also make sure you have the url within quotes if you are specifying it in commandline. ie, beeline -u 'hive2...'

thejas · ‎04-11-2016

Use hive.execution.engine=tez for better performance

thejas · ‎03-23-2016

Looks like you have authorization enabled , but have authorization.manager set to the default (legacy) authorizer. It also looks like you are using some variant of hive that deviates from apache hive. That above error message is not something that appears in apache hive (which Hive in HDP is closely based on). You would want to refer to the documents to setup authorization properly , and use either sql std auth or ranger. Sql std auth docs are here - https://cwiki.apache.org/confluence/display/Hive/SQL+Standard+Based+Hive+Authorization Also, as Geoffrey mentioned, grant/revoke based authorization are not supported/effective in HIVE CLI for sql std auth or ranger. You would want to use beeline with HiveServer2.

thejas · ‎03-14-2016

can you paste the full exception stack trace ?

thejas · ‎03-14-2016

I am wondering if it could be be cause of Hbase configuration not being available in the HS2 oozie action.

thejas · ‎03-08-2016

HTTP transport mode of HS2 + kerberos + Oozie Hive2 action is not supported in current releases of HDP. It will be fixed in next maintenance release of HDP 2.4 . This is the apache jira tracking it - https://issues.apache.org/jira/browse/HIVE-13169

thejas · ‎02-18-2016

Can you please elaborate on what you mean by distributed cache not working ? How are you using distributed cache in your query ?

Online	Offline
Last Visited	‎06-01-2021 01:03 PM

Date Registered	‎10-02-2015 07:01 PM
Last Visited	‎06-01-2021 01:03 PM
Total Messages Posted	19
Total Kudos Received	6

Re: Pig on Spark (How to use spark as an execution...

Re: Oozie Hive2 action in kerberized cluster using...

Re: java.sql.SQLException: Unknown system variable...

Re: Pig on Spark (How to use spark as an execution...

Re: Pig on Spark (How to use spark as an execution...

Re: From where does the Hive Metastore gets databa...

Re: curl with hive on secured cluster

Re: java.sql.SQLException: Unknown system variable...

Re: Download Link Sandbox HDP 2.3.2

Re: curl with hive on secured cluster

Re: Hive Metastore SchemaTool

Re: Hive Metastore SchemaTool

Re: java.lang.NoClassDefFoundError: org/jets3t/ser...

Re: Performance Comparison b/w ORC SNAPPY and ZLib...

Re: How to Enable Zookeeper Discovery for HiveServ...

Re: Unable to use hive.execution.engine=spark In H...

Re: Hive : User does not belong to hive error

Re: After Upgrading HDP 2.2 to 2.3.4, simple query...

Re: Oozie hive action failing with NullPointer exc...

Re: Oozie Hive2 action in kerberized cluster using...

Re: When using TEZ with Pig Latin (HDP Sandbox 2.3...