About deepesh1

deepesh1 · ‎10-13-2015

http://hortonworks.com/products/releases/hdp-2-2/#add_ons

deepesh1 · ‎10-13-2015

Can you try the following connection url (observe the / after the <ZOOKEEPER QUORUM>)? jdbc:hive2://<ZOOKEEPER QUORUM>/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver Above is for binary mode, for http mode jdbc:hive2://<ZOOKEEPER QUORUM>/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver;transportMode=http;httpPath=cliservice For secure environments you will additionally have to add the hive principal, eg. jdbc:hive2://<ZOOKEEPER QUORUM>/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver;principal=hive/_HOST@EXAMPLE.COM;transportMode=http;httpPath=cliservice

deepesh1 · ‎10-13-2015

These permissions should be enough for the oozie user on Oracle: GRANT SELECT_CATALOG_ROLE TO <OOZIE USER>; GRANT CONNECT, RESOURCE TO <OOZIE USER>; For the other databases MySQL and PostgresQL I agree with @David Streever that the permission needs to be set to just Oozie database. Additionally we want to limit the access from just the oozie host. On MySQL this can be: GRANT ALL PRIVILEGES ON <OOZIE DB>.* TO '<OOZIE USER>'@'<OOZIE HOST>'

deepesh1 · ‎10-12-2015

Look at the Oozie coordinator documentation, the timezone value will be "America/New_York" and start would be something like "2015-10-12T08:00Z". Alternatively you can do a cron like thing, see the example in the following blog. Based on that it can be something like: <coordinator-app name="weekdays-at-eight-pm" frequency="0 20 * * 2-6" start="${start}" end="${end}" timezone="America/New_York" xmlns="uri:oozie:coordinator:0.2"> ... </coordinator-app>

deepesh1 · ‎10-12-2015

There is an open intermittent issue where Hive View fails handling submission of multiple statements in a single query/submit. Is that the case here?

deepesh1 · ‎10-09-2015

Can you shed more details on what type of configuration change was applied using Ambari that triggered the hive cli hang?

deepesh1 · ‎10-09-2015

Check out http://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_large_objects

deepesh1 · ‎10-09-2015

We support both Oracle 11g r2 and Oracle 12c.

deepesh1 · ‎10-09-2015

Its not that MR gets resources, Tez just pre-allocates resources, it starts a YARN application as soon as you invoke the Hive CLI. In case of MR a YARN application is launched only when its required by the executing query. This is a performance optimization in Hive on Tez saving the user the overhead of launching a AM everytime a new query is run in the same Hive CLI session.

deepesh1 · ‎10-09-2015

One thing to check is if they are using tez. If they are, it could be possible that they do not have any capacity on their YARN cluster to launch AM. So the Hive CLI waits indefinitely to launch an AM. One thing they can do to alleviate this is to launch hive cli as "hive -hiveconf hive.execution.engine=mr".

deepesh1 · ‎10-09-2015

Yes, that is not documented as its not a common configuration/practice.

deepesh1 · ‎10-08-2015

Row based security can be achieved through SQL Standard Based Hive Authorization.You can create a view with the filter from the original table and then GRANT permissions to role or individual user.

deepesh1 · ‎10-08-2015

Yes, check out http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_yarn_resource_mgt/content/configuring_node_labels.html

deepesh1 · ‎10-08-2015

Yes, those are the right ones, we are in the process of pushing to the web, should be available in a day or two.

deepesh1 · ‎10-07-2015

From the code it looks like ColumnPruner is always applied by optimizer and there is no way to exclude it after HIVE-4113. It would be good to get more details on the scenario where you ran into this. Have you tried this query with Hive v0.14 or later?

deepesh1 · ‎10-07-2015

Coming very soon.

deepesh1 · ‎10-07-2015

For that you will need to get access to javax.security.auth.kerberos.KerberosTicket, the class in Hadoop UesrGroupInformation provides API to manage that, however the closest method to deal with expirations is checkTGTAndReloginFromKeytab which is likely not what you want.

deepesh1 · ‎10-07-2015

The trick will be to manage separate hive configs for each HiveServer2 instance. You will need to use different values for authentication properties and the port on which to start.

deepesh1 · ‎10-07-2015

One way you can achieve the transformation of your CSV data to ORC would be to do the following: 1. Register your CSV GZ data as a text table, something like: create table <tablename>_txt (...) location '...'; 2. Create equivalent ORC table create table <tablename>_orc (...) stored as orc; 3. Populate the data into equivalent ORC table insert overwrite table <tablename>_orc select * from <tablename>_txt; I have used this in the past and worked for me.

deepesh1 · ‎10-07-2015

Could be a real bug. What is HDP/hive version you are using?

deepesh1 · ‎10-07-2015

You can delete Hive tables by calling "drop table <tablename> purge;", this will skip the trash. If this is for testing purposes you can temporarily set fs.trash.interval to 0 and restart namenode. This will globally disable trash collection on HDFS so should only be employed during testing. On your last question about the support of TDE feature, it was available starting HDP 2.3.

deepesh1 · ‎10-06-2015

Note that when running more that one Hiveserver2 instances registered on zookeeper you also get load balancing when client uses the ZK info in JDBC url. ZooKeeper responds to client requests by randomly passing a link to one of the active HS2 instances.

deepesh1 · ‎10-06-2015

One more thing you can try is to give the absolute path to the hive-site.xml in job-xml tag.

deepesh1 · ‎10-06-2015

Any name other than hive-site.xml should have potentially worked. Will update here if I think of anything else. One question (maybe a stupid one) but have you uploaded this on HDFS after changes?

deepesh1 · ‎10-05-2015

I think it should have worked, anyways a workaround you could try is to rename hive-site.xml to oozie-hive-site.xml under script directory and use that path in the workflow xml.

deepesh1 · ‎10-05-2015

I doubt if you can build rpm on your own easily as HDP uses custom packaging scheme /usr/hdp/<version>/<component>. I don't think Maven is used to build these. @ashish@hortonworks.com, @gkesavan@hortonworks.com can you confirm?

deepesh1 · ‎10-05-2015

Apache Drill supports JSON as self describing data format, you can find the usage here. In Hive, HCatalog supports JSON as serde format for reading and writing data into tables.

deepesh1 · ‎10-04-2015

You start the datanodes as root user in secure setup. Did you do that?

deepesh1 · ‎10-04-2015

One possibility this can happen if you previously were on Hive 0.12 and you upgraded the binaries to Hive 0.13 and started metastore, likely datanucleus.autoCreateSchema was set to true. This may have already upgraded some of the tables including the table DBS. Now when you run the schemaTool it fails.

deepesh1 · ‎10-04-2015

Flume provides a Host interceptor that will inserts a header with key host or a configured key whose value is the hostname or IP address of the host, based on configuration. If you want to set a specific key value in the header you can use the Static Interceptor. You should be able to use %{host} in a latter sink. If the host name that you want to add is the hostname where the agent is running and its an HDFS sink you can use %{host} directly without the need of interceptor, see here.

Online	Offline
Last Visited	‎05-09-2018 06:03 AM

Date Registered	‎09-25-2015 05:14 PM
Last Visited	‎05-09-2018 06:03 AM
Total Messages Posted	356
Total Kudos Received	379

Re: Hive 2 is missing after HDP upgrade

Re: Is Hortonworks HDP 2.6.0 supports Centos 6.9

Re: Add Service - BigSQL fails

Re: hi , I am having trouble connecting to beeline...

Re: Hive Metastore throws a null pointer exception

Re: Where do I find Hive ODBC Driver for operating...

Re: How to Enable Zookeeper Discovery for HiveServ...

Re: Oozie Non Default Database Setup

Re: How to schedule an Oozie job to run at 8PM EDT...

Re: With Hive View what causes: "H100 Unable to su...

Re: Hive CLI unresponsive

Re: What is the recommended method to ingest Binar...

Re: using Oracle as the database for Ambari, Hive ...

Re: Hive CLI unresponsive

Re: Hive CLI unresponsive

Re: How to deploy 2 HiveServer2 processes on same ...

Re: What is the best way to implement row-based se...

Re: Per node label capacity queues HDP 2.3

Re: Where can I find the latest Hive ODBC Driver f...

Re: Can you disable Column Pruner?

Re: Where can I find the latest Hive ODBC Driver f...

Re: Is there a way to get Ticket expiration time f...

Re: How to deploy 2 HiveServer2 processes on same ...

Re: Unable to run CTAS query using external table ...

Re: Unable to run CTAS query using external table ...

Re: Can't drop hive tables after moving hive into ...

Re: How to Configure Ranger Repository with Multip...

Re: Failing Oozie Launcher threw exception, hive-s...

Re: Failing Oozie Launcher threw exception, hive-s...

Re: Failing Oozie Launcher threw exception, hive-s...

Re: Building rpm from HDP tar.gz

Re: How can you query using JSON against HDP?

Re: Kerberos Implementation - Datanodes wont start

Re: Upgrading MetaStore schema from 0.12.0 to 0.13...

Re: Can we use variable hostname for flume sink co...