About vgoel

vgoel · ‎08-10-2017

Since you tagged this question with HAWQ, I'm guessing you installed HAWQ on it. One likely reason this is happening is that HAWQ Ambari install will set your datanodes (where HAWQ is installed) to use overcommit of value 2, with a default ratio of 50%, which you're supposed to change based on your memory configs. This ratio should ideally be 90% or more.With 50%, most likely your services didn't get to use half of the datanode RAM. You will find this config under HAWQ service as a slider control (if overcommit is set to 2). You can either change the overcommit value to 0 on HAWQ Segment Nodes, or set it to 2, will a ratio of 90% or higher. You should update via Ambari instead of direct OS. It is strongly recommended to run HAWQ Master node at least, with overcommit value of 2. You may do this by putting it on a dedicated node and creating a separate config group for HAWQ Master(s). Hope this helps.

vgoel · ‎07-18-2017

Please ensure you're following the right instructions from here: http://hdb.docs.pivotal.io/201/hdb/index.html Software is here (use 2.0.1.2, which has important fixes on top of 2.0.1, although Ambari-related code is the same): https://network.pivotal.io/products/pivotal-hdb#/releases/4479 Not sure why you're seeing this, I haven't seen anyone run into this issue before.

vgoel · ‎07-17-2017

What version of Ambari are you using? 2.2.0 or 2.2.2 ? Remember, 2.2.2 is the minimum required. Also, what install instructinos are you following? Did the issue start happening after something specific, or is this a fresh install?

vgoel · ‎07-07-2017

What version of HAWQ/HDB is it? What PXF profile are you using? You should try the new HiveORC profile in HDB 2.2.0.0, if you haven't: http://hdb.docs.pivotal.io/220/hawq/pxf/HivePXF.html#hiveorc-intro

vgoel · ‎06-19-2017

This will need further test/ investigation. If you have the ability to log a support ticket, please do.

vgoel · ‎06-16-2017

Hi Timo, have you tried this yet (need to be on HDB 2.0.1 or above)? Any challenges with it? https://hdb.docs.pivotal.io/220/hawq/pxf/JsonPXF.html

vgoel · ‎06-05-2017

You're probably looking at HAWQ processes that are used to cache YARN containers for a few minutes after query completes in order to avoid a new RPC call to YARN RM for subsequent queries. This is configurable, see this documentation: http://hdb.docs.pivotal.io/220/hawq/resourcemgmt/YARNIntegration.html It's not clear to me at the moment what the second process is, not sure if it is the postmaster process on port 5432, doesn't look like it. You can compare the processes by going from YARN mode to Standalone mode temporarily via Ambari and comparing.

vgoel · ‎05-31-2017

Priyaranjan, are you trying to run this on Amazon Linux ? If so, that could be the issue as it is not a certified OS yet. Perhaps you can use a RHEL/CentOS based AMI. Also, Ambari 2.5 hasn't been certified yet, although it's possible that it might just work fine as-is. See: http://hdb.docs.pivotal.io/220/hdb/releasenotes/HAWQ220ReleaseNotes.html#topic_g53_tgv_2v

vgoel · ‎05-25-2017

Peter, sounds like something that you need to handle on the python script, not HAWQ (you can post your Python script here as well). You may want to post the scenario on Python community discussions for more ideas. Also see this, in case it helps: https://stackoverflow.com/questions/25260088/paramiko-with-continuous-stdout Another troubleshooting exercise you could do is to install the psql client on your client machine and see if psql works fine by itself (just to prove that HAWQ server and client are fully functional). Instructions for psql client below: user-created-hawq-psql-client-pivotal-hdb-docs.pdf

vgoel · ‎05-24-2017

Peter, can you provide more details? What kind of external table? What data format? Is HAWQ processing your query, or just provisioning your data as a file output? What query interface are you using? What happens when you use psql ?

vgoel · ‎04-14-2017

CentOS and Red Hat have the same code base, so CentOS v7.x support implies that RHEL is supported as well.

vgoel · ‎03-23-2017

A colleague once ran into something similar on HDP 2.3.x (seen in this older version only), and did the following: === After we added the following properties to Ranger KMS (via Ambari, Ranger KMS Config tab) and restarted cluster services, it appears that HAWQ is running successfully on YARN. hadoop.kms.proxyuser.rm.users=* hadoop.kms.proxyuser.rm.hosts=* This allowed us to get past the AUTHORIZATION error in Ranger KMS logs due to renewer rm user not being allowed as a proxy to get delegation token for accounts.

vgoel · ‎02-07-2017

Could you please clarify your question further - are you asking about data integration with DB2, or data transfer with DB2, or using DB2 for storing HAWQ metadata? HAWQ is an analytic database itself, built using Postgres database technology. It doesn't use DB2 in its architecture, but you could migrate data from DB2 into HAWQ if you wish to do so. HAWQ stores its metadata internally in the same instance and doesn't support external databases like mysql or oracle ...etc for its metastore.

vgoel · ‎02-02-2017

Did this happen from the beginning right after install? You'll have to check what the logs say on the Standby master: http://hdb.docs.pivotal.io/211/hawq/admin/logfiles.html And is port 5432 in use already on Standby Master node by another postgres instance? What else is located on that node? Looks like standby master didn't get initialized.

vgoel · ‎01-27-2017

How to use Ambari View for HAWQ Query Monitoring? This beta version of Ambari HAWQ View provides live query monitoring directly in Ambari UI, displaying a list of queries running in a HAWQ cluster. This allows admins to see the database activity on the UI in a user-friendly interface. The view front-end is based on EmberJS framework. The Java backend extends the framework provided by the ambari-views project. The front-end polls the REST API every 5 seconds, querying the pg_stat_activity view and rendering reformatted data on the UI. How to Deploy: Deployment simply requires copying the hawq-view jar to the ambari-server host and restarting ambari-server. ssh into ambari_server_host $ cd /var/lib/ambari-server/resources/views/ $ curl -O https://s3-us-west-2.amazonaws.com/pivotal-ambari/stable/hawq-view/hawq-view-1.0.0.0-1.jar $ ambari-server restart Creating an Instance of HAWQ View: a) The HAWQ View instance connects to the HAWQ Master through JDBC. Ensure that HAWQ pg_hba.conf is updated to allow JDBC connections from the ambari_server_host for the HAWQ View backend user account with access to pg_stat_activity view (you may use use gpadmin out-of-the-box for demo purposes). SSH into the HAWQ Master host and update pg_hba.conf (in HAWQ Master Directory, e.g /data/hawq/master) to allow connections. Example of entry needed in pg_hba.conf, where 192.168.64.101 is the ambari_server_host and gpadmin is used to run the HAWQ View instance: host all gpadmin 192.168.64.101/32 trust Restart HAWQ Master via Ambari or CLI for pg_hba.conf changes to take effect. b) Ensure that the steps above have been completed before proceeding. In Ambari UI, navigate to Admin > Manage Ambari > Views page. You should see a HAWQ view in the list. Click on Create Instance under HAWQ View. Under Settings section, provide the HAWQ username and password of the backend user account with access to pg_stat_activity view (the same user that was added to pg_hba.conf above). Once saved, the view will be created and operational. Submit some (long running) queries and ensure they are displayed on the HAWQ View. You may hover over the queries to see the SQL statements. HAWQ Sandbox You can always download a Sandbox VM with HAWQ pre-installed here: https://network.pivotal.io/products/pivotal-hdb Feedback Please give this Ambari View a try and provide feedback by posting your comments below.

vgoel · ‎11-16-2016

There could be a permission issue with the user account running Ambari installation and not having permission to delete directory /var/run/ambari-server/stack-recommendations/1. Check the directory permissions and try a user with permissions on the Ambari directory. I haven't seen this error message before.

vgoel · ‎09-02-2016

I'll just add two more bits that I have seen some people run into often. Hopefully this is helpful in similar scenarios. 1) /etc/ssh/sshd_config file: PasswordAuthentication should be set to yes 2) In AWS, /etc/cloud/cloud.cfg should have ssh_pwauth set to True (or 1, I think). Otherwise, cloud init keeps disabling PasswordAuthentication options in ssh config.

vgoel · ‎08-25-2016

@Ancil McBarnett I'm curious as to why you ran into this issue. Ambari should have automatically created gpadmin account and exchanged keys for you. Unless you manually created gpadmin account, and gave a different password on HAWQ config screen in Ambari during install? http://hdb.docs.pivotal.io/hdb/install/install-ambari.html

vgoel · ‎08-01-2016

@Eric Lemmon What tutorial link are you using? If you are using this: http://hortonworks.com/hadoop-tutorial/getting-started-with-pivotal-hawq-on-hortonworks-sandbox/ ... then note that the above mentioned tutorial was designed for HAWQ 1.3, and is not a good one to use with 2.0 since a lot has changed. While a new tutorial is being crafted, you can try a few things previously posted by @Ali Bajwa and @Zack Odom: https://community.hortonworks.com/repos/48038/lending-club-demo.html https://community.hortonworks.com/articles/34193/install-hdb-hawq-via-ambari-and-use-zeppelin-for-v.html Finally, yes, the link above you mentioned a good place to start for using basic HAWQ commands, many of the commands are available via Ambari as well.

vgoel · ‎07-28-2016

The links have been fixed. Now there is one generic home page link. Thanks!

vgoel · ‎07-14-2016

Are you talking about an error with the installation or an issue with the documentation site? Docs are here: http://hdb.docs.pivotal.io Btw, HDP 2.5 is not supported yet.

Online	Offline
Last Visited	‎05-05-2018 04:29 PM

Date Registered	‎01-15-2016 02:58 PM
Last Visited	‎05-05-2018 04:29 PM
Total Messages Posted	28
Total Kudos Received	6

Re: HAWQ Issues with Ranger KMS

Re: Page not Found for HAWQ Automated (with Amabar...

Re: HDP- Datanodes are crashing

Re: Hawq 2.0.1 stucks forever at "Customize servic...

Re: Hawq 2.0.1 stucks forever at "Customize servic...

Re: Hawq Queries Very Slow

Re: How to create HAWQ external table with complex...

Re: How to create HAWQ external table with complex...

Re: HAWQ Yarn containers are running forever not g...

Re: Installing HAWQ 2.2.0.0 on Ambari 2.5 giving b...

Re: ambari hdp 2.4 sandbox with hawq python progra...

Re: ambari hdp 2.4 sandbox with hawq python progra...

Re: Is it true that Ambari 2.5 is not supported on...

Re: HAWQ Issues with Ranger KMS

Re: Does HAWQ on HDP support DB2?

Re: Why my Hawq stand by master Isn't runing ? And...

Ambari View for live HAWQ Query Monitoring

Re: ERROR 500 received - when installing the PIVOT...

Re: Error Installing HDB HAWQ Standby Master

Re: Error Installing HDB HAWQ Standby Master

Re: Outdated HAWQ tutorial?

Re: Page not Found for HAWQ Automated (with Amabar...

Re: Page not Found for HAWQ Automated (with Amabar...