Hi all, I have an HDP ambari managed cluster with HDF-3.0.1 management pack installed. Ambari release is 2.6.1. I have successfully upgraded the HDF management pack to version HDF-3.1.0, but when I click on the "Install on... <cluster name>" button from the Admin view I been redirected to the "manage version" tab in Ambari, in which I'm only able to see the existing HDP version (2.6.4). How can I perform the upgrade from HDF-3.0.1 to HDF-3.1.0? thank you, ... View more

Hi all, I see in the HDP 2.5.3 documentation that mysql 5.6 is no longer supported as hive nor oozie metastore backend. (hdp docs), but it was in the previous release(hdp docs). Is it an error in the documentation? What if I need to upgrade HDP from 2.4.x to HDP 2.5.x? Also, is mysql 5.7 supported? Thank you very much ... View more

Repo Description Ambari service to run and manage Apache Drill. For more information about Apache Drill visit https://drill.apache.org/ Requirements: - RHEL/CENTOS 7.1 - Ambari 2.X - HDP 2.4 - You need HDFS and Zookeeper up & running on your cluster Features: - Allows to install Apache Drill on an Ambari-managed cluster - Edit drill-overrides.conf and drill-env.sh via ambari - Integration with zookeeper Repo Info Github Repo URL https://github.com/dvergari/ambari-drill-service Github account name dvergari Repo name ambari-drill-service ... View more

In this article, we will install Apache Hawq 2.0.0.0_beta in a cluster composed by: 2 masters (1 active, 1 standby) 3 segments (slaves) On each node on the cluster install repository to fetch libhdfs3: [root@hdpmaster01~]# curl -s -L "https://bintray.com/wangzw/rpm/rpm" -o /etc/yum.repos.d/bintray-wangzw-rpm.repo [root@hdpmaster01~]# Install epel repository: [root@hdpmaster01~]# yum -y install epel-release Install missing dependencies: [root@hdpamaster01~]# yum -y install man passwd sudo tar which git mlocate links make bzip2 net-tools autoconf automake libtool m4 gcc gcc-c++ gdb bison flex cmake gperf maven indent libuuid-devel krb5-devel libgsasl-devel expat-devel libxml2-devel perl-ExtUtils-Embed pam-devel python-devel libcurl-devel snappy-devel thrift-devel libyaml-devel libevent-devel bzip2-devel openssl-devel openldap-devel protobuf-devel readline-devel net-snmp-devel apr-devel libesmtp-devel xerces-c-devel python-pip json-c-devel libhdfs3-devel apache-ivy java-1.7.0-openjdk-devel openssh-clients openssh-server Install postgresql-devel to compile python dependencies [root@hdpmaster01~]# yum install postgresql-devel Now, install python dependencies with pip: pip install pg8000 simplejson unittest2 pycrypto pygresql pyyaml lockfile paramiko psi You can now remove postgresql-*, be sure to not erase existing psql instances Download the source code from github: [root@hdpmaster01~]# cd /root [root@hdpmaster01~]# git clone https://github.com/apache/incubator-hawq.git Cloning into 'incubator-hawq'... remote: Counting objects: 34883, done. remote: Total 34883 (delta 0), reused 0 (delta 0), pack-reused 34883 Receiving objects: 100% (34883/34883), 144.95 MiB | 30.04 MiB/s, done. Resolving deltas: 100% (21155/21155), done. [root@hdpmaster01~]# Before compile hawq, you need to compile and install libyarn, c/c++ interface to yarn, that is shipped with the hawq source code [root@hdpmaster01~]# cd /root/incubator-hawq/depends/libyarn/ && mkdir build/ && cd build [root@hdpmaster01 build]# pwd /root/incubator-hawq/depends/libyarn/build [root@hdpmaster01 build]# [root@hdpmaster01 build]# ../bootstrap [...] bootstrap success. Run "make" to build. [root@hdpmaster01 build]# make -j && make install [...] -- Installing: /root/incubator-hawq/depends/libyarn/dist/include/libyarn/records/YARN_containermanagement_protocol.pb.h -- Installing: /root/incubator-hawq/depends/libyarn/dist/include/libyarn/records/YARNSecurity.pb.h -- Installing: /root/incubator-hawq/depends/libyarn/dist/include/libyarn/libyarncommon/Token.h [root@hdpmaster01 build]# Copy the include dir and the lib dir in the correct file system path, and make the library visible to the operating system with ldconfig: [root@hdpmaster01 build]# cp -R /root/incubator-hawq/depends/libyarn/dist/include/libyarn/ /usr/include/ [root@hdpmaster01 build]# cp /root/incubator-hawq/depends/libyarn/dist/lib/libyarn.so.0.1.13 /usr/lib64/ [root@hdpmaster01 build]# [root@hdpmaster01 build]# ln -s /usr/lib64/libyarn.so.0.1.13 /usr/lib64/libyarn.so.1 [root@hdpmaster01 build]# ln -s /usr/lib64/libyarn.so.1 /usr/lib64/libyarn.so [root@hdpmaster01 build]# ldconfig && ldconfig -p | grep libyarn libyarn.so.1 (libc6,x86-64) => /lib64/libyarn.so.1 libyarn.so (libc6,x86-64) => /lib64/libyarn.so [root@hdpmaster01 build] Now, we can compile and install apache hawq. I use /opt/ as installation directory: [root@hdpmaster01 build]# cd /root/incubator-hawq [root@hdpmaster01 incubator-hawq]# ./configure –prefix=/opt/hawq [...] [root@hdpmaster01 incubator-hawq]# make -j8 && make install [...] make[2]: Leaving directory `/root/incubator-hawq/tools/gpnetbench' make[1]: Leaving directory `/root/incubator-hawq/tools' HAWQ installation complete. Create the user hawqadmin and change the ownership for hawq installation directory [root@hdpmaster01 incubator-hawq]# useradd -s /bin/bash hawqadmin [root@hdpmaster01 incubator-hawq]# passwd hawqadmin Changing password for user hawqadmin. New password: Retype new password: passwd: all authentication tokens updated successfully. [root@hdpmaster01 incubator-hawq]# chown -R hawqadmin.hawqadmin /opt/hawq/ [root@hdpmaster01 incubator-hawq]# Repeat the previous steps on all hosts in your cluster. Now, on the primary master, create the key for user hawqadmin and distribuite the public key to the other hosts (do not set a password for your private key). As hawqadmin user: [hawqadmin@hdpmaster01~]$ ssh-keygen [...] [hawqadmin@hdpmaster01~]$ for i in hdpmaster01 hdpmaster02 hdpslave01 hdpslave02 hdpslave03; do > ssh-copy-id $i >done [...] [hawqadmin@hdpmaster01~]$ Repeat the previous loop on the standby master. On the primary master host, edit /opt/hawq/etc/hdfs-client.xml and /opt/hawq/etc/yarn-client.xml as they fit your needs (eg. For namenode and resourcemanager high availability or for kerberos authentication), then edit the following properties in hawq-site.xml <property> <name>hawq_master_address_host</name> <value>hdpmaster01</value> <description>The host name of hawq master.</description> </property> <property> <name>hawq_master_address_port</name> <value>5432</value> <description>The port of hawq master.</description> </property> <property> <name>hawq_standby_address_host</name> <value>hdpmaster02</value> <description>The host name of hawq standby master.</description> </property> <property> <name>hawq_segment_address_port</name> <value>40000</value> <description>The port of hawq segment.</description> </property> <property> <name>hawq_dfs_url</name> <value>hdfsha/hawq_default</value> <description>URL for accessing HDFS.</description> </property> <property> <name>hawq_master_directory</name> <value>/data01/hawq/masterdd</value> <description>The directory of hawq master.</description> </property> <property> <name>hawq_segment_directory</name> <value>/data01/hawq/segmentdd</value> <description>The directory of hawq segment.</description> </property> <property> <name>hawq_global_rm_type</name> <value>yarn</value> </property> <property> <name>hawq_rm_yarn_address</name> <value>hdpmaster02:8032</value> </property> <property> <name>hawq_rm_yarn_scheduler_address</name> <value>hdpmaster02:8030</value> </property> <property> <name>hawq_rm_yarn_queue_name</name> <value>default</value> <description>The YARN queue name to register hawq resource manager.</description> </property> <property> <name>hawq_rm_yarn_app_name</name> <value>hawq</value> <description>The application name to register hawq resource manager in YARN.</description> </property> You can leave the others options unchanged. NOTE: if you have a postgresql instance running on the master nodes, you must change the property hawq_master_address_port Write the slaves FQDN in the /opt/hawq/etc/slaves file e.g. [hawqadmin@hdpmaster01 etc]$ echo -e "hdpslave01\nhdpslave02\nhdpslave03" > slaves [hawqadmin@hdpmaster01 etc]$ cat slaves hdpslave01 hdpslave02 hdpslave03 Copy the configuration files on all other hosts, in the /opt/hawq/etc/ directory Now, as hdfs user, create the hawqadmin home and the data dir on hdfs [hdfs@hdpmaster01~]$ hdfs dfs -mkdir /user/hawqadmin && hdfs dfs -chown hawqadmin /user/hawqadmin [hdfs@hdpmaster01~]$ hdfs dfs -mkdir /hawq_default && hdfs dfs -chown hawqadmin /hawq_default [hdfs@hdpmaster01~]$ On both masters, create the master data dir: mkdir -p /data01/hawq/masterdd && chown -R hawqadmin /data01/hawq Create the segments data dir on all slaves mkdir -p /data01/hawq/segmentdd && chown -R hawqadmin /data01/hawq Initialize the cluster as hawqadmin user. Remember to source the environment file before execute any action (/opt/hawq/greenplum_path.sh) [hawqadmin@hdpmaster01 hawq]$ cd /opt/hawq/ [hawqadmin@hdpmaster01 hawq]$ source greenplum_path.sh [hawqadmin@hdpmaster01 hawq]$ hawq init cluster -av [...] 20160229:15:42:40:158114 hawq_init:hdpmaster01:hawqadmin-[INFO]:-Init HAWQ cluster successfully [hawqadmin@hdpmaster01 hawq]$ The init statement also starts the cluster, so you can now check the cluster state with the following command hawq state cluster You can also see the running application on YARN: [hawqadmin@hdpmaster01 ~]$ yarn application -list | awk '/application_/ {printf ("%s\t%s\t%s\t%s\t%s\n", $1,$2,$3,$4,$5)}' application_1456240841318_0026 hawq YARN hawqadmin default Now, connect to the database and create a sample table: [hawqadmin@hdpmaster01 hawq]$ psql -d postgres psql (8.2.15) Type "help" for help. postgres=# \d No relations found. postgres=# create table test (field1 int, field2 varchar(30)); CREATE TABLE postgres=# \d+ test Append-Only Table "public.test" Column | Type | Modifiers | Storage | Description --------+-----------------------+-----------+----------+------------- field1 | integer | | plain | field2 | character varying(30) | | extended | Compression Type: None Compression Level: 0 Block Size: 32768 Checksum: f Has OIDs: no Options: appendonly=true Distributed randomly postgres=# insert into test (field1, field2) values (1, 'May the hawq be with you'); INSERT 0 1 postgres=# select * from test; field1 | field2 --------+--------------------------- 1 | May the hawq be with you (1 row) postgres=# That's all! 🙂 ... View more

After installing zeppelin all goes fine, but when you try to connect you may get a "disconnected" status on the notebook tab and you're unable to create any new note, while the interpreter tab works fine. If you checked you can reach the port on which zeppelin is listen, it may be an issue with your content filter firewall and you should see this log in your firewall application: 2016-02-04 15:47:58 Deny 192.168.0.128 40.112.76.49 http/tcp 57772 9995 1-Ecube 0-Internet ProxyDeny: HTTP Invalid Request-Line Format (TCP-UDP-isoardi OUT-00) HTTP-Client.isoardi proc_id="http-proxy" rc="594" msg_id="1AFF-0005" proxy_act="HTTP-Client.isoardi" line="\x81\x8d\xf2\x9eW\xfe\x89\xbc8\x8e\xd0\xa4u\xae\xbb\xd0\x10\xdc\x8f\x81\x8d\xf8]\xb8z\x83\x7f\xd7\x0a" Traffic The solution is to disable the content filter for the domain on which zeppelin is running ... View more