Member since
06-28-2019
8
Posts
6
Kudos Received
0
Solutions
09-27-2019
09:28 AM
1 Kudo
I made the same experience already end of August, however only when trying to deploy with Ambari 2.7.4 I created an issue in https://github.com/hortonworks/ansible-hortonworks/issues/166 (btw highly recommended way to use Ansible with those playbooks to deploy HDP, if you cannot use cloudbreak !) as I initially thought it might be a bug in those ansible playbooks. But it was clearly the problem that the provided HDF mpacks are incompatible with Ambari 2.7.4 (as can be seen from the Exception infos in the issue link above). However the good news: I got a cluster deployed using latest HDP 3.1.4 and HDF 3.4.1.1, simply by using Ambari 2.7.3 (except I missed something, and I just thought that the newest versions were indeed used)
... View more
06-28-2019
02:47 AM
I did a quick research on why rhel7/centos7 still contains such an old mysql java connector/driver version (see "Redhat response"), and checked for working alternative RPMs (as installing via an rpm has many advantages vs a manual tarball install) There are many working RPMs (I tested all of below, with CDH 6.2.0 and openjdk-8): - With Java >=v8, I recommend following: Ref: https://centos.pkgs.org/7/mysql-connectors-i386/mysql-connector-java-8.0.16-1.el7.noarch.rpm.html RPM: yum install http://repo.mysql.com/yum/mysql-connectors-community/el/7/i386//mysql-connector-java-8.0.16-1.el7.noarch.rpm - With Java v7 you can try the latest 5.1.x from Fedora: Ref: https://fedora.pkgs.org/29/fedora-x86_64/mysql-connector-java-5.1.38-7.fc29.noarch.rpm.html RPM: http://download-ib01.fedoraproject.org/pub/fedora/linux/releases/29/Everything/x86_64/os/Packages/m/mysql-connector-java-5.1.38-7.fc29.noarch.rpm - Find RPMs for other distros: https://pkgs.org/download/mysql-connector-java - Official Redhat response: Alternatively, for Java-8, Redhat proposes to use the MariaDB client: https://bugzilla.redhat.com/show_bug.cgi?id=1684349#c7 > this is the best we could do for the customers who need a recent version of the JDBC driver for MySQL/MariaDB. More infos: https://developers.redhat.com/blog/2019/06/25/mariadb-10-3-now-available-on-red-hat-enterprise-linux-7/ For centos7: https://centos.pkgs.org/7/centos-sclo-rh-testing-x86_64/rh-mariadb103-mariadb-java-client-2.4.1-1.el7.noarch.rpm.html I tested the 1-line yum install, but CDH would require more changes, due to the driver installed in /opt: yum install https://buildlogs.centos.org/centos/7/sclo/x86_64/rh/rh-mariadb103/rh-mariadb103-mariadb-java-client-2.4.1-1.el7.noarch.rpm https://buildlogs.centos.org/centos/7/sclo/x86_64/rh/rh-mariadb103/rh-mariadb103-runtime-3.3-3.el7.x86_64.rpm
... View more
02-28-2019
06:04 PM
1 Kudo
1. question: Is this upgrade supported? The Doc suggests this is supported (and why shouldn't it), after reading * table 2 from https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/planning-your-deployment/content/deployment_scenarios.html * https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-upgrade-paths.html: HDF upgrade paths: "HDF 3.2.0" , Supported HDP versions – 3.0.x, 3.1.0 2. question: The exact Upgrade path? Reference Doc: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-pre-upgrade-tasks.html The initial upgrade steps from the Documentation are clear to me (that I assume have to be done for both Upgrade cases: on HDP or pure HDF cluster): * Pre-upgrade tasks: All clear * Upgrade Ambari and the HDF Management Pack: All clear and done Then in https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-upgrade-hdf-container.html it is not clear to me * Upgrading an HDF Cluster : 3.Q: Is it required ? (I assumed this is only required on a pure HDF cluster, though some steps there look they could be useful also on HDP cluster) * Upgrading HDF 3.2.0 services on an HDP cluster: Required (it exactly reflects my upgrade case) ** Upgrade HDP : As I learned (too late), this step should have been done after initial HDF-Upgrade steps. (Because I had done the HDP upgrade 3.0.1 -> 3.1.0 as a separate Upgrade, leaving HDF services untouched. Big mistake ?!) *** In step 1.Select the HDP 3.0.x version to which you want to upgrade, and update the base URLs for HDP, HDF, and HDP UTILs. : 4. Question: What if you forget to provide (updated) HDF base URLs in this step? (Any chance to fix this later? I managed to updated/add a new HDF-3.3.1 repo URL later to the existing HDP-3.1.0.0 stack, but from there Ambari did not offer any way to install/upgrade new HDF packages) ** Upgrade HDF services : clear again I tried to ask simple, atomic questions, to understand what are my options in the situation I got into: In fact I did not follow the exact above order of steps (because initially I only read the HDP Upgrade Doc), so my plan was: * 1.phase (DONE) HDP-3.0.1 (+HDF 3.2.0 services) was upgraded to HDP-3.1.0 (HDF services untouched) * 2.phase (topic here) HDF Upgrade from 3.2.0 to 3.3.1 on the same cluster (TODO describe detailed steps I tried. In short I followed the Doc, and did not follow Upgrading an HDF Cluster (see above) , but later no surprise that Upgrade HDF services failed, because no yum repo, or new HDF rpms installed) New questions arising from that situation: * 5. Question: What is the upgrade path from HDP 3.2 to HDF 3.3 (3.3.1) on an HDP 3.1 Cluster (that already had HDF 3.2 services installed, and was upgraded from HDP 3.0.1 to HDP 3.1.0) ? * 6. Question: If my 2-phase upgrade path is not officially supported, is there still a way to get only the HDF 3.2.0 services upgraded? ** 6b. Question: Maybe manually installing the updated HDF repos and the updated rpms (on centos7 with yum install) ? Would this be promising way out? Major concerns? ** 6c. Question: Would first de-installing all HDF services help? (and then again ensure HDF repo updated to 3.3.1, and then follow the documentation 'Install HDF services on existing HDP 3.x cluster') Thanks in advance! Any hints welcome (no need to know answers to all questions) :+1: Maybe @Jay Kumar SenSharma, @Akhil S Naik or @Saurabh ? --- (Tomorrow I will merge over my learnings from related question, that was in the wrong *track*: https://community.hortonworks.com/questions/242176/how-to-complete-an-hdf-331-upgrade-on-hdp-310-clus.html )
... View more
02-26-2019
05:27 PM
@Jasper did you find a better solution, that to re-install (first HDP than HDF) ? Anyway AFAIU indeed it was/is not supported to start with an HDF cluster, and then add HDP (components), but only the other way around!
... View more
02-26-2019
04:17 PM
Same question here! On a kerberos enabled cluster, where we do not (yet) want to have (all) UIs properly (SPNEGO) kerberized, I did not find a (documented) way to get the Schema Registry working without authentication. --- However i found a work around (hack), to use the registry w//out security, by adding a line security_enabled=False directly in following file (version used in HDF-3.2.0), before Line 101: /var/lib/ambari-agent/cache/common-services/REGISTRY/0.3.0/package/scripts/params.py Plz note: I'ld prefer to get rid of that hack but I could not find any proper Doc in the HWX HDF Docs. The most detailed Registry Doc I found (that does not directly relate to the respective Ambari registry configs): https://registry-project.readthedocs.io/en/latest/security.html#spnego
... View more
02-25-2019
07:13 PM
3 Kudos
Issue Intro We ran into this problem, in the 2. part of our cluster upgrade, that we planned in 2 phases: * 1.phase (DONE) HDP-3.0.1 (with HDF 3.2.0 services) was upgraded to HDP-3.1.0 (HDF services untouched, since I didn't know upcoming issue) * 2.phase (topic here) HDF Upgrade from 3.2.0 to 3.3.1 in the same cluster AFAIU my (Ambari managed) HDF Upgrade reached a dead end, after I omitted an important step (learned later!) during the HDP upgrade (1.phase), namely providing the upgraded HDF repo. Below I described in detail the steps taken in my 'HDF Upgrade' try, to give more context. Now it seems not possible (at least not in the Ambari UI), to add the HDF-3.3 repo to existing (HDP-3.1.0) stack. Possible Solutions ?! Now I am wondering, what is the best way out of this? Ideally there would be a way to add the new HDF repo 'manually' * using the Ambari REST API ? * or directly in the Ambari DB ( maybe TABLE "repo_version" ?) * or is the strategy to create a new 'Target Version' , and then to upgrade to that? ** new target version "HDP-3.1" ? ** new target version "HDF-3.3" ? (I 'ld assume this option is correct only on a pure HDF cluster). ps: 3.3 is only shown (when the hdf mpack was upgraded to 3.3) Thanks in advance! ---- HDP Upgrade Steps done: I followed the Doc, all steps, with entry point: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-pre-upgrade-tasks.html * "Pre-upgrade' steps": went through them, nothing special * "Upgrade Ambari and the HDF Management Pack" As I already upgraded Ambari (to v2.7.3.0) during recent HDP (3.1.0) upgrade, nothing todo here, so I jumped to: * "Upgrade the HDF Management Pack" Done the upgrade of the macp (to hdf 3.3.1) * "Upgrade the Ambari Database Schema" Nothing done (again Ambari upgrade was done before) * "Confirm the Ambari upgrade for HDF services" Then the next section, in my case is "Upgrading HDF 3.2.0 services on an HDP cluster" Ref: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-upgrade-hdf-320-services-on-hdp.html * "Upgrade HDP" This was already done before (I started HDF upgrade), so I went to the next section. But now already it came to me, that I might have done a mistake, that in that HDP Upgrade, I did *not* care about already providing the right (also upgraded) HDF Repo URLs (instead I had kept the HDF to current v3.2.0) * "Upgrade HDF services" Step 3): >Use hdf-select to ensure appropriate links to new installed version. Now this doesn't work (no wonder no new HDF/NiFi rpms not installed yet). Also "hdf-select versions" only shows the current HDF 3.2.0 Add. notes: After I restarted Ambari, I noted that the Nifi+registry services completely disappeared from the Ambari View (also found some warnings in ambari-server log). To fix that situation, I had to un-install the (upgraded) hdf mpack, and re-install the HDF-3.2.0 mpack and again restart ambari, then those HDF services were back. And as I understood later, the problem was upgrading only the mpack without that the new HDF(3.3) repo and rpms were installed (due to my omission of the HDF repo upgrade..) ---- Related posts that did not help in my case: * https://community.hortonworks.com/questions/167396/how-to-upgrade-hdf-on-an-existing-hdp-cluster.html * https://community.hortonworks.com/articles/167366/how-to-perform-rolling-upgrade-from-hdf-302-to-hdf.html : seems only for a pure HDF cluster (upgrade)
... View more
Labels:
04-05-2017
11:13 AM
1 Kudo
Quite unbelievable that Spark is so poorly supported (in Atlas) ! I did some research, I could not find a good 'data lineage' (data Management) solution that integrates well with Spark. Very happy for other ideas! Best solution/tools I could find, that have at least spark integration (but none seem usable in a 'standalone' fashion): * Cloudera Navigator, which is both closed-source and can't be used standalone (only in a Cloudera cluster deployment) * cask.co's CDAP: http://cask.co/products/cdap/ shows nice features in section "Metadata & Lineage" * Talend's platform similarly: https://www.talend.com/blog/2016/10/10/five-pillars-for-succeeding-in-big-data-governance-and-metadata-management-with/ * LinkedIn WhereHows: Only 2 mentions searching for 'spark' on their github project, not looking promising: https://github.com/linkedin/WhereHows/issues/238
... View more
