1. question: Is this upgrade supported? The Doc suggests this is supported (and why shouldn't it), after reading * table 2 from https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/planning-your-deployment/content/deployment_scenarios.html * https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-upgrade-paths.html: HDF upgrade paths: "HDF 3.2.0" , Supported HDP versions – 3.0.x, 3.1.0 2. question: The exact Upgrade path? Reference Doc: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-pre-upgrade-tasks.html The initial upgrade steps from the Documentation are clear to me (that I assume have to be done for both Upgrade cases: on HDP or pure HDF cluster): * Pre-upgrade tasks: All clear * Upgrade Ambari and the HDF Management Pack: All clear and done Then in https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-upgrade-hdf-container.html it is not clear to me * Upgrading an HDF Cluster : 3.Q: Is it required ? (I assumed this is only required on a pure HDF cluster, though some steps there look they could be useful also on HDP cluster) * Upgrading HDF 3.2.0 services on an HDP cluster: Required (it exactly reflects my upgrade case) ** Upgrade HDP : As I learned (too late), this step should have been done after initial HDF-Upgrade steps. (Because I had done the HDP upgrade 3.0.1 -> 3.1.0 as a separate Upgrade, leaving HDF services untouched. Big mistake ?!) *** In step 1.Select the HDP 3.0.x version to which you want to upgrade, and update the base URLs for HDP, HDF, and HDP UTILs. : 4. Question: What if you forget to provide (updated) HDF base URLs in this step? (Any chance to fix this later? I managed to updated/add a new HDF-3.3.1 repo URL later to the existing HDP-3.1.0.0 stack, but from there Ambari did not offer any way to install/upgrade new HDF packages) ** Upgrade HDF services : clear again I tried to ask simple, atomic questions, to understand what are my options in the situation I got into: In fact I did not follow the exact above order of steps (because initially I only read the HDP Upgrade Doc), so my plan was: * 1.phase (DONE) HDP-3.0.1 (+HDF 3.2.0 services) was upgraded to HDP-3.1.0 (HDF services untouched) * 2.phase (topic here) HDF Upgrade from 3.2.0 to 3.3.1 on the same cluster (TODO describe detailed steps I tried. In short I followed the Doc, and did not follow Upgrading an HDF Cluster (see above) , but later no surprise that Upgrade HDF services failed, because no yum repo, or new HDF rpms installed) New questions arising from that situation: * 5. Question: What is the upgrade path from HDP 3.2 to HDF 3.3 (3.3.1) on an HDP 3.1 Cluster (that already had HDF 3.2 services installed, and was upgraded from HDP 3.0.1 to HDP 3.1.0) ? * 6. Question: If my 2-phase upgrade path is not officially supported, is there still a way to get only the HDF 3.2.0 services upgraded? ** 6b. Question: Maybe manually installing the updated HDF repos and the updated rpms (on centos7 with yum install) ? Would this be promising way out? Major concerns? ** 6c. Question: Would first de-installing all HDF services help? (and then again ensure HDF repo updated to 3.3.1, and then follow the documentation 'Install HDF services on existing HDP 3.x cluster') Thanks in advance! Any hints welcome (no need to know answers to all questions) :+1: Maybe @Jay Kumar SenSharma, @Akhil S Naik or @Saurabh ? --- (Tomorrow I will merge over my learnings from related question, that was in the wrong *track*: https://community.hortonworks.com/questions/242176/how-to-complete-an-hdf-331-upgrade-on-hdp-310-clus.html ) ... View more

Issue Intro We ran into this problem, in the 2. part of our cluster upgrade, that we planned in 2 phases: * 1.phase (DONE) HDP-3.0.1 (with HDF 3.2.0 services) was upgraded to HDP-3.1.0 (HDF services untouched, since I didn't know upcoming issue) * 2.phase (topic here) HDF Upgrade from 3.2.0 to 3.3.1 in the same cluster AFAIU my (Ambari managed) HDF Upgrade reached a dead end, after I omitted an important step (learned later!) during the HDP upgrade (1.phase), namely providing the upgraded HDF repo. Below I described in detail the steps taken in my 'HDF Upgrade' try, to give more context. Now it seems not possible (at least not in the Ambari UI), to add the HDF-3.3 repo to existing (HDP-3.1.0) stack. Possible Solutions ?! Now I am wondering, what is the best way out of this? Ideally there would be a way to add the new HDF repo 'manually' * using the Ambari REST API ? * or directly in the Ambari DB ( maybe TABLE "repo_version" ?) * or is the strategy to create a new 'Target Version' , and then to upgrade to that? ** new target version "HDP-3.1" ? ** new target version "HDF-3.3" ? (I 'ld assume this option is correct only on a pure HDF cluster). ps: 3.3 is only shown (when the hdf mpack was upgraded to 3.3) Thanks in advance! ---- HDP Upgrade Steps done: I followed the Doc, all steps, with entry point: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-pre-upgrade-tasks.html * "Pre-upgrade' steps": went through them, nothing special * "Upgrade Ambari and the HDF Management Pack" As I already upgraded Ambari (to v2.7.3.0) during recent HDP (3.1.0) upgrade, nothing todo here, so I jumped to: * "Upgrade the HDF Management Pack" Done the upgrade of the macp (to hdf 3.3.1) * "Upgrade the Ambari Database Schema" Nothing done (again Ambari upgrade was done before) * "Confirm the Ambari upgrade for HDF services" Then the next section, in my case is "Upgrading HDF 3.2.0 services on an HDP cluster" Ref: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-upgrade-hdf-320-services-on-hdp.html * "Upgrade HDP" This was already done before (I started HDF upgrade), so I went to the next section. But now already it came to me, that I might have done a mistake, that in that HDP Upgrade, I did *not* care about already providing the right (also upgraded) HDF Repo URLs (instead I had kept the HDF to current v3.2.0) * "Upgrade HDF services" Step 3): >Use hdf-select to ensure appropriate links to new installed version. Now this doesn't work (no wonder no new HDF/NiFi rpms not installed yet). Also "hdf-select versions" only shows the current HDF 3.2.0 Add. notes: After I restarted Ambari, I noted that the Nifi+registry services completely disappeared from the Ambari View (also found some warnings in ambari-server log). To fix that situation, I had to un-install the (upgraded) hdf mpack, and re-install the HDF-3.2.0 mpack and again restart ambari, then those HDF services were back. And as I understood later, the problem was upgrading only the mpack without that the new HDF(3.3) repo and rpms were installed (due to my omission of the HDF repo upgrade..) ---- Related posts that did not help in my case: * https://community.hortonworks.com/questions/167396/how-to-upgrade-hdf-on-an-existing-hdp-cluster.html * https://community.hortonworks.com/articles/167366/how-to-perform-rolling-upgrade-from-hdf-302-to-hdf.html : seems only for a pure HDF cluster (upgrade) ... View more