Issue Intro We ran into this problem, in the 2. part of our cluster upgrade, that we planned in 2 phases: * 1.phase (DONE) HDP-3.0.1 (with HDF 3.2.0 services) was upgraded to HDP-3.1.0 (HDF services untouched, since I didn't know upcoming issue) * 2.phase (topic here) HDF Upgrade from 3.2.0 to 3.3.1 in the same cluster AFAIU my (Ambari managed) HDF Upgrade reached a dead end, after I omitted an important step (learned later!) during the HDP upgrade (1.phase), namely providing the upgraded HDF repo. Below I described in detail the steps taken in my 'HDF Upgrade' try, to give more context. Now it seems not possible (at least not in the Ambari UI), to add the HDF-3.3 repo to existing (HDP-3.1.0) stack. Possible Solutions ?! Now I am wondering, what is the best way out of this? Ideally there would be a way to add the new HDF repo 'manually' * using the Ambari REST API ? * or directly in the Ambari DB ( maybe TABLE "repo_version" ?) * or is the strategy to create a new 'Target Version' , and then to upgrade to that? ** new target version "HDP-3.1" ? ** new target version "HDF-3.3" ? (I 'ld assume this option is correct only on a pure HDF cluster). ps: 3.3 is only shown (when the hdf mpack was upgraded to 3.3) Thanks in advance! ---- HDP Upgrade Steps done: I followed the Doc, all steps, with entry point: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-pre-upgrade-tasks.html * "Pre-upgrade' steps": went through them, nothing special * "Upgrade Ambari and the HDF Management Pack" As I already upgraded Ambari (to v2.7.3.0) during recent HDP (3.1.0) upgrade, nothing todo here, so I jumped to: * "Upgrade the HDF Management Pack" Done the upgrade of the macp (to hdf 3.3.1) * "Upgrade the Ambari Database Schema" Nothing done (again Ambari upgrade was done before) * "Confirm the Ambari upgrade for HDF services" Then the next section, in my case is "Upgrading HDF 3.2.0 services on an HDP cluster" Ref: https://docs.hortonworks.com/HDPDocuments/HDF3/HDF-3.3.1/ambari-managed-hdf-upgrade/content/hdf-upgrade-hdf-320-services-on-hdp.html * "Upgrade HDP" This was already done before (I started HDF upgrade), so I went to the next section. But now already it came to me, that I might have done a mistake, that in that HDP Upgrade, I did *not* care about already providing the right (also upgraded) HDF Repo URLs (instead I had kept the HDF to current v3.2.0) * "Upgrade HDF services" Step 3): >Use hdf-select to ensure appropriate links to new installed version. Now this doesn't work (no wonder no new HDF/NiFi rpms not installed yet). Also "hdf-select versions" only shows the current HDF 3.2.0 Add. notes: After I restarted Ambari, I noted that the Nifi+registry services completely disappeared from the Ambari View (also found some warnings in ambari-server log). To fix that situation, I had to un-install the (upgraded) hdf mpack, and re-install the HDF-3.2.0 mpack and again restart ambari, then those HDF services were back. And as I understood later, the problem was upgrading only the mpack without that the new HDF(3.3) repo and rpms were installed (due to my omission of the HDF repo upgrade..) ---- Related posts that did not help in my case: * https://community.hortonworks.com/questions/167396/how-to-upgrade-hdf-on-an-existing-hdp-cluster.html * https://community.hortonworks.com/articles/167366/how-to-perform-rolling-upgrade-from-hdf-302-to-hdf.html : seems only for a pure HDF cluster (upgrade)
... View more
