Apache NiFi is currently at version 1.16.x https://nifi.apache.org/
HDF is a frozen project where NiFi is at 1.11.4
Ambari seems to be attic'd.
Is there a way to upgrade the version of NiFi packaged with HDF to a latter version of NiFi?
Is there a way to combine the two standalone apache projects of NiFi and Ambari to get the functionality of Ambari's management but with a later version of NiFi?
I just want to address the assertions in the first paragraph above.
I don't think it's quite correct to deem HDF "a frozen project". it would be more accurate to say that, as a Cloudera product, it was superseded by Cloudera DataFlow (CDF) in the Winter of 2019. The legacy product HDF has already reached it's End of Support (EoS) date in March 2022, so it is safe to assume that no new major updates with new versions of NiFi will be going into it.
There'a always some differences between the versions released by Cloudera in products such as CDF and the release of "upstream" component projects such as Apache NiFi. This is analogous to how there are differences between what mainline kernel is "current" in the open source Linux world and what RedHat, for example, releases as part of Red Hat Enterprise Linux.
Right now, Apache NiFi is a component in a product called Cloudera Flow Management (CFM). My understanding is that CFM 2.1.4 will be based on NiFi 1.16 when it becomes available a little later this year. To find out more about that release schedule, reach out to your Cloudera Account Team. I haven't personally tried doing it, but the documentation for CFM 2.1.1, the currently-released version, indicates that you can install CFM on top of HDF Version 3.5.2, so when the new version comes out, you might not be out of luck if you want to run NiFi version 1.16.x on top of that version of HDF, if CFM maintains that support (I can't offer any assurances that it will, however, given that HDF has reached EoS already).
Now that brings us to Ambari. While Cloudera continues to support customers using Ambari in accordance with its associated products' support lifecycle, including HDF, Cloudera announced it was ending its involvement in the Ambari open source community in 2021. You are correct that at the ASF, Ambari was terminated in January of this year. That means that going forward, there are going to be fewer people using it, and fewer people who have the knowledge of how it works in order to effectively support it. This is one reason that as of this writing, Cloudera recommends upgrading to the latest version of CDP Private Cloud Base as soon as possible, as it has a richer set of features and is the successor to HDF and remains under standard Cloudera support through August 2024.
We welcome your questions, and this thread will remain visible here in the hope that some other member of the Cloudera Community will reply with assistance addressing your second paragraph above.