Am currently evaluating HDP using version 3.1.4 and it appears that this version is not compatible with any form of HDF either whole-platform or the mpack.
Looking at the compatibility matrix, this is what I see:
I also saw this github issue that makes it seem like the latest version of HDP can't even use the HDF mpack to use NiFi: https://github.com/hortonworks/ansible-hortonworks/issues/166
Can anyone confirm these issues? Is there any workaround? What is the expected use case for a version of HDP that does not integrate with the data intake HDF platform?
The work around would be to uncompress the management pack fileset in order to update with the required version of HDP & Ambari. Then recreate the compressed pack, remove old Mpack, add in your new one to ambari-server. Restart Ambari Server. When the version matches up as expected your HDF components should be available when you go to add a new service. This manual work may be easier than waiting for official update to the management pack.
The support matrix indicates what you can install together. If you don't want to get slowed down (or rushed forward) by dependencies, you can install HDF on a separate cluster. This is also what I see most users doing.
Note that you will absolutely be able to access the data on any recent version of HDP, and for this specific example I even expect the atlas integration to be smooth.
I made the same experience already end of August, however only when trying to deploy with Ambari 2.7.4
I created an issue in https://github.com/hortonworks/ansible-hortonworks/issues/166 (btw highly recommended way to use Ansible with those playbooks to deploy HDP, if you cannot use cloudbreak !)
as I initially thought it might be a bug in those ansible playbooks.
But it was clearly the problem that the provided HDF mpacks are incompatible with Ambari 2.7.4 (as can be seen from the Exception infos in the issue link above).
However the good news: I got a cluster deployed using latest HDP 3.1.4 and HDF 22.214.171.124, simply by using Ambari 2.7.3 (except I missed something, and I just thought that the newest versions were indeed used)