I've been troubleshooting for some time now, and really at this point I'm just looking for some suggestions from people who have more experience than I do.
I want to run NiFi 1.10.0 on OpenJDK 11, with NiFi Registry. Previously I was using Cloudera Flow Management 1.0.1, but ran into many issues with mixing Java 8 and Java 11, so I'm making a full switch to exclusively Java 11 now.
CFM 1.0.1 is incompatible with Java 11, so I need to find another option for managing NiFi. In your opinion, would it be easiest for me to manually add NiFi and NiFi Registry nodes to CDH 6.3.2? I understand that Cloudera Data Platform - Data Cloud is going to have support for NiFi 1.10.0, however I currently do not need much more than a couple of nodes... I'm building this for demo purposes and not at scale. Any recommendations would be greatly appreciated.
P.S. Does anyone know when Cloudera Flow Management will be updated to NiFi 1.10.0?
I understand that it is tempting to be on the latest version, but please consider the following:
Cloudera will soon have a release with 1.10. I cannot confirm that it will allow for Java 11 but I would expect this.
Therefore I strongly recommend you to stay on one of the released versions.
Note that this is not only because of the easy management, but more so because of improvements that the release version will bring over the 'raw' version. Think about improved integration, but also some functionality will have been added and more testing has been done.
In short: Regardless of how you deploy it, it is always recommended to use the latest GA version by Cloudera.
NiFi 1.10.0 will certainly work with Java 11. The issue I've had is that I repeatedly run into problems when using CFM 1.0.1 and having differing versions of Java across my cluster. I've had CDH java path configured to the best of my knowledge, and the NiFi node configured to run on OpenJDK 1.8 as the documentation suggests, but I have run into so many problems along the way.
I need to have a product ready to demo really soon, and I've hit so many bumps in the road trying to keep Cloudera Manager happy, in addition to many of my NiFi nodes failing to start up mostly due to java pathing issues (I think), broken user/group permissions, etc.
I wish I were confident enough to be able to use CFM NiFi on Java 8, but all our other machines are connected with Java 11 on CDH 6.3.2.
According to my research, CFM with NiFi 1.10.0 isn't even in the schedule yet, with NiFi 1.10.0 first scheduled to appear with CDP-DC pretty soon. I'd love to know if and when a new CFM will be released!
@DennisJaheruddi makes the most important point, you should not expect these newest releases to be "Production Ready". I can personal validate that 1.10 is not ready yet. You should work with a lower version release, and resolve your problems from conflicting or different java versions versus trying to get the newest releases which are not supported/available/recommended for production.
Tentatively, CFM 2.0 will be the next release, will be based off Apache NiFi 1.11, support Java 11, and be available via the new CDP platform. CDP will offer options like Ranger for authorization, not available within CDH.