I've been troubleshooting for some time now, and really at this point I'm just looking for some suggestions from people who have more experience than I do. I want to run NiFi 1.10.0 on OpenJDK 11, with NiFi Registry. Previously I was using Cloudera Flow Management 1.0.1, but ran into many issues with mixing Java 8 and Java 11, so I'm making a full switch to exclusively Java 11 now. CFM 1.0.1 is incompatible with Java 11, so I need to find another option for managing NiFi. In your opinion, would it be easiest for me to manually add NiFi and NiFi Registry nodes to CDH 6.3.2? I understand that Cloudera Data Platform - Data Cloud is going to have support for NiFi 1.10.0, however I currently do not need much more than a couple of nodes... I'm building this for demo purposes and not at scale. Any recommendations would be greatly appreciated. P.S. Does anyone know when Cloudera Flow Management will be updated to NiFi 1.10.0? ... View more

Question about versioning... We are no longer using Java 8, but the current CFM 1.0.1 has a version of NiFi that only works with Java 8. I'm going to install a NiFi 1.10.0 node on Java 11 without CFM 1.0.1 until CFM gets an update. My question is, will NiFi Registry on CFM 1.0.1 be compatible with Java 11? Thank you! ... View more

If I recall correctly, there are quite a few charts on the homepage of Cloudera Manager upon initial install. I've since removed the admin account and replaced it with my own, and lost most of the charts in the process. Where can I find some neat charts to plug in without building them myself? Thanks! ... View more

Hi. I just stopped all services and entered maintenance mode on all managed hosts to perform a restart on all my VMs. Upon restart, Cloudera Manager doesn't want to be recommissioned (after starting cloudera manager service). What's the command to recommission the CM host? Thanks! ... View more

I've heard this is a "bug" with openssl/keytool. I'm following MattWho's article found here: How to create user generated keys for securing NiFi. I'm getting the following error on my NiFi WebUI:    Hostname nifi.taco.net not verified: certificate: sha256/5REuJXk5ayT2nW5J89AfpW/G3OzXY9lF4n2vE3OxHlE= DN: CN=nifi.taco.net, OU=project taco, O=taco, L=taco, ST=texas, C=US subjectAltNames: [] I'm guessing this is either because of the SAN info being removed when I use x509, or perhaps a misconfiguration in the Cloudera Flow Management NiFi Node config?? ... View more

At long last, I am one step away from having this thing working. My question: is lack of SubjectAlternativeName in the cert keeping this from working, or is it something else? The hostname is correct and is working to direct to the correct IP. Is having the IP address of the host in the SAN going to fix this issue? I'm authenticating via LDAP. LDAP doesn't need access to port 8443 does it? I had this working before with NiFi Toolkit CA standalone, and LDAP was working with that. So I'm pretty sure it's an issue with the SubjectAlternativeNames not being present. Thank you Failed to replicate request GET /nifi-api/flow/current-user to nifi.taco.net:8443 due to javax.net.ssl.SSLPeerUnverifiedException: Hostname nifi.taco.net not verified: certificate: sha256/ulAHv1CZSCcrJL/d9rXRCVGFNtOqvKFXH2bB4Viz6/0= DN: CN=nifi.taco.net, OU=Project taco, O=taco, L=taco, ST=taco, C=US subjectAltNames: [] javax.net.ssl.SSLPeerUnverifiedException: Hostname nifi.taco.net not verified: certificate: sha256/ulAHv1CZSCcrJL/d9rXRCVGFNtOqvKFXH2bB4Viz6/0= DN: CN=nifi.taco.net, OU=Project taco, O=taco, L=taco, ST=taco C=US subjectAltNames: [] at okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:316) at okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270) at okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162) at okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257) at okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135) at okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114) at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:126) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147) at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121) at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:200) at okhttp3.RealCall.execute(RealCall.java:77) at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:138) at org.apache.nifi.cluster.coordination.http.replication.okhttp.OkHttpReplicationClient.replicate(OkHttpReplicationClient.java:132) at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator.replicateRequest(ThreadPoolRequestReplicator.java:647) at org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator$NodeHttpRequest.run(ThreadPoolRequestReplicator.java:839) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)   ... View more

I'm needing to clean a NiFi Node to have it stop getting errors. I've narrowed down the problem quite a bit (I'm trying to have it Authenticate via LDAP, with a self-signed SSL cert, both of these things are set up properly now). However, because I've been through so much troubleshooting and trial and error with this NiFi server, I have experienced WAY more hassle than I would've expected. I've been trying to get this to work for nearly a month now. SSL certs, truststore, keystore, node config (In CM) are all set up to the best of my ability. I stop the service roles in CM, put host in maintenance mode, restart the VM nifi is running on (to clear extra scm-agent processes). I've moved the flowfile.xml.gz, deleted users.xml, authorizations.xml, cleared the _repository folders out (moving them to a backup folder), cleared the archive folder, state/local folder. I've set initial admin to our proper LDAP identity. (I never get to this stage before the node shuts off now though) Jetty is reporting that the there is no valid keystore, but I am not sure that this is the cause of the effect of a different problem. I've been very careful to create the keystores exactly to specification following @MattWho 's article and have verified everything, also I had HTTPS working last night (csr worked, but I could not manage to log in due to "unverified keystore" on the client side I believe). Having as little experience with this sort of thing as I do, I have been so challenged and puzzled to get HTTPS set up for LDAP auth. I keep feeling like I'm only one or two steps away from having things working but then another problem springs up. Here is a piece of the log file from which I've scoured and this is the FIRST sign of something not going correctly: 7:04:23.327 PM INFO _nifi No Spring WebApplicationInitializer types detected on classpath 7:04:23.404 PM INFO ContextHandler Started o.e.j.w.WebAppContext@5b16e486{nifi,/nifi,file:///var/lib/nifi/work/jetty/nifi-web-ui-1.9.0.1.0.1.0-12.war/webapp/,AVAILABLE}{/var/lib/nifi/work/nar/framework/nifi-framework-nar-1.9.0.1.0.1.0-12.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-ui-1.9.0.1.0.1.0-12.war} 7:04:23.719 PM INFO AnnotationConfiguration Scanning elapsed time=181ms 7:04:23.748 PM INFO _nifi_api No Spring WebApplicationInitializer types detected on classpath Followed by this Error, which is followed by a MASSIVE list of missing beans which i have not included. Context initialization failed org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.springframework.security.config.annotation.web.configuration.WebSecurityConfiguration': Unsatisfied dependency expressed through method 'setFilterChainProxySecurityConfigurer' parameter 1; nested exception is org.springframework.beans.factory.BeanExpressionException: Expression parsing failed; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'org.apache.nifi.web.NiFiWebApiSecurityConfiguration': Unsatisfied dependency expressed through method 'setJwtAuthenticationProvider' parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtAuthenticationProvider' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'jwtService' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'jwtService' defined in class path resource [nifi-web-security-context.xml]: Cannot resolve reference to bean 'keyService' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'keyService' defined in class path resource [nifi-administration-context.xml]: Cannot resolve reference to bean 'keyTransactionBuilder' while setting bean property 'transactionBuilder'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'keyTransactionBuilder' defined in class path resource [nifi-administration-context.xml]: Cannot resolve reference to bean 'keyDataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'keyDataSource': FactoryBean threw exception on object creation; nested exception is org.h2.jdbc.JdbcSQLException: Error opening database: "Could not save properties /var/lib/nifi/database_repository/nifi-user-keys.lock.db" [8000-176]  Warn: Error opening database: "Could not save properties /var/lib/nifi/database_repository/nifi-user-keys.lock.db" [8000-176]   Failed startup of context o.e.j.w.WebAppContext@2b85edc7{nifi-api,/nifi-api,file:///var/lib/nifi/work/jetty/nifi-web-api-1.9.0.1.0.1.0-12.war/webapp/,UNAVAILABLE}{/var/lib/nifi/work/nar/framework/nifi-framework-nar-1.9.0.1.0.1.0-12.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-api-1.9.0.1.0.1.0-12.war}  And the final error before shutting down: 7:04:33.701 PM INFO _ No Spring WebApplicationInitializer types detected on classpath 7:04:33.755 PM INFO ContextHandler Started o.e.j.w.WebAppContext@490704a5{nifi-error,/,file:///var/lib/nifi/work/jetty/nifi-web-error-1.9.0.1.0.1.0-12.war/webapp/,AVAILABLE}{/var/lib/nifi/work/nar/framework/nifi-framework-nar-1.9.0.1.0.1.0-12.nar-unpacked/NAR-INF/bundled-dependencies/nifi-web-error-1.9.0.1.0.1.0-12.war} 7:04:33.787 PM WARN JettyServer Failed to start web server... shutting down. java.lang.IllegalStateException: no valid keystore at org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:50) at org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1071) at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:262) at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:229) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:72) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:279) at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.server.Server.doStart(Server.java:398) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:935) at org.apache.nifi.NiFi.<init>(NiFi.java:158) at org.apache.nifi.NiFi.<init>(NiFi.java:72) at org.apache.nifi.NiFi.main(NiFi.java:297) 7:04:33.797 PM INFO NiFi Initiating shutdown of Jetty web server... So. With this, it is failing to even reach the stage where it will generate the users.xml and authorizations.xml file that I'm used to it auto-generating after I remove it. I've had to change the initial admin a couple of times, so I learned how to do that without reinstalling. I've removed the flowfile.xml.gz, which I suspect may be an issue here. As well as the _repository folders, which I was advised to do via other forum posts here, but I may have cleared more than I should have? I have backups saved. Really my big question is: what all CAN i remove/clean without NiFi not being able to recover / auto-gen new files on start. Any ideas would be super appreciated! ... View more

CFM 1.0.1 Could someone please explain to me why, after reconfiguring to disable SSL, when I try to access the webUI it still is attempting to connect on port 8443? Is this a zookeeper issue? How can I fix this? I had another cluster previously that start out as SSL disabled, but after configuring it to have a cert via nifi toolkit CA, and getting it WORKING with LDAP auth, it would try to request things from port 8080 http, then after vote resolved it will kill the NiFi node. So here I am now. Was unable to get SSL verified while following the 5 year old guide courtesy of @MattWho (you are awesome, i really appreciate all the info you've shared) I followed that guide to a T but the server reporting the cert was not verified. Now I'm trying to just connect to the nifi node via HTTP, so i disabled all related SSL things in the config, yet:   Cannot replicate request to Node nifi.domain.net:8443 because the node is not connected It still tries to connect to port 8443. Can someone tell me why it's trying to connect to the old port? Is this a zookeeper issue? If i could resolve this then I feel i could resolve all my other issues. Thanks! ... View more

Hello. I'm curious what the repercussions are if Cloudera Manager is failing to find it's version number. It should be running 6.3.0 right now, as it was previous to the install. However, something isn't correct. Long story, but I unintentionally broke my cluster by accidentally overriding the cluster to Java 8 when I should have not created a master override (All are running Java 11 and some services of dedicated Java 8 installed). After fixing this issue, CM fails to report version. I have a deadline to reach right now and am going to continue working hoping that this will not be a big issue due to something unforeseen. Everything seems to be running fine other than supervisord and failing to report CDH version number. ... View more

Hello! I've got a misconfigured NiFi service that has been failing after a couple of minutes of being online due to something I haven't been able to diagnose, and I think the best option at this point is to just uninstall and reinstall. My question is, what is the best practice for doing so, so that it will not leave remnants of past configuration? I want to add a new NiFi service, fresh out of the box, to my cluster, and remove everything that was associated with the old one. If you could share the best practices for doing such with me, that would be greatly beneficial! ... View more

I'm in the process of adding CFM to my cluster, and I believe I have everything configured properly, except I am using Java 11 on every host in the cluster EXCEPT for my nifi host. Following the documentation I was instructed to set  Java Home Path Override to "the location of the JDK you installed at the beginning of the installation process". I used `rpm -ql java-1.8.0-openjdk` to find the directory and added `/usr/lib/jvm/` to the Java Home Path Override configuration in the Nifi Node. Was this incorrect? What is the location of the JDK that I installed? Using `java -version` I can confirm that I am using java 8. ... View more

I am trying to create a demo environment for a product on a relatively small lab, and the more I learn about all of the capabilities that Cloudera is making possible for people like myself to use (who do not have extensive background in development. Cloudera is amazing!) I was thinking that I would need to do much of this installation by hand (adding new hosts, logging in and editing config files by hand, etc) but now I am seeing that it is possible to make it VERY simple to perform these tasks within Cloudera Manager. The more services I begin to add (I want to use them all!) the more strain it becomes on our little demo lab. I'm seeking advice on bootstrapping the tools we are requiring for our specific product purposes, while cutting out everything unnecessary. So far I would like to use: HDFS YARN Kafka (3 broker, 1 zookeeper) CFM (Specifically NiFi, I don't understand the difference between CFM and CDF just yet, please explain?) PostgreSQL DB Manager Services (Host Monitor, etc) Our lab's specifications are:  The product demo I'm working on will not be hugely demanding, and I believe that Cloudera Services will be the biggest demand out of everything we are implementing. I'm wondering if we even have enough resources for me to continue down this path of "Install CDH, CFM, HDFS service, YARN service, etc...". Is there a way I can optimize these services to be EXTREMELY lightweight since we will be performing such a small test? Or would my better option be to just do it the old fashioned way and install everything by hand without Cloudera functionality? I REALLY want to learn to use this platform, but if I don't cut corners and reduce resource allocation far below what is recommended, it seems there is no way to run all of the services I would like to have running. Where can I cut corners, in your opinion? How should I configure Kafka brokers and ZooKeeper to use the least amount of resources, or anything else? We are running from vCenter, and I'm not 100% sure how to "spread resources" or split services between hosts. Any advice, or documentation you can link me to offering assistance would be greatly helpful. Aloha! ... View more