@Dhwanil Raval I was able to follow the github link I have posted above and setup environment variable in openshift and apply them in the script start.sh The only thing I had to do is convert the script to the Unix format in my text editor as openshift was having trouble running it when deploying the pod and starting nifi. If you are having a specific issue I could help with that but currently its all working as described in github. ... View more

Hello @Shu, I am wondering how can use the same technique but to replace order by clause with nothing. I would like to not use order by at all. Thanks. ... View more

@Harald Berghoff It appears that the start.sh script had to be changed to Unix format. I thought about this late in the day today and that fixed the issue. The NiFi pod is now up and running in openshift with environment variables set-up for ZK connect string. ... View more

@Matt Clarke Another note, I tried Redis as well. That did not work as well. I pretty much get the same error. I will open an Apache NiFi ticket. Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.RuntimeException: Cannot use Local State Provider (org.apache.nifi.redis.state.RedisStateProvider) as it only supports scope(s) {CLUSTER} but instance is configured to use scope LOCAL Thanks for helping troubleshoot this. I will explore my next steps now perhaps a VM of NiFi cluster. ... View more

@Matt Clarke This did not work either. First I couldn't use zk-provider in two places as id. It is already used in cluster-provider. I changed that as follows: <local-provider> <id>local-provider</id> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> <property name="Connect String">myhostname:2181</property> <property name="Root Node">/nifi</property> <property name="Session Timeout">10 seconds</property> <property name="Access Control">Open</property> </local-provider> <cluster-provider> <id>zk-provider</id> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> <property name="Connect String">myhostname:2181</property> <property name="Root Node">/nifi</property> <property name="Session Timeout">10 seconds</property> <property name="Access Control">Open</property> </cluster-provider> The error I get is as follow: java.lang.RuntimeException: Cannot use Local State Provider (org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider) as it only supports scope(s) {CLUSTER} but instance is configured to use scope LOCAL at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:175) <br> ... View more

@Matt Clarke Hello, just wanted to update that this did not work. I changed the state-provider for nifi.state.management.provider.local to zk-provider in nifi.properties file. Then I updated the state-management.xml file with the ZK properties. nifi.properties change is below: # The ID of the local state provider nifi.state.management.provider.local=zk-provider # The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster. nifi.state.management.provider.cluster=zk-provider <local-provider> <id>local-provider</id> <class>org.apache.nifi.controller.state.providers.local.WriteAheadLocalStateProvider</class> <property name="Directory">./state/local</property> <property name="Always Sync">false</property> <property name="Partitions">16</property> <property name="Checkpoint Interval">2 mins</property> </local-provider> <cluster-provider> <id>zk-provider</id> <class>org.apache.nifi.controller.state.providers.zookeeper.ZooKeeperStateProvider</class> <property name="Connect String">myhostname:myport</property> <property name="Root Node">/nifi</property> <property name="Session Timeout">10 seconds</property> <property name="Access Control">Open</property> </cluster-provider> I have the correct hostname and port for zk-provider above in state-management.xml, I just have removed the entries in the post. Question: Should be changing the root node? Here is the error I am getting: Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowService': FactoryBean threw exception on object creation; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'flowController': FactoryBean threw exception on object creation; nested exception is java.lang.IllegalStateException: Cannot create Local State Provider because the 'nifi.state.management.provider.local' property in the NiFi Properties file is set to 'zk-provider', but this id is assigned to a cluster-provider entry in the State Management Configuration File (/etc/nifi/state-management.xml), rather than a local-provider entry Caused by: java.lang.IllegalStateException: Cannot create Local State Provider because the 'nifi.state.management.provider.local' property in the NiFi Properties file is set to 'zk-provider', but this id is assigned to a cluster-provider entry in the State Management Configuration File (/etc/nifi/state-management.xml), rather than a local-provider entry at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.createStateProvider(StandardStateManagerProvider.java:157) at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.createLocalStateProvider(StandardStateManagerProvider.java:87) at org.apache.nifi.controller.state.manager.StandardStateManagerProvider.create(StandardStateManagerProvider.java:73) at org.apache.nifi.controller.FlowController.<init>(FlowController.java:507) at org.apache.nifi.controller.FlowController.createStandaloneInstance(FlowController.java:409) at org.apache.nifi.spring.FlowControllerFactoryBean.getObject(FlowControllerFactoryBean.java:77) at org.springframework.beans.factory.support.FactoryBeanRegistrySupport.doGetObjectFromFactoryBean(FactoryBeanRegistrySupport.java:168) ... 45 common frames omitted 2018-07-19 15:26:03,674 INFO [Thread-1] org.apache.nifi.NiFi Initiating shutdown of Jetty web server... 2018-07-19 15:26:03,821 INFO [Thread-1] o.eclipse.jetty.server.AbstractConnector Stopped ServerConnector@1ffe7c42{HTTP/1.1,[http/1.1]}{hdf-lab-0:8080} 2018-07-19 15:26:03,822 INFO [Thread-1] org.eclipse.jetty.server.session Stopped scavenging Received trapped signal, beginning shutdown... Is there other property settings I should use? I would like to make NiFi work with containers. Thanks. ... View more

Hello, I have a question if you can help answer that would be great. Can we run Nifi in standalone mode (not cluster) in an Open Shift Pod but manage state in zookeeper to ensure if the openshift pod reboots the states of the stateful processors are not lost? it appears to me that NiFi architecture does not fit containerization. Thoughts? ... View more

Thanks, I appreciate the quick response. I had to tweak the approach a bit, but this worked great! I used two routeonattribute processor as the not() was posing issues. First apply schema filter allows matches to flow through using below: ${anyDelineatedValue("${IncludeSchemaList}", ","):equalsIgnoreCase("${db.table.schema}")} Then I apply table filter and only allow unmatched to flow through, using below: ${anyDelineatedValue("${ExcludeTableList}", ","):equalsIgnoreCase("${db.table.name}")} ... View more