Hi, In my NiFi flow, i am receiving message in form of message segments but this segments is not received in the same time, some times the message segments related to each other take 2-4 hours to enter the NiFi Flow, i need to hold/store this segments in some place (Kafka or in NiFi Internal Queue or any other solution) and run query from time to time on them to check if the all segments received using the segment id (group by segment id) which is same for all segments related to the original message and aggregate them to construct the original message using NiFi. What is the best solution scale able solution to acchive this. ... View more

Hi, I am using RouteONAttribute Processor and i defined a property and its value contains multiple AND conditions like the following: Property Name: ROUTE1 Property Value: ${text.tag.type:equals('XXX'):and(${txt:contains('AAA')}):and(${txt:contains('BBB')})} The Issue i am facing is that after generating a flow file with attributes text.tag.type = XXX and txt = AAA,BBB it always route to unmatched while it should route to ROUTE1 Thanks ... ... View more

Hi, I moved some of the NiFi flow to a Process Group, the input of the process group is 3 relations which goes to the input port of the process group, how can i get this 3 relations separatly inside my process group. Thanks,. ... View more

Hi, I need to apply regular expression on file file attribute not on flow file content using ExtractText processor. How can i achive this. Thanks,, ... View more

Hi, I have simple SAM application with one KAFKA Source and KAFKA Sink. after running the App, i got this errors in the streamline.log file: INFO [2018-09-11 14:46:53.085] [dw-41 - GET /api/v1/schemaregistry/schemas/versionsById/3462796923090514477] c.h.r.s.w.SchemaRegistryResource - No schema version is found with schema version id : [3462796923090514477] Question : From where SAM get 3462796923090514477 as version ID for the schema, the schema version is 11 not 3462796923090514477. Error In Storm UI: Spouts (All time) Id Executors Tasks Emitted Transferred Complete latency (ms) Acked Failed Error Host Error Port Last error Error Time 1-KAFKA 1 1 0 0 0.000 0 0 sandbox-hdf.hortonworks.com 6700 com.hortonworks.registries.schemaregistry.exceptions.RegistryException: java.lang.RuntimeException: com.google.common.util.concurrent.UncheckedExecutionException: javax.ws.rs.NotFoundException: HTTP 4 Tue, 11 Sep 2018 20:24:27 Why SAM use Version ID to Query the Schema from Schema registry. from where it get the Version ID. how can i configure it to use Schema name in the query instead of Version ID. Thanks .. ... View more

Hi, I used SAM to build streaming application. the source is KAFKA Topic and the destination (Sink) is another KAFKA Topic and in between i used Rule to filter specific events, also I created Avro schema's using schema manager (two schema one for input stream and one for output stream) and i name them as the name of the KAFKA Topics. them i puplished Avro messages to the KAFKA source Topic. After Running the application no message inserted in Sink KAFKA topic and CPU is consumed a lot and error shown in the streaming application box as show in the pictures and i don't know how to check this errors (nothing shown in the streamline.log). in Strom UI, this error shown: Id Executors Tasks Emitted Transferred Complete latency (ms) Acked Failed Error Host Error Port Last error Error Time 3-KAFKA 1 1 0 0 0.000 0 0 sandbox-hdf.hortonworks.com 6700 com.hortonworks.registries.schemaregistry.exceptions.RegistryException: java.lang.RuntimeException: com.google.common.util.concurrent.UncheckedExecutionException: javax.ws.rs.NotFoundException: HTTP 4 Mon, 10 Sep 2018 19:54:18 Errors Search: Time Error Host Error Port Error 2018-09-10T16:54:28.000Z sandbox-hdf.hortonworks.com 6700 com.hortonworks.registries.schemaregistry.exceptions.RegistryException: java.lang.RuntimeException: com.google.common.util.concurrent.UncheckedExecutionException: javax.ws.rs.NotFoundException: HTTP 404 Not Found at com.hortonworks.registries.schemaregistry.serde.AbstractSnapshotDeserializer.deserialize(AbstractSnapshotDeserializer.java:156) at com.hortonworks.streamline.streams.runtime.storm.spout.AvroKafkaSpoutTranslator.apply(AvroKafkaSpoutTranslator.java:61) at org.apache.storm.kafka.spout.KafkaSpout.emitOrRetryTuple(KafkaSpout.java:506) at org.apache.storm.kafka.spout.KafkaSpout.emitIfWaitingNotEmitted(KafkaSpout.java:474) at org.apache.storm.kafka.spout.KafkaSpout.nextTuple(KafkaSpout.java:341) at org.apache.storm.daemon.executor$fn__9551$fn__9566$fn__9599.invoke(executor.clj:660) at org.apache.storm.util$async_loop$fn__555.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.RuntimeException: com.google.common.util.concurrent.UncheckedExecutionException: javax.ws.rs.NotFoundException: HTTP 404 Not Found at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.getSchemaVersionInfo(SchemaRegistryClient.java:616) at com.hortonworks.registries.schemaregistry.serde.AbstractSnapshotDeserializer.deserialize(AbstractSnapshotDeserializer.java:153) ... 8 more Caused by: com.google.common.util.concurrent.UncheckedExecutionException: javax.ws.rs.NotFoundException: HTTP 404 Not Found at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2203) at com.google.common.cache.LocalCache.get(LocalCache.java:3937) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941) at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824) at com.hortonworks.registries.schemaregistry.cache.SchemaVersionInfoCache.getSchema(SchemaVersionInfoCache.java:103) at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.getSchemaVersionInfo(SchemaRegistryClient.java:612) ... 9 more Caused by: javax.ws.rs.NotFoundException: HTTP 404 Not Found at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:1008) at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:816) at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:92) at org.glassfish.jersey.client.JerseyInvocation$2.call(JerseyInvocation.java:700) at org.glassfish.jersey.internal.Errors.process(Errors.java:315) at org.glassfish.jersey.internal.Errors.process(Errors.java:297) at org.glassfish.jersey.internal.Errors.process(Errors.java:228) at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:444) at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:696) at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:420) at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:316) at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient$16.run(SchemaRegistryClient.java:1083) at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient$16.run(SchemaRegistryClient.java:1080) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:360) at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.getEntity(SchemaRegistryClient.java:1080) at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.doGetSchemaVersionInfo(SchemaRegistryClient.java:639) at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient.access$100(SchemaRegistryClient.java:141) at com.hortonworks.registries.schemaregistry.client.SchemaRegistryClient$1.retrieveSchemaVersion(SchemaRegistryClient.java:243) at com.hortonworks.registries.schemaregistry.cache.SchemaVersionInfoCache$1.load(SchemaVersionInfoCache.java:71) at com.hortonworks.registries.schemaregistry.cache.SchemaVersionInfoCache$1.load(SchemaVersionInfoCache.java:63) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ... 14 more Thanks.. ... View more

Hi, We need to execute KSQL querier to filter our KAFKA streams within NiFi. How can we achive this . Should we implement custom processor. Regards, ... View more

Hi, We need to execute KSQL querier to filter our KAFKA streams within NiFi. How can we achive this . Should we implement custom processor. Regards, ... View more

Hi, Need to parse binary ASN.1 BER .Please suggest to do in Apache NIFI. Regards, ... View more

Hi, I used EncryptContent NiFi Processor to decrypt data. We need to use AES-ECB-NoPadding algo but i cannot find it in the Encryption Algorithm in the properties of the processor. Should we create custom processor for this purpose. Thanks, ... View more

Hi, I used EncryptContent NiFi Processor to decrypt data. We need to use AES-ECB-NoPadding algo but i cannot find it in the Encryption Algorithm in the properties of the processor. Should we create custom processor for this purpose. Thanks,, ... View more

Hi, I used EncryptContent NiFi Processor to decrypt data. We need to use AES-ECB-NoPadding algo but i cannot find it in the Encryption Algorithm in the properties of the processor. Should we create custom processor for this purpose. Thanks,, ... View more

Hi, After reading some articles, I get confused, what is the defferance between Zookeper and NCM. what should we use for creating NiFi Cluster. Thanks.,, ... View more

Hi, We have three VMs and we need to deploy NiFi cluster on them using NiFi docker image (1 VM as Cluster Manager and 2 VMs as NiFi Nodes). How can we achieve this. Thanks, ... View more

Hi, We need to build HDF Cluster for our integration environment using 3 VMs. Should we build HDF cluster from scratch using AMBARI or we can use HDF docker image to create the HDF cluster on this 3 VMs. What is the difference between using HDF docker image and installing HDF Cluster from scratch. In our case we have 3 VMs, which VM will be the master node and slave nodes and data nodes. Thanks. ... View more

Hi, We need to build HDF Cluster for our integration environment using 3 VMs. Should we build HDF cluster from scratch using AMBARI or we can use HDF docker image to create the HDF cluster on this 3 VMs. What is the difference between using HDF docker image and installing HDF Cluster from scratch. In our case we have 3 VMs, which VM will be the master node and slave nodes and data nodes. Thanks. ... View more

Hi, We prepared 3 VM 16 GB RAM each (Ubuntu 16 - 64bit) and we need to deploy HDF 3.1.1 Docker image to them to test HDF cluster. Questions: 1. Should we install Docker on all the three VMs then deploy HDF 3.1.1 Docker image to all the 3 VMs then after that we create NiFI cluster through AMBARI. 2. Do we need to make one VM as Master HDF node and the other two VMs slave Nodes for the HDF cluster. 3. Can we install MySQL Cluster on 2 VMs out of those 3 VMs to store our flows data. 4. For production deployment, can we relay on HDF docker image or we need to deploy HDF from scratch by installing HDF packages. Thanks. Regards, ... View more

Hi, We are planning to deploy HDF 3.1.0 sandbox to two VMs to have clustered NiFi and we have the following questions: 1. Does the HDF sandbox contains MySQL as part of the sandbox. 2. Can we have our data node (MySQL DB) inside the smae VM which has the HDF. 2. Does HDF docker image can be deployed to Clustered Linux VMs. 3. If we deployed two HDF to two VMs and we manage to create NiFI cluster, can we have MySQL cluster in this case. 4. What is the recommended VM Specs in term of RAM, CPU. Thanks. Regards, ... View more