Member since
09-29-2015
871
Posts
723
Kudos Received
255
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 4275 | 12-03-2018 02:26 PM | |
| 3224 | 10-16-2018 01:37 PM | |
| 4336 | 10-03-2018 06:34 PM | |
| 3194 | 09-05-2018 07:44 PM | |
| 2442 | 09-05-2018 07:31 PM |
06-14-2017
05:03 PM
There should be two time fields, event time and event duration... Event time is the time at which the event was generated and usually the event is generated at the end of a processor executing, after it has successfully processed the flow file and is ready to report an event about what it happened. For two provenance events you could take the difference between the event times to see how long it took between those events, but it doesn't guarantee all that time was spent in a processor. Lets say processor A emits a flow which produces a CREATE event, and then processor B writes to it which produces a CONTENT_MODIFIED event. The flow file could have sat in the queue between these two processors for several minutes due to back-pressure or some reason, and was then processed by processor B in a second or two, but the time difference between those two events would be several minutes. Event duration is not guaranteed to be set and is dependent on the processor. Typically a processor will calculate the time it took to perform some operation, for example transferring the content of a flow file to an external system, and then report a provenance event with that duration in it, for example a SEND event.
... View more
06-14-2017
04:24 PM
What version of NiFi or HDF was this? I think this was fixed in https://issues.apache.org/jira/browse/NIFI-3912
... View more
06-14-2017
04:17 PM
I don't know the history behind the state support in UpdateAttribute, but I looked at the code earlier and it looks like it expects "Stateful Variables Initial Value" to just be a single value, which means it would be the same value for all variables. So if you literally enter 0 for the value of that property in the UI, it should initialize all variables to 0.
... View more
06-14-2017
01:41 PM
Have you tried setting "Stateful Variables Initial Value" to 0 for the example? What it should be doing is taking all the names of the dynamic properties (in this case "theCount") and creating an initial map of names to initial value, so you should end up with a map that has theCount -> 0 and then when the processor runs it would increment from 0 to 1 on the first execution.
... View more
06-14-2017
01:33 PM
What version of the Kafka broker are you running? If its installed through HDF, then it is probably Kafka 0.10.x and you should try using ConsumeKafka_0_10. In general it is best to use the processor that goes with the broker version, so.. GetKakfa - Kafka 0.8 ConsumeKafka - Kafka 0.9 ConsumeKafka_0_10 - Kafka 0.1.0
... View more
06-13-2017
01:35 PM
A message just means whatever you are sending to Kafka. For example, if you used Kafka's console producer from their documentation like: bin/kafka-console-producer.sh --broker-list localhost:9092 --topic streams-file-input < file-input.txt Each line of file-input.txt would be published to the topic, so each line here is what I am describing as a message. The RecordTooLargeException is indicating there was a call to publish a message, and the message exceeded to maximum size that the broker is configured for. In NiFi, the messages being published will be based off the content of the flow file... If you specify a "Message Demarcator" in the processor properties, then it will read the content of the flow file and separate it based on the demarcator, sending each separate piece of data as a message. If you don't specify a demarcator then it will take the entire content of the flow file and make a single publish call for the whole content. In the latest version of NiFi, there is also PublishKafka_0_10_Record which would read the incoming flow file using a configured record reader and send each record to Kafka as a message.
... View more
06-12-2017
06:42 PM
Correct, you would have to implement a ConfluentSchemaRegistry controller service for NiFi.
... View more
06-12-2017
05:49 PM
The TokenTooLargeException was coming from NiFi's code comparing against the "Max Request Size" property, which is why changing it to 2MB got past that error. The RecordTooLargeException is coming from the Kafka client code... I think the Kafka client is talking to broker and determining that the message is too large based config on the broker, but I'm not 100% sure. The first answer on this post might help: https://stackoverflow.com/questions/21020347/kafka-sending-a-15mb-message Also may want to make sure you can publish a 2MB message from outside of NiFi, which right now I would suspect you can't until changing some config on the broker.
... View more
06-12-2017
05:30 PM
2 Kudos
I can't speak to the differences between the two registries, but I don't believe they are completely API compatible so I don't think you can point NiFi's HortonworksSchemaRegistry service at a Confluent schema registry.
... View more
06-12-2017
05:28 PM
1 Kudo
What do you have the value of "Max Request Size" set to in PublishKafka_0_10? From the error message it looks like you still have it set to 1MB which is 1048576 bytes.
... View more