I am facing a memory issue with a specific processor ConvertJSONToSQL Not sure if there is a way to configure processor specific memory, as when this error comes i found other processor and services running fine in nifi. Is there a way to address this apart from looking extending heapspace in general to whole nifi , currently its 8GB. Below is the stack trace of error: 2018-02-09 05:42:53,197 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.ConvertJSONToSQL ConvertJSONToSQL[id=6f1f8d3a-20b3-3fc5-884e-73f1e7443b58] ConvertJSONToSQL[id=6f1f8d3a-20b3-3fc5-884e-73f1e7443b58] failed to process session due to java.lang.OutOfMemoryError: Java heap space: {} java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Arrays.java:3236) at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118) at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93) at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153) at java.io.DataOutputStream.writeLong(DataOutputStream.java:224) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:117) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:69) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:56) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:47) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:147) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:96) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:56) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:47) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:143) at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:69) Thanks, Tarun Kumar ... View more

We have been facing an intermittent issue in our QA env, where PutHDFS processor goes stuck and not able to release item from upstream processor queue. This does not look related to load as the no.of messages in queue and queue size was not too high. We tried to stop/start the processor but this does not work as after stopping the processor we don't see the start button on PutHDFS. The only way we are currently fixing this is by restarting the nifi which we would like to avoid in Prod env. At the time of stuck No of threads we can see on UI 2 and processor is configured with default 1 parallel level. In order to further drill down we tried to take multiple thread-dumps and notices that a particular thread was always blocked with same stack trace and linked to PutHDFS processor. Stack trace is as below. Timer-Driven Process Thread-2" Id=106 BLOCKED on java.io.BufferedInputStream@b7146b3 at java.io.BufferedInputStream.read(BufferedInputStream.java:336) at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284) at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ------- at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:255) at java.security.AccessController.doPrivileged(Native Method) puthdfs-stuck.jpgthread-dump.txtthread-dump-2.txtthread-dump-3.txt I am attaching the UI screen-shot at time of stuck and thread-dumps for reference. Can someone please help us in finding the root-cause for this ? ... View more

Hi Team, we noticed ListS3 processor is processing same file twice or more, while there was no change in the source file. As we go though the "Component State" which clearly says, nifi manages the file states with latest time-stamp and list only when a new file has been added or existing file has been modified. We are seeing ListS3 processor is reprocessing same file from Data Provenance which has the same s3.lastModifed property value, We believe file should not be re-processed unless its changed per "State Management" docs. Kindly help us on this issue. Let us know for any information needed. Note: ListS3 Processor has been setup to run only primary node. ... View more

We are trying to setup Kafka with kerberos and facing some issue in running defaultproducer/consumer example. Taken help from : https://community.hortonworks.com/articles/55154/kafka-ssl-kerberos-cheat-sheet-settingsconsole-com.html [tarun.kumar@aw1frtdvhdp02 bin]$ /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:6667 --topic test Hello [2017-05-29 14:11:56,293] ERROR Error when sending message to topic test with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) While trying to run the producer/consumer above ways seeing above error. Can anyone please help me in understanding /resolving this ? ... View more