Member since
05-29-2017
11
Posts
4
Kudos Received
0
Solutions
02-09-2018
06:14 AM
1 Kudo
I am facing a memory issue with a specific processor ConvertJSONToSQL Not sure if there is a way to configure processor specific memory, as when this error comes i found other processor and services running fine in nifi. Is there a way to address this apart from looking extending heapspace in general to whole nifi , currently its 8GB. Below is the stack trace of error: 2018-02-09 05:42:53,197 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.ConvertJSONToSQL ConvertJSONToSQL[id=6f1f8d3a-20b3-3fc5-884e-73f1e7443b58] ConvertJSONToSQL[id=6f1f8d3a-20b3-3fc5-884e-73f1e7443b58] failed to process session due to java.lang.OutOfMemoryError: Java heap space: {}
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:3236)
at java.io.ByteArrayOutputStream.grow(ByteArrayOutputStream.java:118)
at java.io.ByteArrayOutputStream.ensureCapacity(ByteArrayOutputStream.java:93)
at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:153)
at java.io.DataOutputStream.writeLong(DataOutputStream.java:224)
at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:117)
at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:69)
at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:56)
at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:47)
at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:147)
at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:96)
at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:56)
at org.apache.nifi.repository.schema.SchemaRecordWriter.writeRecordFields(SchemaRecordWriter.java:47)
at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldValue(SchemaRecordWriter.java:143)
at org.apache.nifi.repository.schema.SchemaRecordWriter.writeFieldRepetitionAndValue(SchemaRecordWriter.java:69) Thanks, Tarun Kumar
... View more
Labels:
- Labels:
-
Apache NiFi
12-27-2017
01:29 PM
Thank You very much @jwitt for useful insights with above statements.
... View more
12-26-2017
03:28 PM
1 Kudo
Thank You @jwitt . Please confirm is this suggestion as a recommended fix for stuck issue with HDFS-Processors in nifi in the context of above thread? Please also help with some relevant information/link in this regard to relate this issue.
... View more
12-21-2017
06:46 AM
Thank You @Jeff Storck for the reply. Please see below response inline. What values are set for the ticket lifetime and ticket renewal lifetime in the KDC for the principal you have set in PutHDFS?
TK: Not sure of KDC setting, Where shall I check these? What values are set for the ticket lifetime and ticket renewal liftetime in the krb5.conf that you have set for NiFi in Ambari?
TK: Ticket_Lifetime: 24h , renew_lifetime:7d Does this issue occur consistently? Would it seem to happen around the time that the principal's kerberos ticket would be getting renewed, some time between 80% and 100% of the ticket lifetime?
TK: This issue is coming intermittently and can’t relate any pattern. How often are files sent to PutHDFS incoming queue? On a regular interval, or is it sporadic?
TK: Don’t think we have a regular interval for files to be sent . What is the "Relogin Period" property set to in PutHDFS' configuration?
TK: Default 4 hours Unfortunately don't have nifi-app and bootstarp.logs for QA env issue when it occurred last time.But we also notice this on Dev env with FetchHDFS Processor, attached are thread-dumps and logs for that time.dev-fetchhdfs-stuck.jpgthread-dump-1.txtthread-dump-2.txt
... View more
12-21-2017
06:08 AM
Thank You @Milan Das. So the workaround you suggested require changes to nifi code ? For now I had added a DetectDuplicate processor.
... View more
12-18-2017
02:25 PM
1 Kudo
We have been facing an intermittent issue in our QA env, where PutHDFS processor goes stuck and not able to release item from upstream processor queue. This does not look related to load as the no.of messages in queue and queue size was not too high. We tried to stop/start the processor but this does not work as after stopping the processor we don't see the start button on PutHDFS. The only way we are currently fixing this is by restarting the nifi which we would like to avoid in Prod env. At the time of stuck No of threads we can see on UI 2 and processor is configured with default 1 parallel level. In order to further drill down we tried to take multiple thread-dumps and notices that a particular thread was always blocked with same stack trace and linked to PutHDFS processor. Stack trace is as below. Timer-Driven Process Thread-2" Id=106 BLOCKED on java.io.BufferedInputStream@b7146b3
at java.io.BufferedInputStream.read(BufferedInputStream.java:336)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326) ------- at org.apache.nifi.processors.hadoop.PutHDFS$1.run(PutHDFS.java:255)
at java.security.AccessController.doPrivileged(Native Method) puthdfs-stuck.jpgthread-dump.txtthread-dump-2.txtthread-dump-3.txt I am attaching the UI screen-shot at time of stuck and thread-dumps for reference. Can someone please help us in finding the root-cause for this ?
... View more
Labels:
- Labels:
-
Apache NiFi
12-18-2017
02:00 PM
Thank You Matt, I too was facing similar issue and your suggestion worked.
... View more
12-14-2017
08:59 AM
1 Kudo
Hi Team, we noticed ListS3 processor is processing same file twice or more, while there was no change in the source file. As we go though the "Component State" which clearly says, nifi manages the file states with latest time-stamp and list only when a new file has been added or existing file has been modified. We are seeing ListS3 processor is reprocessing same file from Data Provenance which has the same s3.lastModifed property value, We believe file should not be re-processed unless its changed per "State Management" docs. Kindly help us on this issue. Let us know for any information needed. Note: ListS3 Processor has been setup to run only primary node.
... View more
Labels:
- Labels:
-
Apache NiFi
05-29-2017
01:33 PM
We are trying to setup Kafka with kerberos and facing some issue in running defaultproducer/consumer example. Taken help from : https://community.hortonworks.com/articles/55154/kafka-ssl-kerberos-cheat-sheet-settingsconsole-com.html [tarun.kumar@aw1frtdvhdp02 bin]$ /usr/hdp/current/kafka-broker/bin/kafka-console-producer.sh --broker-list localhost:6667 --topic test Hello
[2017-05-29 14:11:56,293] ERROR Error when sending message to topic test with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) While trying to run the producer/consumer above ways seeing above error. Can anyone please help me in understanding /resolving this ?
... View more
Labels:
- Labels:
-
Apache Kafka