Member since
11-08-2022
8
Posts
0
Kudos Received
0
Solutions
03-01-2023
02:54 AM
Can anyone please guide us on the above issue? We are still getting that error whenever we run a nifi process that generates too many flowfiles.
... View more
01-31-2023
05:55 AM
@MattWho Thank you for your valuable suggestion!! 1: We had separate directory for flowfile and provenance repositories earlier as well: Path for flowfile : "/flowfile_rep/nifi/nifi-1.16.2/flowfile_repository" Path for Provenance: "/flowfile_rep/nifi/nifi-1.16.2/provenance_repository" As per your suggestion we have changed the path as below and restarted the nifi: New Path for Flow file: "/flowfile_repo/nifi/nifi-1.16.2_flowfileRepository/flowfile_clean_repository" New Path for Provinance: "/flowfile_repo/nifi-1.16.2_provenanceRepository" but it seems the issue still persist... As we are already using a new directory path so we don't have to delete the old flowfile and provenance repo files right ? Please suggest if we have to implement any other solution to resolve this issue.. Thank you!!
... View more
01-30-2023
12:59 AM
Hello @MattWho Can you please guide me on this...Do I need to change the configuration of provenance in nifi.properties from 1GB to 10 GB ?
... View more
01-04-2023
05:01 AM
Greetings Cloudera Community!!! We have recently upgraded from Nifi 1.9.2 version to Nifi version 1.16.2 and we are getting the "Failed to index Provenance Events" Error, not all the time but it occurs occasionally. We have not faced such kind of error in nifi version 1.9.2. Can anyone please help us to figure out the possible cause of the occurrence of that error. Though it was not impacting the flow but still its occurrence is random. Please find the error below in nifi-app.log. Logs: 2023-01-04 06:21:23,713 ERROR [Index Provenance Events-1] o.a.n.p.index.lucene.EventIndexTask Failed to index Provenance Events org.apache.lucene.store.AlreadyClosedException: this IndexWriter is closed at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:877) at org.apache.lucene.index.IndexWriter.ensureOpen(IndexWriter.java:891) at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1468) at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1444) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.index(LuceneEventIndexWriter.java:70) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:202) at org.apache.nifi.provenance.index.lucene.EventIndexTask.run(EventIndexTask.java:113) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: org.apache.lucene.store.AlreadyClosedException: Underlying file changed by an external force at 2022-12-14T11:18:50Z, (lock=NativeFSLock(path=/flowfile_repo/nifi/nifi-1.16.2/provenance_repository/lucene-8-index-1671016730507/write.lock,impl=sun.nio.ch.FileLockImpl[0:9223372036854775807 exclusive valid],creationTime=2022-12-14T11:18:50.521826Z)) at org.apache.lucene.store.NativeFSLockFactory$NativeFSLock.ensureValid(NativeFSLockFactory.java:191) at org.apache.lucene.store.LockValidatingDirectoryWrapper.createOutput(LockValidatingDirectoryWrapper.java:43) at org.apache.lucene.store.TrackingDirectoryWrapper.createOutput(TrackingDirectoryWrapper.java:43) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesConsumer.<init>(Lucene80DocValuesConsumer.java:79) at org.apache.lucene.codecs.lucene80.Lucene80DocValuesFormat.fieldsConsumer(Lucene80DocValuesFormat.java:161) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:227) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.getInstance(PerFieldDocValuesFormat.java:163) at org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat$FieldsWriter.addNumericField(PerFieldDocValuesFormat.java:109) at org.apache.lucene.index.NumericDocValuesWriter.flush(NumericDocValuesWriter.java:108) at org.apache.lucene.index.DefaultIndexingChain.writeDocValues(DefaultIndexingChain.java:345) at org.apache.lucene.index.DefaultIndexingChain.flush(DefaultIndexingChain.java:225) at org.apache.lucene.index.DocumentsWriterPerThread.flush(DocumentsWriterPerThread.java:350) at org.apache.lucene.index.DocumentsWriter.doFlush(DocumentsWriter.java:476) at org.apache.lucene.index.DocumentsWriter.flushAllThreads(DocumentsWriter.java:656) at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:3365) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3771) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3729) at org.apache.nifi.provenance.lucene.LuceneEventIndexWriter.commit(LuceneEventIndexWriter.java:101) at org.apache.nifi.provenance.index.lucene.EventIndexTask.commit(EventIndexTask.java:253) at org.apache.nifi.provenance.index.lucene.EventIndexTask.index(EventIndexTask.java:232) ... 6 common frames omitted
... View more
Labels:
- Labels:
-
Apache NiFi
12-01-2022
02:19 AM
@ask_bill_brooks Thanks for the updates:) So, apart from upgrading from nifi 1.16.2 to newer nifi version, Can you please suggests any other fix that we can implement in our production environment to mitigate Log4shell vulnerability. But before that we just need to confirm: Do we really need to upgrade our current nifi version 1.16.2 to newer version ? As it is mentioned in NIFI-10648 , under worklog that apache nifi does not include any direct references to the vulnerability instance. Please refer to the attached screenshot and let us know your suggestions. Thank you!
... View more
11-21-2022
09:18 AM
@ask_bill_brooks Our application went through a security scan for this vulnerability which scans for "commons-text" library versions. If "commons-text" library version from 1.5 to 1.9 is present in your Application Lib folder then it would raise a security vulnerability alert. But we are yet to figure out if nifi 1.16.2 is actually vulnerable to this Text4Shell or not. Because our application is not using StringSubstitutor API.
... View more
11-08-2022
12:55 AM
Greetings Cloudera Community!! Text4shell vulnerability is impacting the apache application which is using commons-text version 1.5 to 1.9 and our application Nifi version 1.16.2 hosted on linux server (Red Hat Enterprise Linux Server 7.9) is using commons-text version 1.8 jar file in lib folder. Can anyone please help to figure out the best solution to handle this vulnerability in our production servers: We have few queries for the vulnerability: 1:Is the nifi version 1.16.2 application is affected by this vulnerability? 2: In Nifi configuration files, we are not using any calls related to StringSubstitutor API. Are we still vulnerable to test4Shell? 3: If nifi version 1.16.2 vulnerable then can we just replace the commons-text jar file from 1.8 to 1.10 in nifi 1.16.2. Is there any impact of this in our prod servers? Please do let us know on this vulnerability for nifi 1.16.2. If it is impacting nifi 1.16.2 version then what would be the best solution to mitigate this vulnerability. Vulnerability Details: Release Date: 18th October 2022 CVE Detail: CVE-2022-42889 CVSS Score: Critical (9.8) Affected Products: * Apache Commons Text versions 1.5 through 1.9 * This vulnerability is a remote code execution (RCE) vulnerability, that arises from insecure implementation of Commons Text's variable interpolation functionality, where some default lookup strings could potentially accept untrusted input from remote attackers, such as DNS requests, URLs, or inline scripts and can allow an attacker to execute arbitrary scripts passed to the created interpolator object. * This vulnerability exists in the StringSubstitutor interpolator object. Recommendation: * Upgrade immediately to Apache Commons Text version 1.10.0 Ref: https://www.imperva.com/blog/apache-commons-text-vulnerability-cve-2022-42889/ Ref: https://www.rapid7.com/blog/post/2022/10/17/cve-2022-42889-keep-calm-and-stop-saying-4shell/#:~:text=CVE%2D2022%2D42889%2C%20which,originally%20reported%20by%20Alvaro%20Munoz. Thank you! Girish
... View more
Labels:
- Labels:
-
Apache NiFi