Member since
01-04-2023
3
Posts
1
Kudos Received
0
Solutions
01-20-2023
04:59 AM
Hi Folks, I've been facing an issue with nifi pod restarting only when persistence is enabled, Below is the app.log excerpt. I doubt this could be related to provenance. Can someone help? APP.log 2023-01-20 12:47:20,598 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loading Application Properties [/opt/nifi/nifi-current/./conf/nifi.properties] 2023-01-20 12:47:29,181 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@68321229 finished recovering records. Performing Checkpoint to ensure proper state of Partitions before updates 2023-01-20 12:47:29,181 INFO [main] org.wali.MinimalLockingWriteAheadLog Successfully recovered 0 records in 2 milliseconds 2023-01-20 12:47:29,192 INFO [main] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@68321229 checkpointed with 0 Records and 0 Swap Files in 11 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 1 millis), max Transaction ID -1 2023-01-20 12:47:29,991 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded 117 properties from system properties and environment variables 2023-01-20 12:47:29,991 INFO [main] o.a.n.r.v.FileBasedVariableRegistry Loaded a total of 117 properties. Including precedence overrides effective accessible registry key size is 117 2023-01-20 12:47:31,323 INFO [main] o.a.n.c.r.WriteAheadFlowFileRepository Initialized FlowFile Repository 2023-01-20 12:47:31,328 INFO [main] o.apache.nifi.controller.FlowController Creating Provenance Repository [org.apache.nifi.provenance.WriteAheadProvenanceRepository] 2023-01-20 12:47:31,598 INFO [main] o.a.n.p.store.WriteAheadStorePartition After recovering ../provenance_repository, next Event ID to be generated will be 0 2023-01-20 12:47:31,754 INFO [main] o.a.n.p.index.lucene.LuceneEventIndex Will avoid re-indexing Provenance Events because the newest index is defunct, so it will be re-indexed in the background 2023-01-20 12:47:31,806 INFO [main] o.apache.nifi.controller.FlowController Creating Content Repository [org.apache.nifi.controller.repository.FileSystemRepository] 2023-01-20 12:47:31,827 INFO [main] o.a.n.c.repository.FileSystemRepository Maximum Threshold for Container default set to 9126805504 bytes; if volume exceeds this size, archived data will be deleted until it no longer exceeds this size 2023-01-20 12:47:31,842 INFO [main] o.a.n.c.repository.FileSystemRepository Initializing FileSystemRepository with 'Always Sync' set to false 2023-01-20 12:49:29,376 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog@68321229 checkpointed with 0 Records and 0 Swap Files in 182 milliseconds (Stop-the-world time = 1 milliseconds, Clear Edit Logs time = 2 millis), max Transaction ID -1 2023-01-20 12:49:55,078 INFO [main] org.apache.nifi.NiFi Launching NiFi... 2023-01-20 12:49:55,188 INFO [main] o.a.nifi.properties.NiFiPropertiesLoader Loading Application Properties [/opt/nifi/nifi-current/./conf/nifi.properties] 2023-01-20 12:49:55,198 INFO [main] org.apache.nifi.NiFi Application Properties loaded [142] 2023-01-20 12:49:55,291 INFO [main] org.apache.nifi.BootstrapListener Started Bootstrap Listener, Listening for incoming requests on port 40217 2023-01-20 12:49:55,314 INFO [main] org.apache.nifi.BootstrapListener Successfully initiated communication with Bootstrap 2023-01-20 12:49:55,390 INFO [main] org.apache.nifi.nar.NarUnpacker Expanding 122 NAR files with all processors... Persistence settings configStorage: size: 100Mi authconfStorage: size: 100Mi # Storage capacity for the 'data' directory, which is used to hold things such as the flow.xml.gz, configuration, state, etc. dataStorage: size: 1Gi # Storage capacity for the FlowFile repository flowfileRepoStorage: size: 3Gi # Storage capacity for the Content repository contentRepoStorage: size: 3Gi # Storage capacity for the Provenance repository. When changing this, one should also change the properties.provenanceStorage value above, also. provenanceRepoStorage: size: 4Gi # Storage capacity for nifi logs logStorage: size: 3Gi Nifi.properties related to provenance nifi.provenance.repository.directory.default=../provenance_repository nifi.provenance.repository.max.storage.time=10 days nifi.provenance.repository.max.storage.size={{.Values.properties.provenanceStorage}} nifi.provenance.repository.rollover.time=30 secs nifi.provenance.repository.rollover.size=100 MB nifi.provenance.repository.query.threads=2 nifi.provenance.repository.index.threads=2 nifi.provenance.repository.compress.on.rollover=true nifi.provenance.repository.always.sync=false nifi.provenance.repository.journal.count=16 # Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are: # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship # FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type nifi.provenance.repository.indexed.attributes= # Large values for the shard size will result in more Java heap usage when searching the Provenance Repository # but should provide better performance nifi.provenance.repository.index.shard.size=400 MB # Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from # the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved. nifi.provenance.repository.max.attribute.length=65536
... View more
Labels:
- Labels:
-
Apache NiFi
01-12-2023
01:30 PM
@SachinMehndirat There is NO replication of data from the four NiFi repositories across all NiFi nodes in a NiFi cluster. Each NiFi node in the cluster is only aware of and only excutes against the FlowFile on that specific node. As such, NiFi nodes can not share a common set of repositories. Each node must have their own repositories and it is important to protect those repositories from data loss (flowfile_repository and content_repository being most important). - flowfile_repository - contain metadata/attributes about FlowFiles actively processing thorugh your NiFi dataflow(s). This includes metadata on location of content of queued FlowFiles. - content_repository - contains content claims that can hold the content for 1 too many FlowFiles actively being processed or temporarily archived post termination at end of dataflow(s) - provenance_repository - contains historical lineage information about FlowFile currently or previously processed through your NiFi dataflows. - database_repository - contains flow configuration history which is a record of changes made via NiFi UI (adding, modifying, deleting, stopping, starting, etc...). Also contain info about users currently authenticated in to the NiFi node. Processors that record cluster wide state would use zookeeper to store and retrieve that stored state needed by all nodes. Processors that use local state will write that state to NiFi locally configured state directory. So in addition to protect the repositories mentioned above from dataloss, you'll also want to make sure local state (unique to each node in the NiFi cluster) directory is also protected. The embedded documentation in NiFi for each component has a section "State management:" that will tell you if that component use local and/or cluster state. You may find some of the info found in the following articles useful: https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999 https://community.cloudera.com/t5/Community-Articles/Understanding-how-NiFi-s-Content-Repository-Archiving-works/ta-p/249418 https://blogs.apache.org/nifi/entry/load-balancing-across-the-cluster If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more
01-05-2023
11:51 AM
@SachinMehndirat The exception comes from java and not NiFi's core code base. The exception is possibly being thrown as a result of a failed mutual TLS handshake. Without the complete stack trace and preceding log output it is difficult to give a detailed response here, so below is based on assumptions from what little has been shared. However, when you see "chain" issues (keeping in mind that output shared is incomplete), that points at missing TrustedCertEntry(s) in the truststore used one side or the other of that TLS handshake. Both the client and server sides of a TLS handshake present their clientAuth or serverAuth certificate info. These PrivateKeyEntry certificates will have an owner and issuer (signing Certificate Authority (CA) for the owner). In order for the receiving sides to trust the certificate presented from the other side, it must trust that CA. This means that within its truststore, it must have a TrustedCertEntry for the CA that signed the PrivateKey presented in the TLS handshake. BUT... it does not end there. The CA may be an intermediate CA meaning that the certificate for that CA was signed by yet another CA. In that case the truststore would also need the TrustedCertEntry for the next CA in that chain of authority. A complete trust "chain" would consist of all CAs from signer of PrivateKey to the root CA (root CA public key will have same Distinguished Name (DN) for both owner and signer of the TrustedCertEntry). So on one side or the other you do not have the complete trustchain in the truststore. If you found that the provided solution(s) assisted you with your query, please take a moment to login and click Accept as Solution below each response that helped. Thank you, Matt
... View more