Hi , Thank you for your assistance with this matter. The answers to your questions is as follows: Is that complete stack trace from the nifi-app.log? No, the complete stack trace is the following one: 2023-03-29 10:02:21,002 ERROR [Timer-Driven Process Thread-24] o.a.n.p.standard.PartitionRecord PartitionRecord[id=3be1c42e-5fa9-3144-3365-f568bb616028] Processing halted: yielding [1 sec] java.lang.IllegalArgumentException: newLimit > capacity: (92 > 83) at java.base/java.nio.Buffer.createLimitException(Buffer.java:372) at java.base/java.nio.Buffer.limit(Buffer.java:346) at java.base/java.nio.ByteBuffer.limit(ByteBuffer.java:1107) at java.base/java.nio.MappedByteBuffer.limit(MappedByteBuffer.java:235) at java.base/java.nio.MappedByteBuffer.limit(MappedByteBuffer.java:67) at org.xerial.snappy.Snappy.compress(Snappy.java:156) at org.apache.parquet.hadoop.codec.SnappyCompressor.compress(SnappyCompressor.java:78) at org.apache.hadoop.io.compress.CompressorStream.compress(CompressorStream.java:81) at org.apache.hadoop.io.compress.CompressorStream.finish(CompressorStream.java:92) at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.compress(CodecFactory.java:167) at org.apache.parquet.hadoop.ColumnChunkPageWriteStore$ColumnChunkPageWriter.writePage(ColumnChunkPageWriteStore.java:168) at org.apache.parquet.column.impl.ColumnWriterV1.writePage(ColumnWriterV1.java:59) at org.apache.parquet.column.impl.ColumnWriterBase.writePage(ColumnWriterBase.java:387) at org.apache.parquet.column.impl.ColumnWriteStoreBase.flush(ColumnWriteStoreBase.java:186) at org.apache.parquet.column.impl.ColumnWriteStoreV1.flush(ColumnWriteStoreV1.java:29) at org.apache.parquet.hadoop.InternalParquetRecordWriter.flushRowGroupToStore(InternalParquetRecordWriter.java:185) at org.apache.parquet.hadoop.InternalParquetRecordWriter.close(InternalParquetRecordWriter.java:124) at org.apache.parquet.hadoop.ParquetWriter.close(ParquetWriter.java:319) at org.apache.nifi.parquet.record.WriteParquetResult.close(WriteParquetResult.java:69) at java.base/jdk.internal.reflect.GeneratedMethodAccessor983.invoke(Unknown Source) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:254) at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler.access$100(StandardControllerServiceInvocationHandler.java:38) at org.apache.nifi.controller.service.StandardControllerServiceInvocationHandler$ProxiedReturnObjectInvocationHandler.invoke(StandardControllerServiceInvocationHandler.java:240) at com.sun.proxy.$Proxy316.close(Unknown Source) at org.apache.nifi.processors.standard.PartitionRecord.onTrigger(PartitionRecord.java:274) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1356) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246) at org.apache.nifi.controller.scheduling.AbstractTimeBasedSchedulingAgent.lambda$doScheduleOnce$0(AbstractTimeBasedSchedulingAgent.java:59) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)   What version of Apache NiFi? Currently running on Apache NiFi open source 1.19.1 What version of Java? Currently running on openjdk version "11.0.17" 2022-10-18 LTS Have you tried using ConsumeKafkaRecord processor instead of ConsumeKafka --> MergeContent? No I did not, but for a good reason. The files coming out of Kafka require some "data manipulation" before using PartitionRecord, where I have defined the CSVReader and the ParquetRecordSetWriter. If I were to use ConsumeKafkaRecord, I would have to define a CSV Reader and the Parquet(or CSV)RecordSetWriter and the result will be very bad, as the data is not formatted as per the required schema. I will give it a try with ConsumeKafkaRecord using CSVReader and CSVRecordSetWriter, to see if I still encounter the same issue. Do you have issue only when using the ParquetRecordSetWriter? Unfortunately I can only test with parquet as this file format is somehow mandatory for the current project. I will try to reproduce the flow with an AVRO format, to see if I can reproduce the error or not. How large are the FlowFiles coming out of the MergeContent processor? So directly out of Kafka, 1 FlowFile has around 600-700 rows, as text/plain and the size is 300-600KB. Using MergeContent, I combine a total of 100-150 files, resulting in a total of 50MB. Have you tried reducing the size of the Content being output from MergeContent processor? Yes, I have played with several combinations of sizes and most of them either resulted in the same error or in an "to many open files" error.   ... View more

hi @Sivagopal, As far as I know, what you are asking is not possible directly from within NiFi. To remove the node from the cluster you have two options: - the manual remove, done from the Menu - Cluster - Nodes --> disconnect Node. ( see https://docs.cloudera.com/HDPDocuments/HDF3/HDF-3.5.1/nifi-configuration-best-practices/content/disconnect-nodes.html) - the NiFi's REST API. For that you will need to use the Controller commands, select the DELETE option and perform the call --> /controller/cluster/nodes/{id}. (see https://nifi.apache.org/docs/nifi-docs/rest-api/index.html --> Controller Section / DELETE)   As for the update in authorizers.xml, you will need to write a script which will perform this action, while NiFi is down. As far as I know, these XML Files get initialized when NiFi starts so even though you perform some actions while NiFi is up and running, these changes will not get taken into considerations. Take note that if you are using the embedded zookeeper, you will have to modify the zookeeper.properties file as well. In addition to these files, you have to take a look to state-management.xml as well. ... View more

So the question with you entering the correct password is still valid, even though it is a dumb question. 🙂 Nevertheless, would it be possible for you to add the nifi.properties config file here? You can replace the password properties with something else. ... View more

What do you mean with it is working fine on local server but there is a problem when you access it remotely? How do you access it remotely? Are you using the Apache NiFi version or the Clouder version?  Now, as I do not have all the info yet, I would go ahead and ask a dumb question. If you are trying to access it remotely, I assume that you are using a different device (pc, laptop, vm, etc.), so therefore, are you sure that you are using the same keyboard layout and you are typing the correct password? ... View more

To be honest I do not really understand your question. If you split your original file into smaller chunks, you will have X flowfiles instead of a single flowfile. Those flowfiles will then go sequentially in your HTTP response Processor, based on your load balance strategy. ... View more

hi @saquibsk,   I assume that you are using the single user provider policy and not the integration with the LDAP/Kerberos and so on.   Most likely, when you deleted and re-extracted your NiFi Folder you have over-written your authorizations.xml, authorizers.xml, login-identity-provider.xml and users.xml files. From my point of view, your problem is related to the login-identity-providers.xml, as this is the file where you have your password for your user. You could try and get a back-up of this file, extract the password from there and paste it into your current file. Otherwise, you can generate a new password for your user. Or (not sure if it will work) you can see what encryption NiFi uses for hashing the password, and generate your old password using that hash and paste it within your login-identity-provider.xml file. PS: make sure that you stop NiFi before modifying the file 🙂 ... View more

Well, there you have it 🙂 Your problem is not related directly to NiFi and it is caused by the executed SQL statement. I am not very experienced with SQL Servers but you could try selecting all the fields from the table and see if you still encounter an error message. If the error still persists, you could use the convert function on the where clause --> CONVERT(datetime,your_value,25) or select convert(varchar, your_value, 25) ... View more

I see that you are executing a different select in your database than the one executed in NiFi. Try with that exact SQL statement and see if it works. In what you are executing now, I see no WHERE condition set on that column date. In NiFi you are executing: "SELECT dt_change FROM CompanyXYZ.dbo.test WHERE dt_change > '2023-03-28 09:55:38.783' " As for the convert function, you receive no error message, it works just fine. But as stated above, you need to select the correct format for your use case. All the formats are present in that link. ... View more

I see that you are executing a different select in your database than the one executed in NiFi. Try with that exact SQL statement and see if it works. In what you are executing now, I see no WHERE condition set on that column date. In NiFi you are executing: "SELECT dt_change FROM CompanyXYZ.dbo.test WHERE dt_change > '2023-03-28 09:55:38.783' " As for the convert function, you receive no error message, it works just fine. But as stated above, you need to select the correct format for your use case. All the formats are present in that link. ... View more