The idea is to decode 1600 files which contain 4000 records each on average, that is 6.400.000 records in total, these must be processed and sent through nifi in less than 10 minutes to another server sftp. Thanks for the help 😉 ... View more

Hi, Lester. I appreciate very much that you have responded. You are right, a server is very little :), but it is what I have for now, if the project goes ahead our manager will provide us with more workers. Next, I send screenshots with the topology processing. Currently the server has 56 cores and 256 GB of RAM ... View more

Hi Matt, Thanks for answering This is the nifi-app.log 2018-11-07 08:59:56,176 INFO [Process Cluster Protocol Request-10] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 68d679fd-b03d-465f-92b2-d01561d2411a (type=HEARTBEAT, length=1838 bytes) from desbighdf001:9090 in 0 millis 2018-11-07 08:59:56,177 INFO [Clustering Tasks Thread-3] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-11-07 08:59:56,175 and sent to desbighdf001:9088 at 2018-11-07 08:59:56,177; send took 2 millis 2018-11-07 08:59:57,101 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.lucene.SimpleIndexManager Index Writer for /var/lib/nifi/provenance_repository/index-1540398231000 has been returned to Index Manager and is no longer in use. Closing Index Writer 2018-11-07 08:59:57,150 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (34599 records) into single Provenance Log File /var/lib/nifi/provenance_repository/14892361.prov in 4165 milliseconds 2018-11-07 08:59:57,151 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 20824 records. In the past 5 minutes, 589689 events have been written to the Provenance Repository, totaling 1.64 GB 2018-11-07 08:59:58,143 ERROR [Timer-Driven Process Thread-23] company.asn1.asn1.MyProcessor MyProcessor[id=ee79d747-0165-1000-ffff-ffffee553d3a] MyProcessor[id=ee79d747-0165-1000-ffff-ffffee553d3a] failed to process due to java.lang.ArrayIndexOutOfBoundsException; rolling back session: {} java.lang.ArrayIndexOutOfBoundsException: null 2018-11-07 08:59:58,143 ERROR [Timer-Driven Process Thread-23] company.asn1.asn1.MyProcessor MyProcessor[id=ee79d747-0165-1000-ffff-ffffee553d3a] MyProcessor[id=ee79d747-0165-1000-ffff-ffffee553d3a] failed to process session due to java.lang.ArrayIndexOutOfBoundsException: {} java.lang.ArrayIndexOutOfBoundsException: null 2018-11-07 08:59:58,143 WARN [Timer-Driven Process Thread-23] company.asn1.asn1.MyProcessor MyProcessor[id=ee79d747-0165-1000-ffff-ffffee553d3a] Processor Administratively Yielded for 1 sec due to processing failure ... View more