Member since
10-17-2018
7
Posts
0
Kudos Received
0
Solutions
02-21-2019
10:38 PM
The idea is to decode 1600 files which contain 4000 records each on average, that is 6.400.000 records in total, these must be processed and sent through nifi in less than 10 minutes to another server sftp. Thanks for the help 😉
... View more
02-21-2019
10:34 PM
Hi, Lester. I appreciate very much that you have responded. You are right, a server is very little :), but it is what I have for now, if the project goes ahead our manager will provide us with more workers. Next, I send screenshots with the topology processing. Currently the server has 56 cores and 256 GB of RAM
... View more
02-18-2019
09:37 PM
hello community, We created a custom storm processor which decodes information from asn1 to xml (ASN12XLTE), which consumes a lot of processing. Currently we have a single server where everything is installed (ambari, kafka, storm, etc). What parameters can I check so that in this processor does not form a bottleneck. Thank you very much for your help
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache Storm
11-13-2018
06:33 PM
Hi Matt, Thanks for answering This is the nifi-app.log 2018-11-07 08:59:56,176 INFO [Process Cluster Protocol Request-10] o.a.n.c.p.impl.SocketProtocolListener Finished processing request 68d679fd-b03d-465f-92b2-d01561d2411a (type=HEARTBEAT, length=1838 bytes) from desbighdf001:9090 in 0 millis
2018-11-07 08:59:56,177 INFO [Clustering Tasks Thread-3] o.a.n.c.c.ClusterProtocolHeartbeater Heartbeat created at 2018-11-07 08:59:56,175 and sent to desbighdf001:9088 at 2018-11-07 08:59:56,177; send took 2 millis
2018-11-07 08:59:57,101 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.lucene.SimpleIndexManager Index Writer for /var/lib/nifi/provenance_repository/index-1540398231000 has been returned to Index Manager and is no longer in use. Closing Index Writer
2018-11-07 08:59:57,150 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully merged 16 journal files (34599 records) into single Provenance Log File /var/lib/nifi/provenance_repository/14892361.prov in 4165 milliseconds
2018-11-07 08:59:57,151 INFO [Provenance Repository Rollover Thread-2] o.a.n.p.PersistentProvenanceRepository Successfully Rolled over Provenance Event file containing 20824 records. In the past 5 minutes, 589689 events have been written to the Provenance Repository, totaling 1.64 GB
2018-11-07 08:59:58,143 ERROR [Timer-Driven Process Thread-23] company.asn1.asn1.MyProcessor MyProcessor[id=ee79d747-0165-1000-ffff-ffffee553d3a] MyProcessor[id=ee79d747-0165-1000-ffff-ffffee553d3a] failed to process due to java.lang.ArrayIndexOutOfBoundsException; rolling back session: {}
java.lang.ArrayIndexOutOfBoundsException: null
2018-11-07 08:59:58,143 ERROR [Timer-Driven Process Thread-23] company.asn1.asn1.MyProcessor MyProcessor[id=ee79d747-0165-1000-ffff-ffffee553d3a] MyProcessor[id=ee79d747-0165-1000-ffff-ffffee553d3a] failed to process session due to java.lang.ArrayIndexOutOfBoundsException: {}
java.lang.ArrayIndexOutOfBoundsException: null
2018-11-07 08:59:58,143 WARN [Timer-Driven Process Thread-23] company.asn1.asn1.MyProcessor MyProcessor[id=ee79d747-0165-1000-ffff-ffffee553d3a] Processor Administratively Yielded for 1 sec due to processing failure
... View more
11-08-2018
06:28 PM
Hi all,
We create a custom processor to decode asn1 to xml, the processor works correctly with a concurrent task, but if more than one task is configured, it generates the error message: 9:12:31 COTERROR8afd3800-4a4f-1f31-85e1-4848d10ea5fc
desbighdf001:9090MyProcessor[id=8afd3800-4a4f-1f31-85e1-4848d10ea5fc] MyProcessor[id=8afd3800-4a4f-1f31-85e1-4848d10ea5fc] failed to process session due to java.lang.ArrayIndexOutOfBoundsException: null
09:12:31 COTWARNING8afd3800-4a4f-1f31-85e1-4848d10ea5fc
desbighdf001:9090MyProcessor[id=8afd3800-4a4f-1f31-85e1-4848d10ea5fc] Processor Administratively Yielded for 1 sec due to processing failur
... View more
Labels:
- Labels:
-
Apache NiFi
10-17-2018
09:59 PM
@ Vino chithra Sriraman Al ejecutar el comando se genera un error, ¿cómo puedo eliminar los esquemas que no uso? curl -X BORRAR http: // localhost: 7788 / subject / TEST_2 / versions / 1 {"responseMessage": "Se produjo una excepción con el mensaje [Método HTTP 405 No permitido] mientras se procesaba la solicitud."}
... View more