Created on 06-07-2019 06:04 PM - edited 09-16-2022 07:26 AM
Replciating data from ORacle => Couchabse fine.
QueryDatabaseTable=>SplitArvo=>ConvertArvotoJson=>PutCouchbaseKey.
I add a replacetextprocessor after the query (Literla text replace) and now errors
===========================
2019-06-06 20:36:46,516 WARN [Timer-Driven Process Thread-3] o.a.n.controller.tasks.ConnectableTask Administratively Yielding SplitAvro[id=241d3e17-016b-1000-68f3-2a34dba86e4f] due to uncaught Exception: java.lang.IllegalArgumentException java.lang.IllegalArgumentException: null at java.nio.ByteBuffer.allocate(ByteBuffer.java:334) at org.apache.avro.io.BinaryDecoder.readBytes(BinaryDecoder.java:288) at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:112) at org.apache.avro.file.DataFileStream.(DataFileStream.java:84) at org.apache.nifi.processors.avro.SplitAvro$RecordSplitter$1.process(SplitAvro.java:271) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180) at org.apache.nifi.processors.avro.SplitAvro$RecordSplitter.split(SplitAvro.java:267) at org.apache.nifi.processors.avro.SplitAvro.onTrigger(SplitAvro.java:224) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Created 06-10-2019 12:35 PM
Your syntax in replace text is invalid, resulting in the next error in the Avro Processor.
Confirm your syntax is operational. Another recommendation is stop Avro, and inspect the FlowFile and Attributes are what you expect before trying play through to the Avro Processor.
Created on 06-10-2019 03:31 PM - edited 08-17-2019 03:05 PM
Hi @steven Matison
My textreplace processor is pretty simple (literal).
(I've tried with and without quotes).
It passes this processor ok and actually fails on an existing processor (unchanged) for SplitArvo
Created 06-10-2019 08:24 PM
So I replaced this with a flow that goes .....
querydatabasetable=>ConvertArvotoJSON=>ReplaceTEXT=>PutCouchbaseKey
Works perfect.
Still confused on when to use which processors
i.e I used ConvertArvotoJson only because I wanted JSON format. Was looking for a SQLtoJSOON - but doesnt exist.
So quesitons
1) arvo/binary/text processors - how do you know if you need to convert between two main processors ? Based on format etc. eg When it comes out of the querydatabasetable => what format is it in ?
2) How do you do a bulk load - and then an incremental sync from a DB or a file.