Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Problems with flow after adding replacetext processor

Problems with flow after adding replacetext processor

New Contributor

Replciating data from ORacle => Couchabse fine.

QueryDatabaseTable=>SplitArvo=>ConvertArvotoJson=>PutCouchbaseKey.


I add a replacetextprocessor after the query (Literla text replace) and now errors


===========================

2019-06-06 20:36:46,516 WARN [Timer-Driven Process Thread-3] o.a.n.controller.tasks.ConnectableTask Administratively Yielding SplitAvro[id=241d3e17-016b-1000-68f3-2a34dba86e4f] due to uncaught Exception: java.lang.IllegalArgumentException java.lang.IllegalArgumentException: null at java.nio.ByteBuffer.allocate(ByteBuffer.java:334) at org.apache.avro.io.BinaryDecoder.readBytes(BinaryDecoder.java:288) at org.apache.avro.file.DataFileStream.initialize(DataFileStream.java:112) at org.apache.avro.file.DataFileStream.(DataFileStream.java:84) at org.apache.nifi.processors.avro.SplitAvro$RecordSplitter$1.process(SplitAvro.java:271) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2212) at org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2180) at org.apache.nifi.processors.avro.SplitAvro$RecordSplitter.split(SplitAvro.java:267) at org.apache.nifi.processors.avro.SplitAvro.onTrigger(SplitAvro.java:224) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1162) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:209) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

3 REPLIES 3

Re: Problems with flow after adding replacetext processor

Super Collaborator

Your syntax in replace text is invalid, resulting in the next error in the Avro Processor.
Confirm your syntax is operational. Another recommendation is stop Avro, and inspect the FlowFile and Attributes are what you expect before trying play through to the Avro Processor.

Re: Problems with flow after adding replacetext processor

New Contributor

Hi @steven Matison

My textreplace processor is pretty simple (literal).

109297-1560180519290.png

(I've tried with and without quotes).

It passes this processor ok and actually fails on an existing processor (unchanged) for SplitArvo

109323-1560180577450.png


Highlighted

Re: Problems with flow after adding replacetext processor

New Contributor

So I replaced this with a flow that goes .....
querydatabasetable=>ConvertArvotoJSON=>ReplaceTEXT=>PutCouchbaseKey

Works perfect.


Still confused on when to use which processors


i.e I used ConvertArvotoJson only because I wanted JSON format. Was looking for a SQLtoJSOON - but doesnt exist.


So quesitons


1) arvo/binary/text processors - how do you know if you need to convert between two main processors ? Based on format etc. eg When it comes out of the querydatabasetable => what format is it in ?


2) How do you do a bulk load - and then an incremental sync from a DB or a file.


Don't have an account?
Coming from Hortonworks? Activate your account here