Support Questions

Find answers, ask questions, and share your expertise

NiFi Error on QueryRecord

avatar
Master Guru

2019-01-10 10:12:56,417 ERROR [Timer-Driven Process Thread-7] o.a.nifi.processors.standard.QueryRecord QueryRecord[id=d2db37b1-0c1a-10cc-7112-fadb5aef597a] Unable to query StandardFlowFileRecord[uuid=c270a5dc-098d-413d-848c-9f1c0d58307c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1547133113269-19924, container=default, section=468], offset=278962, length=62787],offset=0,name=tello-2019-01-08_134755.csv,size=62787] due to org.apache.nifi.processor.exception.ProcessException: Failed to read next record in stream for StandardFlowFileRecord[uuid=c270a5dc-098d-413d-848c-9f1c0d58307c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1547133113269-19924, container=default, section=468], offset=278962, length=62787],offset=0,name=tello-2019-01-08_134755.csv,size=62787] due to Field imu_vg_z cannot be null: org.apache.nifi.processor.exception.ProcessException: Failed to read next record in stream for StandardFlowFileRecord[uuid=c270a5dc-098d-413d-848c-9f1c0d58307c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1547133113269-19924, container=default, section=468], offset=278962, length=62787],offset=0,name=tello-2019-01-08_134755.csv,size=62787] due to Field imu_vg_z cannot be null org.apache.nifi.processor.exception.ProcessException: Failed to read next record in stream for StandardFlowFileRecord[uuid=c270a5dc-098d-413d-848c-9f1c0d58307c,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1547133113269-19924, container=default, section=468], offset=278962, length=62787],offset=0,name=tello-2019-01-08_134755.csv,size=62787] due to Field imu_vg_z cannot be null at org.apache.nifi.queryrecord.FlowFileEnumerator.moveNext(FlowFileEnumerator.java:65) at Baz$1$1.moveNext(Unknown Source) at org.apache.calcite.linq4j.Linq4j$EnumeratorIterator.<init>(Linq4j.java:676) at org.apache.calcite.linq4j.Linq4j.enumeratorIterator(Linq4j.java:96) at org.apache.calcite.linq4j.AbstractEnumerable.iterator(AbstractEnumerable.java:33) at org.apache.calcite.avatica.MetaImpl.createCursor(MetaImpl.java:90) at org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:184) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:64) at org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:43) at org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667) at org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:566) at org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675) at org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) at org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:227) at org.apache.nifi.processors.standard.QueryRecord.query(QueryRecord.java:527) at org.apache.nifi.processors.standard.QueryRecord.onTrigger(QueryRecord.java:303) at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27) at org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1165) at org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:203) at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.nifi.serialization.SchemaValidationException: Field imu_vg_z cannot be null at org.apache.nifi.serialization.record.MapRecord.checkTypes(MapRecord.java:81) at org.apache.nifi.serialization.record.MapRecord.<init>(MapRecord.java:52) at org.apache.nifi.csv.JacksonCSVRecordReader.nextRecord(JacksonCSVRecordReader.java:168) at org.apache.nifi.serialization.RecordReader.nextRecord(RecordReader.java:50) at org.apache.nifi.queryrecord.FlowFileEnumerator.moveNext(FlowFileEnumerator.java:62) ... 26 common frames omitted

3 REPLIES 3

avatar
Master Guru

What does your schema look like? I'm guessing that your "imu_vg_z" field is of some type (maybe "bytes"), rather than a nullable type (such as ["null", "bytes"])?

avatar
Explorer

@mburgess  

Am facing the same error. Any help would be appreciated.

Am a beginner in Nifi, and trying out some example use-cases available over internet. 

 

Trying out "Real-Time SQL On Event Streams"

Use case: CSV to Json Conversion

CSV is like :

purchase_no, customer_id, item_id, item_name, price, quantity
10280, 40070, 1028, Box of pencils, 6.99, 2
10280, 40070, 4402, Stapler, 12.99, 1
12440, 28302, 1029, Box of ink pens, 8.99, 1
28340, 41028, 1028, Box of pencils, 6.99, 18
28340, 41028, 1029, Box of ink pens, 8.99, 18
28340, 41028, 2038, Printer paper, 14.99, 10
28340, 41028, 4018, Clear tape, 2.99, 10
28340, 41028, 3329, Tape dispenser, 14.99, 10
28340, 41028, 5192, Envelopes, 4.99, 45
28340, 41028, 3203, Laptop Computer, 978.88, 2
28340, 41028, 2937, 24\" Monitor, 329.98, 2
49102, 47208, 3204, Powerful Laptop Computer, 1680.99, 1

 

and my schema looks like:

{
"name": "helloWorld",
"namespace": "org.apache.nifi.blogs",
"type": "record",
"fields": [
{ "name": "purchase_no", "type": "long" },
{ "name": "customer_id", "type": "long" },
{ "name": "item_id", "type": ["null", "long"] },
{ "name": "item_name", "type": ["null", "string"] },
{ "name": "price", "type": ["null", "double"] },
{ "name": "quantity", "type": ["null", "int"] },
{ "name": "total_price", "type": ["null", "double"] }
]
}

 

I made sure that all the field type is matched, still facing the below error.

ERROR 418b70f8-b8a3-3b60-5042-85ad82a7bf72
QueryRecord[id=418b70f8-b8a3-3b60-5042-85ad82a7bf72] Unable to query StandardFlowFileRecord[uuid=80edfc74-9637-4eaa-abcb-ad2444f1c2da,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1580960212569-1, container=default, section=1], offset=4270, length=599],offset=0,name=80edfc74-9637-4eaa-abcb-ad2444f1c2da,size=599] due to Failed to read next record in stream for StandardFlowFileRecord[uuid=80edfc74-9637-4eaa-abcb-ad2444f1c2da,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1580960212569-1, container=default, section=1], offset=4270, length=599],offset=0,name=80edfc74-9637-4eaa-abcb-ad2444f1c2da,size=599] due to Error while getting next record. Root cause: java.lang.NumberFormatException: For input string: "purchase_no, customer_id, item_id, item_name, price, quantity": org.apache.nifi.processor.exception.ProcessException: Failed to read next record in stream for StandardFlowFileRecord[uuid=80edfc74-9637-4eaa-abcb-ad2444f1c2da,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1580960212569-1, container=default, section=1], offset=4270, length=599],offset=0,name=80edfc74-9637-4eaa-abcb-ad2444f1c2da,size=599] due to Error while getting next record. Root cause: java.lang.NumberFormatException: For input string: "purchase_no, customer_id, item_id, item_name, price, quantity"

 

 

My Queries are as below: [added below as attributes in QueruRecord Processor to query ]

Query 1:  Will include the purchase_no and customer_id fields of any purchase that cost more than $1,000.00 and will also include a new field named total_price that is the dollar amount for the entire purchase.

Property name : "over.1000"

Value : SELECT purchase_no, customer_id, SUM(price * quantity) AS total_priceFROM FLOWFILEGROUP BY purchase_no, customer_idHAVING SUM(price * quantity) > 1000

 

Query 2: Will contain the purchase_no, customer_id, and total price of the most expensive single purchase (as defined by price times quantity) in the data: Property : "largest.order"  Value : SELECT purchase_no, customer_id, SUM(price * quantity) AS total_priceFROM FLOWFILEGROUP BY purchase_no, customer_idORDER BY total_price DESCLIMIT 1

 

 

 

avatar
Master Guru

That error indicates that your CSVReader is not treating the first line as a header, and thus even if you specify an explicit schema, if you don't set Treat First Line As Header to true, the reader will think the header line is a data line, and when it tries to parse it as data (numbers, e.g.) it will fail.