Support Questions

Find answers, ask questions, and share your expertise

Trouble with JsontoSQL processor in Nifi

avatar
Expert Contributor

I am having trouble with my JsonToSQL processor in nifi. I am trying to post to

This table

'Home_Events', 'CREATE TABLE `Home_Events` (\n `hub_insteon_id` varchar(255) DEFAULT NULL,\n `device_insteon_id` varchar(45) DEFAULT NULL,\n `device_group` varchar(45) DEFAULT NULL,\n `status` varchar(45) DEFAULT NULL,\n `recieved_at` varchar(45) DEFAULT NULL\n) ENGINE=InnoDB DEFAULT CHARSET=latin1'

With this JSON

{"hub_insteon_id":"","device_group":"1","device_insteon_id":"3F68A2","recieved_at":"","status":"on"}

Getting This Error

ConvertJSONToSQL[id=d0dd4cc5-f2ab-43ab-8921-b2aafea03cb5] Failed to convert StandardFlowFileRecord[uuid=611848ee-f0e8-40a7-8119-0539d4b531dd,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1461081489294-74, container=default, section=74], offset=8383, length=127],offset=0,name=180088917788802,size=127] to a SQL INSERT statement due to org.apache.nifi.processor.exception.ProcessException: None of the fields in the JSON map to the columns defined by the Home_Automation.Home_Events table; routing to failure: org.apache.nifi.processor.exception.ProcessException: None of the fields in the JSON map to the columns defined by the Home_Automation.Home_Events table

Any ideas how to resolve this?

1 ACCEPTED SOLUTION

avatar
Rising Star

Make the following changes:

  1. Change your connection string to include the database. For example: jdbc:mysql://localhost:3309/[databasename]
  2. Take out the schema and the catalog name from the jsontosql processor configuration and just leave the table name. Your schema will come from the connection string

And you should be good to go.

View solution in original post

13 REPLIES 13

avatar
Guru

Chris can you validate that your DBCPConnectionPool controller is pointing the the appropriate database instance? The JSONToSQL processor will attempt a "describe" using the Connection Service and often this error is the result of that Connection Service not being pointed to the desired database. OR if you using using a Phoenix table be careful as the Phoenix JDBC driver is case sensitive and can make things a little more tricky.

avatar
Guru

ignore the comment about Phoenix. I see you are using MySQL by the "ENGINE=InnoDB" now.

avatar
Expert Contributor

So the execute SQL processor successfully connects with the same settings.

3535-executesql.png

returns

Objavro.schemaú{"type":"record","name":"Home_Events","namespace":"any.data","fields":[{"name":"hub_insteon_id","type":["null","string"]},{"name":"device_insteon_id","type":["null","string"]},{"name":"device_group","type":["null","string"]},{"name":"status","type":["null","string"]},{"name":"recieved_at","type":["null","string"]}]}

and the JsonToSQL processor is configured as

3536-processor-config.png

avatar
Guru

Your configuration looks valid to me. Can you post a screenshot showing your configuration for what is being written to the FlowFile contents and feed to the ConvertJSONToSQL processor? It also might help to validate that the JSON payload you expect is actually in the FlowFile's content by using a LogAttribute processor and setting the "Log Payload" Property to true right before going to the ConvertJSONToSQL processor.

avatar
Expert Contributor

The flow leading into that

3537-config.png

and the AttributesToJson Configuration

3538-attr-to-json.png

avatar
Guru

Ok so the only way you should be seeing this is if JSON isn't in the format the ConvertJSONToSQL is expecting. The processor does a final Iterator<String> fieldNames = rootNode.getFieldNames(); and then performs a while loop on that Iterator incrementing a "fieldCount" variable each time. The only way you could see this is if the JSON isn't really what you think it is. I see the connection between "AttributesToJSON" and "ConvertJSONToSQL" has some FlowFiles in there. Can you right click that connection and list the contents and paste the exact contents of one of them here? Wondering if "AttributesToJSON" is doing something squirrely. I wrote it so its certainly possible ...

avatar
Expert Contributor

{"hub_insteon_id":"","device_group":"1","device_insteon_id":"368D4E","recieved_at":"2016-04-11T23:36:36.332240Z","status":"on"}

avatar
Guru

Ahh I think I see what the problem is I think it is because you have "Include Core Attributes" set to true in AttributesToJSON and some extra fields are getting introduced into the JSON not present in the database table. Please paste that content I mentioned earlier however so I can validate.

avatar
Expert Contributor

Didn't fix it