How to convert the sensor data which is in unusual format to the Jason format.

[#1]

I am very new to the nifi and have problem casting stream data are generated through sensors over the Internet of Things environment as follows.I want to convert that data to the Jason format using nifi. Sensor data in edge side saved into the sensordata.csv file and using [Tailfile processer] in Minifi send data to the Nifi using S2S(siteTosite). Data reached to the nifi side and s2s works fine. I tried to do this using below procesers in nifi, but not going well.

convertCSVtoAvro > convertAvroToJason

Anyone has good idea to do this?

Thanks in advance!

Edge data as follows.

"{'magnetometer': (75.26935286935287, 0.14993894993894993, -13.194627594627594), 'barometer': (27.7, 1010.44), 'temperature': 0.0, 'accelerometer': (-0.01123046875, 0.0419921875, 2.02001953125), 'battery': 79, 'timestamp': '2019-02-20 17:52:39.099323+09:00', 'humidity': (27.42401123046875, 46.136474609375), 'lightmeter': (-0.00341796875, 0.03564453125, 2.0185546875), 'address': u'98:07:2d:26:d6:82', 'gyroscope': (-2.4871826171875, 1.15966796875, 0.823974609375)}"

"{'magnetometer': (75.26935286935287, 1.4993894993894994, -12.444932844932845), 'barometer': (27.71, 1010.46), 'temperature': 0.0, 'accelerometer': (-0.01171875, 0.03662109375, 2.02685546875), 'battery': 79, 'timestamp': '2019-02-20 17:52:53.226606+09:00', 'humidity': (27.42401123046875, 46.136474609375), 'lightmeter': (0.0, 0.04443359375, 2.01904296875), 'address': u'98:07:2d:26:d6:82', 'gyroscope': (-2.47955322265625, 0.885009765625, 0.87738037109375)}"

"{'magnetometer': (76.01904761904761, 0.29987789987789987, -10.195848595848595), 'barometer': (27.71, 1010.37), 'temperature': 0.0, 'accelerometer': (-0.0068359375, 0.04541015625, 2.01513671875), 'battery': 79, 'timestamp': '2019-02-20 17:53:07.076272+09:00', 'humidity': (27.40386962890625, 46.234130859375), 'lightmeter': (-0.0126953125, 0.0458984375, 2.021484375), 'address': u'98:07:2d:26:d6:82', 'gyroscope': (-2.35748291015625, 0.946044921875, 0.67901611328125)}"

"{'magnetometer': (73.76996336996336, -0.29987789987789987, -13.194627594627594), 'barometer': (27.71, 1010.44), 'temperature': 0.0, 'accelerometer': (-0.00537109375, 0.041015625, 2.01611328125), 'battery': 79, 'timestamp': '2019-02-20 17:53:21.106742+09:00', 'humidity': (27.40386962890625, 46.234130859375), 'lightmeter': (-0.00537109375, 0.0419921875, 2.01708984375), 'address': u'98:07:2d:26:d6:82', 'gyroscope': (-2.0904541015625, 1.01470947265625, 0.9613037109375)}"

[#2 · Accepted solution]

Is this a standard format (I know it's not proper JSON but is it some other standard format)? If not, you could write a ScriptedReader to parse your records. If all the data looks like the above though, you could use ReplaceText to replace ( with [ and ) with ], also remove the "u" prefix from some strings (like the address field), thereby converting it to proper JSON so you can use components like JsonTreeReader for example.

[#3]

Thanks Matt your reply and its very helpful to me. yes..this is not proper JSON. It's just like RAW data.

That means I have to shape the data first and then convert it to the JSON.

I will try to do that above way you mentioned.

I have a another question about nifi.

What are the key features or benefits of using NiFi and how it different from other products?

Thanks in advance!