@Sanaz Janbakhsh i tried with your json message and eval json path processor with payload property as $.payload processor working as expected.. Output:- Can you share screenshots about your processors configurations,input to processor and flow, that would be great help to find root cause of your issue. ... View more

@LOKASHIS RANA As the logs shows ORA-01840: input value not long enough for date format Try to use some value for last value like '1900-01-01 00:00:00.0' instead of 0 and run again with below sqoop import statement. sqoop import--connect jdbc:oracle:thin:system/system@192.168.XXX.XXX:1521:xe --username sonu -password sonu --table DEMO_ORDERS --target-dir '/user/root/orders/' --as-textfile --split-by CUSTOMER_ID --incremental lastmodified --check-column ORDER_TIMESTAMP --last-value '1900-01-01 00:00:00.0'--num-mappers 5 --fields-terminated-by',' In addition once make sure how the data looks like for ORDER_TIMESTAMP column in source table if it is having just date in it like 2011-01-01 then you need to pass last-value same like the data in source Example:- if source table ORDER_TIMESTAMP data looks like 2011-01-01 then --last-value 1900-01-01 //we need to give last value same format matches with the data in source table. ... View more

Hi @Tarek Elgamal, Click on Global menu and Click on Controller settings button then in General tab from this tab you can control threads running in NiFi instance. In addition you can refer to below community links https://community.hortonworks.com/questions/76117/how-to-improve-nifi-concurrency.html ... View more

@sally sally yeah, you can do that by using replace text processor with search value property as <details>\s*([\s\S]+.*)\n+\s+<\/details> //capture every thing enclosed in details tag as capture group 1 then in replacement value <details> ${filename} $1 </details> you can customize the replacement value as per your needs. Replace Text processor Configs:- input:- <?xml version="1.0" encoding="UTF-8"?> <service> <Person> <details> <start>2017-10-22</start> <id>*******</id> <makeVersion>1</makeVersion> <patch>patch</patch> <parameter>1</parameter> </details> </Person> </service> output:- <?xml version="1.0" encoding="UTF-8"?> <service> <Person> <details> 1497701925152409 <start>2017-10-22</start> <id>*******</id> <makeVersion>1</makeVersion> <patch>patch</patch> <parameter>1</parameter> </details> </Person> </service ... View more

@Sanaz Janbakhsh Can you once try the below Spec [{ "operation": "shift", "spec": { "payloadMetaData": { "rawPayload": "rawPayload", "fcount": "fcount", "fport": "fport", "applicationMetaData": { "id": { "entityType": "entityType", "id": "id" }, "customerId": { "entityType": "entityTypeCustomer", "id": "idCustomer" } }, "gatewayMetaDataList": { "*": { //* means if you are having more than 1 message in array then it results array of mac addresses in it. "mac": "mac" } }, "deviceMetaData": { "id": { "entityType": "entityTypeDevice", "id": "DeviceId" }, "name": "DeviceName", "deviceClass": "deviceClass", "deviceEUI": "deviceEUI", "appEUI": "appEUI" } }, "payload": "payload" } }] input:- { "payloadMetaData": { "applicationMetaData": { "id": { "entityType": "APPLICATION", "id": "7d11abf0-aa82-11e7-afd4-63e5a28ad7fa" }, "customerId": { "entityType": "CUSTOMER", "id": "a84ab1a0-aa81-11e7-afd4-63e5a28ad7fa" }, "subCustomerId": null, "name": "TekTelic-Industrial" }, "gatewayMetaDataList": [ { "id": { "entityType": "GATEWAY", "id": "e5ca6840-aa81-11e7-afd4-63e5a28ad7fa" }, "name": "ColinsGateway", "mac": "647FDAFFFE00417C", "latitude": 51.04697912502887, "longitude": -114.06121730804443, "altitude": null, "rxInfo": { "channel": 2, "codeRate": "4/5", "crcStatus": 1, "dataRate": { "modulation": "LORA", "spreadFactor": 7, "bandwidth": 125 }, "frequency": 902700000, "loRaSNR": 6.8, "mac": "647fdafffe00417c", "rfChain": 0, "rssi": -51, "size": 21, "time": "2017-11-01T21:51:27Z", "timestamp": 1400148651, "rsig": null } } ], "deviceMetaData": { "id": { "entityType": "DEVICE", "id": "61348ab0-ada2-11e7-ac06-63e5a28ad7fa" }, "name": "TekTelic-Industrial-0017", "deviceClass": "CLASS_A", "deviceEUI": "647FDA0000000155", "appEUI": "647FDA8000000155" }, "fcount": 3706, "fport": 10 } } Output:- { "fcount" : 3706, "fport" : 10, "entityType" : "APPLICATION", "id" : "7d11abf0-aa82-11e7-afd4-63e5a28ad7fa", "entityTypeCustomer" : "CUSTOMER", "idCustomer" : "a84ab1a0-aa81-11e7-afd4-63e5a28ad7fa", "mac" : "647FDAFFFE00417C", "entityTypeDevice" : "DEVICE", "DeviceId" : "61348ab0-ada2-11e7-ac06-63e5a28ad7fa", "DeviceName" : "TekTelic-Industrial-0017", "deviceClass" : "CLASS_A", "deviceEUI" : "647FDA0000000155", "appEUI" : "647FDA8000000155" } in addition if you want to flatten out all the elements of array then extract all the json message contents by using eval json path processor with Destination as flowfile-attribute, then use split json processor with JsonPath Expression as $.gatewayMetaDataList then use jolt transform with below spec [{ "operation": "shift", "spec": { "*": { "*": "id-&", "id": { "*": "&" }, "rxInfo": { "*": "rxInfo-&", "dataRate": { "*": "dataRate-&" } } } } }] Right now you are doing flattening out json array elements then use attributes to json processor and keep all the attributes that you have extracted, now this processor will create new json message which is flattened out with each array elements in it. (or) if you want to add all array elements to respective keys then use below spec [{ "operation": "shift", "spec": { "payloadMetaData": { "gatewayMetaDataList": { "*": { "*": "id-&", "id": { "*": "&" }, "rxInfo": { "*": "rxInfo-&", "dataRate": { "*": "dataRate-&" } } } } } } } ... View more

@dhieru singh Can you once check your replace text processor configurations if the Replacement Strategy property as Prepend that means you are adding new line at starting of the file. As that new line won't show in NiFi UI but when you do cat on the file then only it shows the new line. ... View more