Member since
08-16-2017
39
Posts
3
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3841 | 07-12-2018 05:49 AM |
07-12-2018
05:49 AM
@Matt Burgess So it is needed to replace ESC (\x1b) with delimiter: '|' using UpdateRecord processor. Could you please help to configure this processor to impement this replacement in flow file records!
... View more
07-12-2018
04:25 AM
Hi, @Matt Burgess! My sample formatted JSON file content (in original flow file JSON objects are separated by new line) {
"schemaNameSpace": "CPMCDM.com.bis.bss.cpm.event.schema",
"schemaName": "CpmCustomerChangeEvent",
"schemaVersion": "5.1.1",
"eventHeader": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.EventHeader": {
"eventCreationTime": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.Time": {
"timestamp": {
"string": "2018-04-03T23:08:38.652+03:00"
},
"timeZoneType": {
"string": "SYSTEM_TIME_ZONE"
},
"zoneName": {
"string": "Europe/Kiev"
}
}
},
"cpmInstanceHost": {
"string": "env6-cpm1.dbss.bis.ua"
},
"recordUniqueId": {
"string": "77ECA557BFA74C98B2792222C9C72CED"
}
}
},
"customerInformation": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.CustomerInformation": {
"customerId": {
"string": "5A834B1C27DE4FAF9F038B370AA3DDA4"
},
"partyId": null
}
},
"genericInterfaceParameters": null,
"requestInfo": null,
"partyChangeResult": null
}{
"schemaNameSpace": "com.bis.bss.edm.eventDataEnrichment.schema",
"schemaName": "EventDataEnrichment",
"schemaVersion": "1.0.0",
"enrichedData": [
]
}
{
"schemaNameSpace": "CPMCDM.com.bis.bss.cpm.event.schema",
"schemaName": "CpmCustomerChangeEvent",
"schemaVersion": "5.1.1",
"eventHeader": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.EventHeader": {
"eventCreationTime": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.Time": {
"timestamp": {
"string": "2018-04-03T23:08:39.652+03:10"
},
"timeZoneType": {
"string": "SYSTEM_TIME_ZONE"
},
"zoneName": {
"string": "Europe/Kiev"
}
}
},
"cpmInstanceHost": {
"string": "env6-cpm1.dbss.bis.ua"
},
"recordUniqueId": {
"string": "72DEA157BFA74C98B2792222C0C11CBE"
}
}
},
"customerInformation": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.CustomerInformation": {
"customerId": {
"string": "1E2234B1C27DE4FAF9F038B370AA3DBE4"
},
"partyId": null
}
},
"genericInterfaceParameters": null,
"requestInfo": null,
"partyChangeResult": null
}
{
"schemaNameSpace": "CPMCDM.com.bis.bss.cpm.event.schema",
"schemaName": "CpmCustomerChangeEvent",
"schemaVersion": "5.1.1",
"eventHeader": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.EventHeader": {
"eventCreationTime": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.Time": {
"timestamp": {
"string": "2018-04-03T23:08:40.652+02:20"
},
"timeZoneType": {
"string": "SYSTEM_TIME_ZONE"
},
"zoneName": {
"string": "Europe/Kiev"
}
}
},
"cpmInstanceHost": {
"string": "env6-cpm1.dbss.bis.ua"
},
"recordUniqueId": {
"string": "55CBA557BFA74C98B2792222C9A11CDE"
}
}
},
"customerInformation": {
"CPMCDM.com.bis.bss.cpm.event.schema.cpmCustomerChangeEvent.CustomerInformation": {
"customerId": {
"string": "1E244B1C27DE4FAF9F038B370AA3DDD5"
},
"partyId": null
}
},
"genericInterfaceParameters": null,
"requestInfo": null,
"partyChangeResult": null
}{
"schemaNameSpace": "com.bis.bss.edm.eventDataEnrichment.schema",
"schemaName": "EventDataEnrichment",
"schemaVersion": "1.0.0",
"enrichedData": [
]
}
<br> Important thing to notice here is that some of the JSON objects in the flow file contain extention seperated with ESC (\x1b). {
"schemaNameSpace": "com.bis.bss.edm.eventDataEnrichment.schema",
"schemaName": "EventDataEnrichment",
"schemaVersion": "1.0.0",
"enrichedData": [
]
}<br> Schema looks like below: {mainJSON}{extentionJSON} {mainJSON} {mainJSON}{extentionJSON}
..... In the output I would like to have the following format: mainJSON | extentionJSON mainJSON | mainJSON | extentionJSON etc... Thank you!
... View more
07-11-2018
12:12 PM
Hi, dear Experts! Could you please help with following issue: I have a processor ExtractText that processes JSON flow file and creates two attributes with large JSON text. mainJSON = ^.*(?=\x1b)|^((?!\x1b).)*$) enrichmentJSON = ((?<=\x1b).*) As an output there were created 3 attributes for mainJSON and enrichmentJSON mainJSON: mainJSON, mainJSON.0, mainJSON.1 --- each containing the same portion of expected result. Isn't it possible to store large text value in an attribute? Is there another way to store and pass large text value as an attribute? As a next step I wanted to combine these two attributes with other attributes in ReplaceText processor and put them into hive table as separate columns of one row. Thanks in advance!
... View more
Labels:
- Labels:
-
Apache NiFi
07-06-2018
05:06 AM
1 Kudo
Thank you very much @Shu and @Matt Burgess !! @Shu, Split Content processor really does great thing! I have managed to split the nested JSON objects into separate objects using Text as Byte Sequence Format and }{ as Byte Sequence. Thank you very much for your advise!! with all the best wishes, Gulshan
... View more
07-05-2018
10:54 AM
Hi! My Flow file contains more then one JSON objects {
Nested JSON object 1
}{
Nested JSON object 2
}{
Nested JSON object N
} Can you please help to split JSON flow file into separate JSON objects using Nifi processors. What should be the JsonPath Expression in SplitJSON processor? thanks in advance!
... View more
Labels:
- Labels:
-
Apache NiFi
05-16-2018
09:59 AM
Oh, great! Thank you very much Sandeep!!
... View more
05-16-2018
09:14 AM
Hi! Can you please help to permanently remove a disk from all the datanodes in working cluster. DataNode directories /grid/0,/grid/2,/grid/3,/grid/4,/grid/5,/grid/6,/grid/7,/grid/8,/grid/9,/grid/10,/grid/11,/grid/12,/grid/14,/grid/15 I would like to remove /grid/15. Thanks in advance!! with regards, Gulshan
... View more
Labels:
04-23-2018
05:38 AM
Hi, dear Experts! Could you please help with the following issue? Ranger Audit Access tab is failing with the following error: Error Error running solr query, please check solr configs. Could not find a healthy node to handle the request. in /hadoop/log/ambari-infra-solr/solr.log : ERROR [c:audit_logs s:shard0 r:core_node2 x:audit_logs_shard0_replica1] org.apache.solr.common.SolrException (SolrException.java:148) - org.apache.solr.common.SolrException: Invalid Date String:'2017-09-05T02017 with regards, Gulshan
... View more
Labels:
- Labels:
-
Apache Solr
02-15-2018
06:38 AM
Hi, @Jay Kumar SenSharma! Thank you, for the link you have provided. My problem was also because of "quasi-installed state where its repo is listed as 'installed' even though it is not" I could resolve the problem by removing the package that was installed(partially or somewhat oddly) and relaunching the installation of packages from ambari again Thanks!!
... View more
02-15-2018
06:17 AM
@Jay Kumar SenSharma Hi, Jay Kumar! But I'm using Apache Ambari Version 2.6.1.0. Before starting upgrade of HDP, I 've upgraded Ambari to 2.6.1.0. From the link you gave I couldn't find the workaround.
... View more
- « Previous
-
- 1
- 2
- Next »