Member since
12-03-2017
148
Posts
25
Kudos Received
11
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
1212 | 11-03-2023 12:17 AM | |
2958 | 12-12-2022 09:16 PM | |
1124 | 07-14-2022 03:25 AM | |
1793 | 07-28-2021 04:42 AM | |
2079 | 06-23-2020 10:08 PM |
10-30-2018
11:27 AM
Hello All, I have requirement of caching some information in nifi and which should be used for every flow file validation (more than 1000 per second). I am thinking to store information in Cassandra table and update that to Nifi cache every 24 hours or so.. (Nifi cache update using PutDistributedMapCache processor) But problem is how to hold/re-load cache value when Nifi restarts due to various reasons.. I also want to understand where these cached message will be stored actually in multinode cluster nifi. Thanks in advance. Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
07-23-2018
09:20 AM
@Matt Burgess , @Shu- Your help would be appreciated. Thanks
... View more
07-23-2018
09:19 AM
Hello, I have a json input for which I need to add one key-value("correlationId": "123456789") under 'tripSummaryMetaData' section dynamically. I tried with below spec and its working fine, but the issues is when 'tripSummaryMetaData' is not available the it is creating new 'tripSummaryMetaData' and adding my attribute. When 'tripSummaryMetaData' is not available the it should not add my new attribute, is that possible in JOLT? Thanks in advance. Input: {
"tripSummaryMetaData": {
"orgId": "demo",
"message_type_id": "MB-CD-MSG0001"
},
"eventSummary": {
"tripId": "2018072204cc7851d3e84f7ea8a1155c95f65b06",
"referenceData": "050bfa62-430a-45a5-b7a6-78d688d45167",
"gpsData": [{
"gpsTimeReceived": 1532319157338,
"gpsTime": "2018-07-22T21:12:38-07:00"
}]
}
} Output: {
"tripSummaryMetaData" : {
"orgId" : "demo",
"message_type_id" : "MB-CD-MSG0001",
"correlationId" : "123456789"
},
"eventSummary" : {
"tripId" : "2018072204cc7851d3e84f7ea8a1155c95f65b06",
"referenceData" : "050bfa62-430a-45a5-b7a6-78d688d45167",
"gpsData" : [ {
"gpsTimeReceived" : 1532319157338,
"gpsTime" : "2018-07-22T21:12:38-07:00"
} ]
}
} Spec : [{
"operation": "default",
"spec": {
"tripSummaryMetaData": {
"correlationId": "123456789"
}
}
}] Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
05-03-2018
09:33 AM
Yes, its possible. Alter table <table_name> set tblproperties ('transactional'='true');
... View more
05-03-2018
09:31 AM
Thanks for the nice explanation Ned. When I have created the flow, NiFi processor is able to store data to the table but its very slow in the cluster environment. I am able to push only 2-3 flowfile/records (few kb in size) per second. I tried with more thread counts as well, but no luck. Any suggestion will be much appreciated.
... View more
04-14-2018
06:00 PM
Thanks for replies, Issue was due to concurrent thread configuration. When we increase the max allowed thread count in controller settings, issue solved. We stopped all flows, increased the count, restarted the node and then flows.
... View more
04-14-2018
05:56 PM
hello, Facing an issue with 'InvokeHTTP' NiFi processor. Have configured below URL for GET operation - https://******/**/users/${userId} When I run the processor some time it will get proper response from url. But if I restart the processor, suddenly it will start telling 'URL not found' error and route the flow file to 'No Retry' relationship. If I try to restart the processor 2-3 times then once it will start responding correctly. Please suggest me to resolve the issue. screen shots of config - Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
03-29-2018
04:26 PM
Hi, I have a kafka consumer, that is consuming messages after that flowfiles are not moving at all. Suddenly this issue started. many messages are stuck at diff connection/queue in btw processors even though processor running it is not getting processing dataflow-struck.png Thanks, Mahendra
... View more
Labels:
- Labels:
-
Apache NiFi
03-19-2018
11:09 AM
I am enriching my request payload with the payload attribute 'http.headers.orgId' and its working fine. SPEC: [{
"operation": "default", "spec": { "*": {
"orgId": "${http.headers.orgId}"
} } }] INPUT : {
"drivewiseDeviceData": {
"imei": "sampleimei",
"tripId": "3d9df81a-1828-40eb-940e-d2434eb5f50f",
"sumVer": null,
"payloadType": "Binary",
"packetInfo": "",
"payloadData": "g1bjSWX",
"recordinfo": null,
"sequenceNum": null,
"timeStamp": "2017-05-26T12:07:29.968Z" }
} OUTPUT: { "drivewiseDeviceData": { "imei": "sampleimei", "tripId": "3d9df81a-1828-40eb-940e-d2434eb5f50f", "sumVer": null, "payloadType": "Binary", "packetInfo": "", "payloadData": "g1bjSWX", "recordinfo": null, "sequenceNum": null, "timeStamp": "2017-05-26T12:07:29.968Z", "orgId": "Sample Org Name Here" } } ......................................................................................................................................... Now scenarios is I may get header attribute as 'http.headers.OrgID' or 'http.headers.orgId', So is there any way to handle both scenarios in my JOLT spec? If 'http.headers.orgId' value is null/empty/attribute-not-present then I should use 'http.headers.OrgID' for enrich.
... View more
Labels:
- Labels:
-
Apache NiFi
02-23-2018
10:47 AM
Thanks for the quick reply @Matt Clarke 🙂 One more query, I may have N number of files, so how can I find out the number of files in directory and ensure that all files reached Mergecontent processor before I proceed further to create trigger file and FTP ?
... View more
- « Previous
- Next »