Hi,   I was trying to use nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository. To do so i have to comment java.arg.13=-XX:+UseG1GC to get rid of any performance issues as per  https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999 However i find it difficult on how to do it via Cloudera Manager.  Please help. ... View more

Hi,   When i try to restart the cluster after installing a workflow, I end up getting ERROR org.apache.nifi.controller.StandardFlowService: Failed to load flow from cluster due to: org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster due to: org.apache.nifi.encrypt.EncryptionException: org.apache.nifi.encrypt.EncryptionException: Could not decrypt sensitive value (nifi-app.log). Not sure what would be the cause for this error.  ... View more

Hi, I have the following JSON Record in my incoming Flowfile to QueryRecord Processor and wanted specific fields from the content. I want Index, Sentiment and max(SentimentScore) {     "ResultList": [         {             "Index": 0,              "Sentiment": "NEUTRAL",              "SentimentScore": {                 "Mixed": 0.014603961259126663,                  "Positive": 0.2183697521686554,                  "Neutral": 0.7095515727996826,                  "Negative": 0.05747472867369652             }         },          {             "Index": 1,              "Sentiment": "NEGATIVE",              "SentimentScore": {                 "Mixed": 0.20378299057483673,                  "Positive": 0.11562751978635788,                  "Neutral": 0.10683445632457733,                  "Negative": 0.5737550258636475             }         },          {             "Index": 2,              "Sentiment": "NEGATIVE",              "SentimentScore": {                 "Mixed": 0.007578125223517418,                  "Positive": 0.004836051259189844,                  "Neutral": 0.01562119647860527,                  "Negative": 0.9719645380973816             }         },          {             "Index": 3,              "Sentiment": "POSITIVE",              "SentimentScore": {                 "Mixed": 0.008649133145809174,                  "Positive": 0.8763676881790161,                  "Neutral": 0.11021242290735245,                  "Negative": 0.004770802799612284             }         },          {             "Index": 4,              "Sentiment": "NEUTRAL",              "SentimentScore": {                 "Mixed": 0.0017022271640598774,                  "Positive": 0.03283606469631195,                  "Neutral": 0.9602330327033997,                  "Negative": 0.005228717811405659             }         },          {             "Index": 5,              "Sentiment": "NEGATIVE",              "SentimentScore": {                 "Mixed": 0.02237289398908615,                  "Positive": 0.2910439670085907,                  "Neutral": 0.3056293725967407,                  "Negative": 0.38095375895500183             }         },          {             "Index": 6,              "Sentiment": "POSITIVE",              "SentimentScore": {                 "Mixed": 0.00902993232011795,                  "Positive": 0.976735532283783,                  "Neutral": 0.011653305031359196,                  "Negative": 0.002581255976110697             }         },          {             "Index": 7,              "Sentiment": "NEGATIVE",              "SentimentScore": {                 "Mixed": 0.011669524013996124,                  "Positive": 0.0006034726393409073,                  "Neutral": 0.17051522433757782,                  "Negative": 0.8172117471694946             }         },          {             "Index": 8,              "Sentiment": "NEGATIVE",              "SentimentScore": {                 "Mixed": 0.010518478229641914,                  "Positive": 0.002629985800012946,                  "Neutral": 0.01846659556031227,                  "Negative": 0.9683849811553955             }         },          {             "Index": 9,              "Sentiment": "NEUTRAL",              "SentimentScore": {                 "Mixed": 0.009224030189216137,                  "Positive": 0.04947839304804802,                  "Neutral": 0.7379242181777954,                  "Negative": 0.20337335765361786             }         }     ],      "ErrorList": [] } Am using the Following Schema in the JsonTreeReader, {   "type" : "record",   "name" : "SentimentOutput",   "fields" : [ {     "name" : "ResultList",     "type" : {       "type" : "array",       "items" : {         "type" : "record",         "name" : "ResultList",         "fields" : [ {           "name" : "Index",           "type" : "int",           "doc" : "Type inferred from '0'"         }, {           "name" : "Sentiment",           "type" : "string",           "doc" : "Type inferred from '\"NEUTRAL\"'"         }, {           "name" : "SentimentScore",           "type" : {             "type" : "record",             "name" : "ResultList",             "namespace" : "SentimentScore",             "fields" : [ {               "name" : "Mixed",               "type" : "double",               "doc" : "Type inferred from '0.014603961259126663'"             }, {               "name" : "Positive",               "type" : "double",               "doc" : "Type inferred from '0.2183697521686554'"             }, {               "name" : "Neutral",               "type" : "double",               "doc" : "Type inferred from '0.7095515727996826'"             }, {               "name" : "Negative",               "type" : "double",               "doc" : "Type inferred from '0.05747472867369652'"             } ]           },           "doc" : "Type inferred from '{\"Mixed\":0.014603961259126663,\"Positive\":0.2183697521686554,\"Neutral\":0.7095515727996826,\"Negative\":0.05747472867369652}'"         } ]       }     },     "doc" : "Type inferred from '[{\"Index\":0,\"Sentiment\":\"NEUTRAL\",\"SentimentScore\":{\"Mixed\":0.014603961259126663,\"Positive\":0.2183697521686554,\"Neutral\":0.7095515727996826,\"Negative\":0.05747472867369652}},{\"Index\":1,\"Sentiment\":\"NEGATIVE\",\"SentimentScore\":{\"Mixed\":0.20378299057483673,\"Positive\":0.11562751978635788,\"Neutral\":0.10683445632457733,\"Negative\":0.5737550258636475}},{\"Index\":2,\"Sentiment\":\"NEGATIVE\",\"SentimentScore\":{\"Mixed\":0.007578125223517418,\"Positive\":0.004836051259189844,\"Neutral\":0.01562119647860527,\"Negative\":0.9719645380973816}},{\"Index\":3,\"Sentiment\":\"POSITIVE\",\"SentimentScore\":{\"Mixed\":0.008649133145809174,\"Positive\":0.8763676881790161,\"Neutral\":0.11021242290735245,\"Negative\":0.004770802799612284}},{\"Index\":4,\"Sentiment\":\"NEUTRAL\",\"SentimentScore\":{\"Mixed\":0.0017022271640598774,\"Positive\":0.03283606469631195,\"Neutral\":0.9602330327033997,\"Negative\":0.005228717811405659}},{\"Index\":5,\"Sentiment\":\"NEGATIVE\",\"SentimentScore\":{\"Mixed\":0.02237289398908615,\"Positive\":0.2910439670085907,\"Neutral\":0.3056293725967407,\"Negative\":0.38095375895500183}},{\"Index\":6,\"Sentiment\":\"POSITIVE\",\"SentimentScore\":{\"Mixed\":0.00902993232011795,\"Positive\":0.976735532283783,\"Neutral\":0.011653305031359196,\"Negative\":0.002581255976110697}},{\"Index\":7,\"Sentiment\":\"NEGATIVE\",\"SentimentScore\":{\"Mixed\":0.011669524013996124,\"Positive\":6.034726393409073E-4,\"Neutral\":0.17051522433757782,\"Negative\":0.8172117471694946}},{\"Index\":8,\"Sentiment\":\"NEGATIVE\",\"SentimentScore\":{\"Mixed\":0.010518478229641914,\"Positive\":0.002629985800012946,\"Neutral\":0.01846659556031227,\"Negative\":0.9683849811553955}},{\"Index\":9,\"Sentiment\":\"NEUTRAL\",\"SentimentScore\":{\"Mixed\":0.009224030189216137,\"Positive\":0.04947839304804802,\"Neutral\":0.7379242181777954,\"Negative\":0.20337335765361786}}]'"   }, {     "name" : "ErrorList",     "type" : {       "type" : "array",       "items" : "null"     },     "doc" : "Type inferred from '[]'"   } ] } And the Query i use is; select (ResultList, '/Index') as "index", (ResultList, '/Sentiment') as "sentiment", (ResultList, max('/SentimentScore')) as "score" from FLOWFILE While Executing i get the following Error, Can someone help me achieving this use case? Tagging : @Shu,@Matt Burgess,@Matt Clarke ... View more

Hi, I have a JSON record like given below, {     "entitlement": {         "packageCodes": ["A", "B", "C"],         "accountToken": "accountTokenXYZ",         "creationDate": "2018-08-03 11:58:54",         "lastModifiedDate": "2019-04-23 11:58:03",         "status": "ACTV",         "expirationDate": null     },     "op_type": "U",     "op_ts": "2019-05-08 03:41:07.007764",     "table": "tablename" } Expected Output: PackageCodes,accountToken,creationDate,lastModifiedDate,status,expirationDate,op_type,op_ts,table "A,B,C",accountTokenXYZ,2018-08-03 11:58:54,2019-04-23 11:58:03,ACTV,,U,2019-05-08 03:41:07.007764,tablename How this can be achieved using NiFi? Please help. Thanks, Bala. ... View more

Hi, am running a select query on the flowfile content using queryRecord processor. However i want the results in single row rather than multiple rows. I tried using Group_concat but queryRecord supports only ANSI SQL standards. Group_Concat is not ANSI SQL command. Any suggestions to get this done? ... View more

Hi, Am trying to run the below aws comprehend CLI command using the ExecuteProcess. aws comprehend detect-sentiment \ --region us-west-2 \ --language-code "en" \ --text "It is raining today in Seattle." However End up getting the following error, Failed to create process due to Cannot run program "aws comprehend detect-sentiment --region us-west-2 --language-code "en" --text "It is raining today in Seattle."": error=13, Permission denied: java.io.IOException: Cannot run program "aws comprehend detect-sentiment --region us-west-2 --language-code "en" --text "It is raining today in Seattle."": error=13, Permission denied Execute Process Configs: When i run the command on the NiFi Nodes, Its was working fine. The issue is when trying to execute the same command using the execute processor. Please help. ... View more

Hi, I have the incoming flowfiles with the json content which have multiple event types. I need to filter only certain events based on two of the JSON key/value. Sample input data: {"eventType":"Mobile","timestamp":1551280374552,"name":"ERROR"} {"eventType":"Mobile","timestamp":1551280374552,"name":"APP_START"} {"eventType":"Immobile","timestamp":1551280374552,"name":"ERROR"} {"eventType":"Immobile","timestamp":1551280374552,"name":"ERROR"} {"eventType":"Mobile","timestamp":1551280374552,"name":"PLAYBACK_ERROR"} {"eventType":"Mobile","timestamp":1551280374552,"name":"Other"} {"eventType":"MobileCrash","timestamp":1551280374552,"name":"ERROR"} Filter Condition to be applied : Flowfile content having If eventtype = "Mobile" and name = (ERROR or APP_START or PLAYBACK_ERROR) or eventtype = "MobileCrash" Also, after the filter i have to rename the json key as, eventType = vendorEventType timestamp = currentTimestamp name = someName Desired Output: {"vendorEventType":"Mobile","currentTimestamp":1551280374552,"someName":"ERROR"} {"vendorEventType":"Mobile","currentTimestamp":1551280374552,"someName":"APP_START"}{"vendorEventType":"Mobile","currentTimestamp":1551280374552,"someName":"PLAYBACK_ERROR"}{"vendorEventType":"MobileCrash","currentTimestamp":1551280374552,"someName":"ERROR"} Please help. ... View more

Hi, Per my workflow, Im trying to merge the records into one file before writing it to the HDFS using MergeContent --> PutHDFS. Unfortunately, we can only have one client writing/appending to the same file in HDFS at a time. Is there really a way to address this issue? Note : I cannot use primary only for all the processors in the flow just because of PutHDFS. Thanks, Bala. ... View more

Hi, I have a use case to add a new row from the value for the last column in the CSV record.Can someone help me doing this using the NiFi processors? Thanks, Bala. ... View more

Hi, I have a use case to rename 3 of the columns in an incoming JSON file and store it into HDFS. I was using the JOLTTransformJSON processor to achieve this. The processor is working fine renaming the columns as expected. However the issue is it transforms only the first record and resulting one only record written to HDFS instead of few thousands present in the input. The spec i used is, [ { "operation": "shift", "spec": { "accountId": "vendorAccountId", "masterTransactionId": "errorMasterTransactionId", "requestUrl": "errorRequestUrl", "*": { "@": "&" } } } ] . Please help. ... View more

Hi, I Have configured the NiFi Workflow to copy data from Kafka cluster 1 to Kafka cluster 2. As a result, it was taking around 11 mins to copy 1 Million messages from source cluster to destination. Can anyone help me improving the performance which can match the mirrormaker time taken for transferring similar amount of data. Source Cluster : 3 Brokers, The source topic with 3 partitions Destination Cluster : 3 Brokers, The Destination topic with 3 partitions NiFi Cluster : 3 Nodes Attached the workflow, consumerKafka Configs and PublishKafka Configs which i used. ... View more