Member since
10-30-2017
32
Posts
5
Kudos Received
0
Solutions
02-10-2021
11:05 PM
Thanks @MattWho. Got it !!
... View more
02-09-2021
12:48 PM
Hello @tjangid, Thanks for your reply. When i search for "ignore" under NiFi Configurations i was not getting any results. I also i checked in advanced category and i could find only " NiFi Node Advanced Configuration Snippet (Safety Valve) for staging/bootstrap.conf.xml" not any ignored section. Please let me know if am missing anything here. Thanks, Bala
... View more
09-15-2020
09:38 AM
Hi, I was trying to use nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository. To do so i have to comment java.arg.13=-XX:+UseG1GC to get rid of any performance issues as per https://community.cloudera.com/t5/Community-Articles/HDF-CFM-NIFI-Best-practices-for-setting-up-a-high/ta-p/244999 However i find it difficult on how to do it via Cloudera Manager. Please help.
... View more
Labels:
07-29-2020
09:54 PM
Hi, When i try to restart the cluster after installing a workflow, I end up getting ERROR org.apache.nifi.controller.StandardFlowService: Failed to load flow from cluster due to: org.apache.nifi.cluster.ConnectionException: Failed to connect node to cluster due to: org.apache.nifi.encrypt.EncryptionException: org.apache.nifi.encrypt.EncryptionException: Could not decrypt sensitive value (nifi-app.log). Not sure what would be the cause for this error.
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
03-03-2020
10:57 AM
Thanks @venkatsambath. It worked.
... View more
03-02-2020
04:39 PM
Hi,
Which property in Cloudera manager should i update to toggle between the cluster and standalone mode for the nifi cluster? Also, the location of the nifi.properties in the CFM.... Its not under /var/lib/nifi/
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera Manager
06-21-2019
11:07 PM
1 Kudo
Hi, I have the following JSON Record in my incoming Flowfile to QueryRecord Processor and wanted specific fields from the content. I want Index, Sentiment and max(SentimentScore) {
"ResultList": [
{
"Index": 0,
"Sentiment": "NEUTRAL",
"SentimentScore": {
"Mixed": 0.014603961259126663,
"Positive": 0.2183697521686554,
"Neutral": 0.7095515727996826,
"Negative": 0.05747472867369652
}
},
{
"Index": 1,
"Sentiment": "NEGATIVE",
"SentimentScore": {
"Mixed": 0.20378299057483673,
"Positive": 0.11562751978635788,
"Neutral": 0.10683445632457733,
"Negative": 0.5737550258636475
}
},
{
"Index": 2,
"Sentiment": "NEGATIVE",
"SentimentScore": {
"Mixed": 0.007578125223517418,
"Positive": 0.004836051259189844,
"Neutral": 0.01562119647860527,
"Negative": 0.9719645380973816
}
},
{
"Index": 3,
"Sentiment": "POSITIVE",
"SentimentScore": {
"Mixed": 0.008649133145809174,
"Positive": 0.8763676881790161,
"Neutral": 0.11021242290735245,
"Negative": 0.004770802799612284
}
},
{
"Index": 4,
"Sentiment": "NEUTRAL",
"SentimentScore": {
"Mixed": 0.0017022271640598774,
"Positive": 0.03283606469631195,
"Neutral": 0.9602330327033997,
"Negative": 0.005228717811405659
}
},
{
"Index": 5,
"Sentiment": "NEGATIVE",
"SentimentScore": {
"Mixed": 0.02237289398908615,
"Positive": 0.2910439670085907,
"Neutral": 0.3056293725967407,
"Negative": 0.38095375895500183
}
},
{
"Index": 6,
"Sentiment": "POSITIVE",
"SentimentScore": {
"Mixed": 0.00902993232011795,
"Positive": 0.976735532283783,
"Neutral": 0.011653305031359196,
"Negative": 0.002581255976110697
}
},
{
"Index": 7,
"Sentiment": "NEGATIVE",
"SentimentScore": {
"Mixed": 0.011669524013996124,
"Positive": 0.0006034726393409073,
"Neutral": 0.17051522433757782,
"Negative": 0.8172117471694946
}
},
{
"Index": 8,
"Sentiment": "NEGATIVE",
"SentimentScore": {
"Mixed": 0.010518478229641914,
"Positive": 0.002629985800012946,
"Neutral": 0.01846659556031227,
"Negative": 0.9683849811553955
}
},
{
"Index": 9,
"Sentiment": "NEUTRAL",
"SentimentScore": {
"Mixed": 0.009224030189216137,
"Positive": 0.04947839304804802,
"Neutral": 0.7379242181777954,
"Negative": 0.20337335765361786
}
}
],
"ErrorList": []
} Am using the Following Schema in the JsonTreeReader, {
"type" : "record",
"name" : "SentimentOutput",
"fields" : [ {
"name" : "ResultList",
"type" : {
"type" : "array",
"items" : {
"type" : "record",
"name" : "ResultList",
"fields" : [ {
"name" : "Index",
"type" : "int",
"doc" : "Type inferred from '0'"
}, {
"name" : "Sentiment",
"type" : "string",
"doc" : "Type inferred from '\"NEUTRAL\"'"
}, {
"name" : "SentimentScore",
"type" : {
"type" : "record",
"name" : "ResultList",
"namespace" : "SentimentScore",
"fields" : [ {
"name" : "Mixed",
"type" : "double",
"doc" : "Type inferred from '0.014603961259126663'"
}, {
"name" : "Positive",
"type" : "double",
"doc" : "Type inferred from '0.2183697521686554'"
}, {
"name" : "Neutral",
"type" : "double",
"doc" : "Type inferred from '0.7095515727996826'"
}, {
"name" : "Negative",
"type" : "double",
"doc" : "Type inferred from '0.05747472867369652'"
} ]
},
"doc" : "Type inferred from '{\"Mixed\":0.014603961259126663,\"Positive\":0.2183697521686554,\"Neutral\":0.7095515727996826,\"Negative\":0.05747472867369652}'"
} ]
}
},
"doc" : "Type inferred from '[{\"Index\":0,\"Sentiment\":\"NEUTRAL\",\"SentimentScore\":{\"Mixed\":0.014603961259126663,\"Positive\":0.2183697521686554,\"Neutral\":0.7095515727996826,\"Negative\":0.05747472867369652}},{\"Index\":1,\"Sentiment\":\"NEGATIVE\",\"SentimentScore\":{\"Mixed\":0.20378299057483673,\"Positive\":0.11562751978635788,\"Neutral\":0.10683445632457733,\"Negative\":0.5737550258636475}},{\"Index\":2,\"Sentiment\":\"NEGATIVE\",\"SentimentScore\":{\"Mixed\":0.007578125223517418,\"Positive\":0.004836051259189844,\"Neutral\":0.01562119647860527,\"Negative\":0.9719645380973816}},{\"Index\":3,\"Sentiment\":\"POSITIVE\",\"SentimentScore\":{\"Mixed\":0.008649133145809174,\"Positive\":0.8763676881790161,\"Neutral\":0.11021242290735245,\"Negative\":0.004770802799612284}},{\"Index\":4,\"Sentiment\":\"NEUTRAL\",\"SentimentScore\":{\"Mixed\":0.0017022271640598774,\"Positive\":0.03283606469631195,\"Neutral\":0.9602330327033997,\"Negative\":0.005228717811405659}},{\"Index\":5,\"Sentiment\":\"NEGATIVE\",\"SentimentScore\":{\"Mixed\":0.02237289398908615,\"Positive\":0.2910439670085907,\"Neutral\":0.3056293725967407,\"Negative\":0.38095375895500183}},{\"Index\":6,\"Sentiment\":\"POSITIVE\",\"SentimentScore\":{\"Mixed\":0.00902993232011795,\"Positive\":0.976735532283783,\"Neutral\":0.011653305031359196,\"Negative\":0.002581255976110697}},{\"Index\":7,\"Sentiment\":\"NEGATIVE\",\"SentimentScore\":{\"Mixed\":0.011669524013996124,\"Positive\":6.034726393409073E-4,\"Neutral\":0.17051522433757782,\"Negative\":0.8172117471694946}},{\"Index\":8,\"Sentiment\":\"NEGATIVE\",\"SentimentScore\":{\"Mixed\":0.010518478229641914,\"Positive\":0.002629985800012946,\"Neutral\":0.01846659556031227,\"Negative\":0.9683849811553955}},{\"Index\":9,\"Sentiment\":\"NEUTRAL\",\"SentimentScore\":{\"Mixed\":0.009224030189216137,\"Positive\":0.04947839304804802,\"Neutral\":0.7379242181777954,\"Negative\":0.20337335765361786}}]'"
}, {
"name" : "ErrorList",
"type" : {
"type" : "array",
"items" : "null"
},
"doc" : "Type inferred from '[]'"
} ]
} And the Query i use is; select (ResultList, '/Index') as "index", (ResultList, '/Sentiment') as "sentiment", (ResultList, max('/SentimentScore')) as "score" from FLOWFILE While Executing i get the following Error, Can someone help me achieving this use case? Tagging : @Shu,@Matt Burgess,@Matt Clarke
... View more
Labels:
- Labels:
-
Apache NiFi
05-14-2019
06:22 PM
Hi, am running a select query on the flowfile content using queryRecord processor. However i want the results in single row rather than multiple rows. I tried using Group_concat but queryRecord supports only ANSI SQL standards. Group_Concat is not ANSI SQL command. Any suggestions to get this done?
... View more
Labels:
- Labels:
-
Apache NiFi
04-29-2019
08:48 PM
@Shu I have the read/write permissions to the config and creds files for NiFi user while setting the AWS CLI on the nifi nodes...
... View more
04-26-2019
08:18 PM
1 Kudo
Hi, Can someone please provide some directions on this issue? Will be really helpful.
... View more
04-23-2019
11:56 PM
Hi, Am trying to run the below aws comprehend CLI command using the ExecuteProcess. aws comprehend detect-sentiment \ --region us-west-2 \ --language-code "en" \ --text "It is raining today in Seattle." However End up getting the following error, Failed to create process due to Cannot run program "aws comprehend detect-sentiment --region us-west-2 --language-code "en" --text "It is raining today in Seattle."": error=13, Permission denied: java.io.IOException: Cannot run program "aws comprehend detect-sentiment --region us-west-2 --language-code "en" --text "It is raining today in Seattle."": error=13, Permission denied Execute Process Configs: When i run the command on the NiFi Nodes, Its was working fine. The issue is when trying to execute the same command using the execute processor. Please help.
... View more
Labels:
- Labels:
-
Apache NiFi
03-01-2019
09:57 PM
Hi, I have the incoming flowfiles with the json content which have multiple event types. I need to filter only certain events based on two of the JSON key/value. Sample input data: {"eventType":"Mobile","timestamp":1551280374552,"name":"ERROR"} {"eventType":"Mobile","timestamp":1551280374552,"name":"APP_START"} {"eventType":"Immobile","timestamp":1551280374552,"name":"ERROR"} {"eventType":"Immobile","timestamp":1551280374552,"name":"ERROR"} {"eventType":"Mobile","timestamp":1551280374552,"name":"PLAYBACK_ERROR"} {"eventType":"Mobile","timestamp":1551280374552,"name":"Other"} {"eventType":"MobileCrash","timestamp":1551280374552,"name":"ERROR"} Filter Condition to be applied : Flowfile content having If eventtype = "Mobile" and name = (ERROR or APP_START or PLAYBACK_ERROR) or eventtype = "MobileCrash" Also, after the filter i have to rename the json key as, eventType = vendorEventType timestamp = currentTimestamp name = someName Desired Output: {"vendorEventType":"Mobile","currentTimestamp":1551280374552,"someName":"ERROR"} {"vendorEventType":"Mobile","currentTimestamp":1551280374552,"someName":"APP_START"}{"vendorEventType":"Mobile","currentTimestamp":1551280374552,"someName":"PLAYBACK_ERROR"}{"vendorEventType":"MobileCrash","currentTimestamp":1551280374552,"someName":"ERROR"} Please help.
... View more
Labels:
- Labels:
-
Apache NiFi
02-19-2019
06:05 PM
Thanks @Matt Clarke. I went with the option 2 and it worked. Thanks again for the quick reply.
... View more
02-19-2019
07:44 AM
Hi, Am trying to replace all characters after ";" including ; in the flowfile record. I was using replacetext processor with ;.*$ as the search value and set empty value for Replacement value properties to achieve this. However it is not working out. Any inputs in this regard would be really helpful. Example: Input Replace;text;processor Expected Output: Replace Thanks, Bala.
... View more
Labels:
- Labels:
-
Apache NiFi
02-19-2019
01:10 AM
Hi, Can anyone please help me with my use case?
... View more
02-16-2019
02:01 AM
Hi, I have a use case to add a new row from the value for the last column in the CSV record.Can someone help me doing this using the NiFi processors? Thanks, Bala.
... View more
Labels:
- Labels:
-
Apache NiFi
04-16-2018
05:45 PM
@suresh g /usr/bin/kafka-consumer-groups --zookeeper zk01.example.com:2181 --describe --group <<consumer-group-name>>. This will list all the producers under a given consumer group.
... View more
04-13-2018
08:55 PM
@Gourav Bhattacharya If i understand the issue correctly, the workflow is running forever without getting a single byte of data from FTP servers. Can you ping or Telnet the FTP hosts from the NiFi hosts? It may be due to the connectivity between the hosts.
... View more
04-12-2018
11:32 PM
@Ankit Juneja Please run your NiFi workflows using primary node only option. Also, schedule the workflows to run every one hour or so based on the use instead of running it again and again.
... View more
02-18-2018
09:51 PM
@Andrew Lim Below are the lines from the nifi-registry-app.log, 2018-02-18 16:46:01,149 INFO [NiFi Registry Web Server-36] o.a.n.r.w.m.ResourceNotFoundExceptionMapper org.apache.nifi.registry.exception.ResourceNotFoundException: No policy found for action='read', resource='/buckets/1f76510d-c48e-4ee6-b883-7edfcfe57e40'. Returning Not Found response.
2018-02-18 16:46:01,239 INFO [NiFi Registry Web Server-30] o.a.n.r.w.m.ResourceNotFoundExceptionMapper org.apache.nifi.registry.exception.ResourceNotFoundException: No policy found for action='write', resource='/buckets/1f76510d-c48e-4ee6-b883-7edfcfe57e40'. Returning Not Found response.
2018-02-18 16:46:01,331 INFO [NiFi Registry Web Server-15] o.a.n.r.w.m.ResourceNotFoundExceptionMapper org.apache.nifi.registry.exception.ResourceNotFoundException: No policy found for action='delete', resource='/buckets/1f76510d-c48e-4ee6-b883-7edfcfe57e40'. Returning Not Found response. s your NiFi Registry secured? Yes -that you have buckets in your Registry - Yes. have created one bucket -that your user has privilege to access the buckets in your Registry - Yes, I have assigned new policy for the bucket created Couple of questions. 1) Where do you run your nifi-registry service? on Ambari server or NiFi Nodes? 2) Do we need to specify Node identity while configuring nifi-registry?
... View more
02-17-2018
12:16 AM
AM using HDF 3.1.0 with NiFi and Nifi Registry. When i try use start version control from NiFi processors, am ending up getting Unable to obtain listing of buckets: org.apache.nifi.registry.client.NiFiRegistryException:
... View more
Labels:
- Labels:
-
Apache NiFi
-
Cloudera DataFlow (CDF)
02-08-2018
08:03 PM
Thanks Bryan. However using demarcator on the publisher will cause ordering and the single message broken into multiple messages issue. I have experienced this earlier.
... View more
02-07-2018
10:38 PM
Hi, I Have configured the NiFi Workflow to copy data from Kafka cluster 1 to Kafka cluster 2. As a result, it was taking around 11 mins to copy 1 Million messages from source cluster to destination. Can anyone help me improving the performance which can match the mirrormaker time taken for transferring similar amount of data. Source Cluster : 3 Brokers, The source topic with 3 partitions Destination Cluster : 3 Brokers, The Destination topic with 3 partitions NiFi Cluster : 3 Nodes Attached the workflow, consumerKafka Configs and PublishKafka Configs which i used.
... View more
Labels:
12-22-2017
09:31 PM
1 Kudo
HI, Now its working fine. The issue is with the configuartion in EvaluateJSONPath processor. I missed a "." for the attribute to fetch. I have accepted the answer. Thanks a lot for your help!!! It should be "$.extract_date" instead of "$extract_date"
... View more
12-22-2017
08:47 PM
1 Kudo
@Shu, My Bad. I Missed to configure EvaluateJSONPath. Here is my EvaluateJSONPath Processor Config, workflow: However while configuring EvaluateJSONPath Processor, Am ending up getting the error message as given below, Hope my workflow sequencing is correct. Please correct me if am wrong.
... View more
12-22-2017
07:33 PM
@Shu, Thanks for your quick reply. After i configure the above mentioned processor[updateAttribute] same as directed, Am ending up with a directory structure mentioned below, /folder/year=/month=/day=/hour=/677944128880138. The year, month, day and hour folders do not have the proper value populated..... It should be created as /folder/year=2017/month=12/day=22/hour=19/{filename} Note : The current run have the"extract_date": as "2017-12-22 19:16:17.0" Update Processor Configuration:
... View more
12-22-2017
06:52 PM
Hi, Am ingesting the data from a MySQL DB using the executeSQL-->ConvertAvroToJSON-->PublishKafka_0_10 processor. The result will have selected columns from the Table and the extract_date. The output will look {"col1": val1, "col2": "val2", "col3": "val3", "col4": val4,"extract_date": "2017-12-21 00:17:10.0"} and the same will be stored into a Kafka topic. After which i have another workflow to consume from the kafka topic and write it into a HDFS folder. [ConsumeKafka_0_10 --> PutHDFS] My Requirement is, while consuming the messages from the kafka topic, use hour value from the extract_date field and push the messages to the corresponding hour folder in the HDFS. For Example : If the field "extract_date": is having "2017-12-21 00:17:10.0" as the value. This message should be written into HDFS under /folder/year=2017/month=12/day=21/hour=00/{filename} Is there a way to achieve this use case. Thanks, Bala
... View more
Labels:
- Labels:
-
Apache NiFi
11-07-2017
07:43 PM
@Matt, Even after i configure as per the instructions... Am still getting the same error...
... View more
10-30-2017
10:53 PM
@Matt Clarke, Does the nifi.kerberos.krb5.file= should have the /<path-to>/kafka-jaas.conf?
... View more