Member since
10-27-2016
81
Posts
10
Kudos Received
0
Solutions
09-20-2018
12:06 PM
I see below log in solr when I try access saved dashboard from banana. 2018-09-20 12:05:21,618 [qtp705265961-29] INFO [c:banana-int s:shard1 r:core_node1 x:banana-int_shard1_replica1] org.apache.solr.core.SolrCore (SolrCore.java:2482) - [banana-int_shard1_replica1] webapp=/solr path=/select params={q=title:"test_dashboard"&wt=json} hits=1 status=0 QTime=1
... View more
09-20-2018
12:03 PM
I created a dashboard in Banana and saved the dashboard to solr. It works fine. I can see the entry in solr banana-int collection. But when I try to access the saved dashboard from Banana, I get a blank page and some errors in browser debug console. The url to saved dashboard looks like below http://<SolrInstance>:8983/solr/banana/index.html#/dashboard/solr/test_dashboard?server=%2Fsolr%2F I am unable to understand what is going wrong and I could get much information from documentation what to check when blank screen is seen for saved dashboard. Can I get help to resolve the issue. Thanks.
... View more
Labels:
- Labels:
-
Apache Solr
08-27-2018
08:51 AM
Hi, I would like to know where the email notifications created from Ambari are stored into and how can we access them. The goal is to use same notification configuration defined in ambari to send other custom notifications from another application. Can anyone help me to understand the above. Thanks, Anil.
... View more
Labels:
- Labels:
-
Apache Ambari
04-13-2018
05:12 PM
@nallen @asubramanian Can you please help us how to configure metron alerts UI to make it functional. We are able to see the events in the alerts-ui, able to filter with search strings. But unable to mark event as dismiss/evaluate/resolved etc, unable to comment on events.
... View more
03-23-2018
01:32 PM
@nallen can I define the triage in profile definition as blow ?? "triage": {
"name":"Login Failed Anomaly",
"rule": SHELL_GET_EXPRESSION("profileName == 'demo_iplogon_failed' and count > 20"),
"score":"10",
"logon_failed_count": "count"
}
... View more
03-22-2018
09:16 AM
I am trying to develop an usecase with metron. The end-to-end usecase is, we need to calculate the count of
number of events happened for each ip’s. Based on the count reaching above the
threshold count generate an alerts in the merton alerts-UI. For that, I have the events data coming to “indexing” kafka
topic. I created a profile in ${METRON_HOME}/config/zookeeper/profiler.json as
below. {
"profiles": [
{
"profile": "demo_iplogon_failed",
"foreach": "ip_address",
"onlyif": "source.type == 'demo_windowsnxlog' and event_id ==
'4625'",
"init": {
"count": "0"
},
"update": {
"count": "count + 1"
},
"result": {
"profile" : "count",
"triage": {
"logon_failed_count": "count"
}
}
}
]} After reloading the zookeeper config files, I see the
indices created in ElasticSearch with format profile_index_yyyy.MM.dd.HH and
the snapshot of record is in attached snapshot I see all the records in the profiler_index_* pattern have field is_alert=”true”. But I don’t see these records turn up as alerts in the
metron-alerts UI. I would like to figure out the reason for alerts not turning
up in the metron-alerts UI ? I am seeing is_alerts=”true” for all the records under
profiler_index_*. How can I configure to set is_alert=”true” only when the
count exceeds the threshold value ? @nallen @asubramanian: tagging you as you can help me quickly. Please don't mind!
... View more
Labels:
- Labels:
-
Apache Metron
03-22-2018
09:07 AM
@nallen thank you very much for the quick response. I was able to figure out the issue of adding profile to result and implemented!
... View more
03-16-2018
10:40 AM
Hi @asubramanian @nallen do you guys please look the issue I am facing and help me!
... View more
03-15-2018
03:06 PM
I am trying implement POC with usecase as below I have windows log data being loaded to Indexing topic. I want to define a profiler which checks for failed logons in 15 minutes time frame and generate alerts in metron UI when failed logons count goes beyond threshold level. For that I just started trying to define logon failed count profiler as below {
"profiles": [
{
"profile": "demo_iplogon_failed",
"foreach": "ip_address",
"onlyif": "source.type == 'demo_windowsnxlog' and event_id == 4625",
"init": {
"count": "0"
},
"update": {
"count": "count + 1"
},
"result": {
"triage": {
"logon_failed_count": "count"
}
}
}
]
}
But when I try to run the profiler, it is giving me an exception as below [!] Assignment expression failed
java.lang.IllegalArgumentException: Assignment expression failed
at org.apache.metron.stellar.common.shell.StellarResult.error(StellarResult.java:115)
at org.apache.metron.stellar.common.shell.specials.AssignmentCommand.execute(AssignmentCommand.java:82)
at org.apache.metron.stellar.common.shell.DefaultStellarShellExecutor.execute(DefaultStellarShellExecutor.java:252)
at org.apache.metron.stellar.common.shell.cli.StellarShell.execute(StellarShell.java:357)
at org.jboss.aesh.console.AeshProcess.run(AeshProcess.java:53)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Can someone help me to figure out what is the problem in above profiler configuration? Also, is there any other better way to implement my usecase with metron?
... View more
Labels:
- Labels:
-
Apache Metron
01-30-2018
12:41 PM
I am trying to convert the epoch in milliseconds to TimeStamp of specific timezone. I tried using "format" of nifi expression language as below in updateAttribute processor ${closed_epoch:format("yyyy", "GMT")} But updateAttribute processor is showing error saying "unexpected token ','. Is it the bug with Nifi expression language? and what is the timezone that Nifi uses by default when I try ${closed_epoch:format("yyyy")}? Is there any other way to convert epoch to timestamp of specific timezone ?
... View more
Labels:
- Labels:
-
Apache NiFi
01-25-2018
09:10 AM
1 Kudo
It works, Thanks alot @Shu
... View more
01-24-2018
06:35 PM
1 Kudo
I am trying to replace the content in the flowfile using replaceText processor. If the attribute is "myattr" and I replace the content of the flowfile using replaceText processor with ${myattr}. I am seeing the below results if myattr is "$$this is test$$", then the content of flowfile is replaced as "\$this is test\$" if myttr is "\$$$this is test$$", then the content of flowfile is replaced as "\$\$this is test\$" May I know how can I overcome the issue of '\'
... View more
Labels:
- Labels:
-
Apache NiFi
01-24-2018
06:24 PM
it seems like replaceText processor is replacing '$$$$$*' in the content to '\$' while replacing the flow file content! seems like actual issue is I have an attribute namely "myattr" when myattr is "$$This is test$$", if I replace the flowfile content with myattr, the content in the flowfile is "\$this is test\$"
... View more
01-24-2018
06:06 PM
what might be the reason for the extra character '\' ?
... View more
01-24-2018
02:42 PM
1 Kudo
I have requirement where I need to append and prepend the attribute with "$$$". how ever, Nifi expression language is appending or prepending with "\$" For example: if attribute named "myattr" value is "This is Test" when I do ${myattr:prepend('$$$'):append('$$$')} I am expecting "$$$This is Test$$$" but the output is "\$This is Test\$" Is there any way to append or prepend the attribute with "$$$"
... View more
Labels:
- Labels:
-
Apache NiFi
-
Apache Pig
10-18-2017
09:18 AM
1 Kudo
I would like to know if there is a way to encrypt the non-empty HDFS directory using RangerKMS?? I am getting below exception while trying to create encryptZone for non-empty directory. ************** hdfs crypto -createZone -keyName demokey -path /user/demo
RemoteException: Attempt to create an encryption zone for a non-empty directory. ***************
... View more
Labels:
10-06-2017
09:17 AM
Back pressure threshold is set to infinite. Queues can hold infinite number of flow files for infinite time. I see processors not running as usual. They are executing only once for 10 secs.
... View more
10-05-2017
07:05 AM
Hi, I am observing the below behavior when there huge number of flow files are in processing by the data flow. All the nifi processors in the data flow are executing once per 10 secs. For example: I have a updateAttribute processor which is configured to perform 1 concurrent task. I see that it is processing 6 records per minutes even though there are thousands of flow files in the queue. At the start of loading huge data, the flow works fine. It processes thousands of records in just seconds. But I see the above kind of behavior after sometime. I am unable to understand what is happening in the backend of nifi that is causing for this kind of behavior. Can I get some help to understand/troubleshoot the issue. Thanks.
... View more
Labels:
- Labels:
-
Apache NiFi
08-24-2017
03:47 PM
Hi, I am facing an issue with consumeKafka_0_10 processor in my local nifi instance. I have seen the Jira created for my issue and was resolved long time back. So, I believe my local nifi instance does not have the fix for the ConsumeKafka_0_10 processor. I would like to know 1) where can I get the latest nar file for consumeKafka_0_10 processor 2) what is the process to update the consumeKafka_0_10 processor with latest in my local nifi instance
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
08-24-2017
03:44 PM
@Geoffrey Shelton Okot Yes, that makes sense. But I am unable to figureout where can I configure those parameters in ConsumeKafka_0_10 processors
... View more
08-24-2017
03:10 PM
Attached are the logs that are related to my getKafka processor. Can someone please go through the same and let me know what might be going wrong!getkafka-processor-log.txt
... View more
08-24-2017
02:12 PM
I did the changes as specified in the above link(https://community.hortonworks.com/articles/28180/how-to-configure-hdf-12-to-send-to-and-get-data-fr.html) and configured getkafka processor. But I am getting below error when I start the getKafka processor. ********************Error Log********* 2017-08-24 10:10:19,964 INFO [test-consumer-group_storagev4-1c-1503583093158-c73a2251-leader-finder-thread] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1503583093205] Added fetcher for partitions ArrayBuffer()
2017-08-24 10:10:20,169 WARN [test-consumer-group_storagev4-1c-1503583093158-c73a2251-leader-finder-thread] k.c.ConsumerFetcherManager$LeaderFinderThread [test-consumer-group_storagev4-1c-1503583093158-c73a2251-leader-finder-thread], Failed to find leader for Set([indexing,0])
java.lang.NullPointerException: null ************************
... View more
08-17-2017
03:07 PM
I tried the possible settings with higher session timeouts. But I could not able to resolve the issue with consumeKafka. Where as I dint faced such issue with getKafka processor. So, I am using getKafka. But now I am facing the issue with getKafka in kerberos enabled environment. I would to know how to configure getKafka in kerberos enabled environment and would like to hear possible workarounds to resolve the issue that I mentioned in the logs.
... View more
08-17-2017
02:38 PM
Nope. issue is not resolved!
... View more
08-17-2017
02:36 PM
when I use consume kafka, I am facing other issue as mentioned in the below link https://community.hortonworks.com/answers/114429/view.html
... View more
08-17-2017
10:38 AM
Hi, I am trying to read the data from kafka topic on the cluster where kerbaros is enabled. I am getting errors like below for all the topics that I am trying to read the data from. ***************Error logs******************* 2017-08-17 06:18:26,505 INFO [test-consumer-group_storagev4-1c-1502965058303-e4ac2072-leader-finder-thread] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1502965058340] Added fetcher for partitions ArrayBuffer()
2017-08-17 06:18:26,599 WARN [test-consumer-group_storagev4-1c-1502965084565-3028bb7a-leader-finder-thread] k.c.ConsumerFetcherManager$LeaderFinderThread [test-consumer-group_storagev4-1c-1502965084565-3028bb7a-leader-finder-thread], Failed to find leader for Set([demo_unix_indexing,0])
java.lang.NullPointerException: null
2017-08-17 06:18:26,599 INFO [test-consumer-group_storagev4-1c-1502965084565-3028bb7a-leader-finder-thread] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1502965084750] Added fetcher for partitions ArrayBuffer()
2017-08-17 06:18:26,710 WARN [test-consumer-group_storagev4-1c-1502965058303-e4ac2072-leader-finder-thread] k.c.ConsumerFetcherManager$LeaderFinderThread [test-consumer-group_storagev4-1c-1502965058303-e4ac2072-leader-finder-thread], Failed to find leader for Set([indexing,0])
java.lang.NullPointerException: null
2017-08-17 06:18:26,710 INFO [test-consumer-group_storagev4-1c-1502965058303-e4ac2072-leader-finder-thread] kafka.consumer.ConsumerFetcherManager [ConsumerFetcherManager-1502965058340] Added fetcher for partitions ArrayBuffer()
2017-08-17 06:18:26,805 WARN [test-consumer-group_storagev4-1c-1502965084565-3028bb7a-leader-finder-thread] k.c.ConsumerFetcherManager$LeaderFinderThread [test-consumer-group_storagev4-1c-1502965084565-3028bb7a-leader-finder-thread], Failed to find leader for Set([demo_unix_indexing,0])
java.lang.NullPointerException: null ******************** I have done the necessary configurations but I could not able to figure out the cause of the issue. Note: I am able to read the data from kafka topic with consumeKafka processor. Nifi details: Nifi is running on standalone single node. Zookeeper and kafka are running on 3 node cluster. ***********jass file configiuration******* Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/nifi.keytab"
storeKey=true
useTicketCache=false
principal="nifi@TESTSSTECH.LOCAL";
}; KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="kafka"
useKeyTab=true
keyTab="/etc/security/keytabs/nifi.keytab"
principal="nifi@TESTSSTECH.LOCAL";
}; **************************
... View more
Labels:
- Labels:
-
Apache Kafka
-
Apache NiFi
08-07-2017
11:51 AM
@Dan Chaffelson I tried that too. that does not help!
... View more
08-03-2017
12:53 PM
I would like to extract the property "feedts" from the flowfile JSON using EvaluateJSONProcessor. JSON is as below **********************JSON**************** {"latestFeedJobExecutionContext":{"demo.msexchange":{"jobExecutionId":87,"startTime":1501760142595,"endTime":1501760304816,"executionContext":{"feedts":"1501760190824"}}},"feedName":"test.test_trigger_data_transition","feedId":"474a06bc-7d47-4404-9935-7f7edea55979","dependentFeedNames":["demo.msexchange"] ********************* I would like to know how can I overcome the character "." in the property name. In this case I have its "demo.msexchange". Is there anyway I can extract the property "feedts" from above JSON.
... View more
Labels:
- Labels:
-
Apache NiFi
07-20-2017
08:45 AM
We have that flow running fine with out Nifi in our current setup. We are trying to migrate to Nifi, hence build the flow in Nifi using Consume Kafka.
... View more
07-19-2017
03:27 PM
I tried get some help from blogs. But I am unable to find the relevant information to resolve the issue. It would be helpful if any one have configured Consumer Kafka processor in Nifi successfully to process thousands of records per minutes can share the way the processor is ocnfigured! Thanks, Anil.
... View more