Hi, I have json file of dashboard exported from one of the Banana instance. Now I want to import/create the dashboard in another Banana instance with same name and id. Is there anyway I achieve it using commands through terminal or using any REST Api calls to import dashboard to Banana ? ... View more

Hi, I have a flow where ListFile processor listens the new files from the source folder and processes the files in the next steps. What I observed is ListFile processor listens all the new files from source fine but all of sudden after few days (after some unexpected time) the processor stops listening to the new files. When I check the "view state", it shows the old timestamp as processed even though there are new files after that timestamp. I also dont see any errors at the processor level and in Nifi logs too I am unable to trace the error at ListFile processor. I would like to get some help to figure out why the ListFile processor stopped listening for new files in the source folder. Any tips to troubleshoot will be much appreciated. Note: restarting the nifi brings back the processor work normally. We are seeing the issue frequently after some unexpected time. ... View more

I created a dashboard in Banana and saved the dashboard to solr. It works fine. I can see the entry in solr banana-int collection. But when I try to access the saved dashboard from Banana, I get a blank page and some errors in browser debug console. The url to saved dashboard looks like below http://<SolrInstance>:8983/solr/banana/index.html#/dashboard/solr/test_dashboard?server=%2Fsolr%2F I am unable to understand what is going wrong and I could get much information from documentation what to check when blank screen is seen for saved dashboard. Can I get help to resolve the issue. Thanks. ... View more

Hi, I would like to know where the email notifications created from Ambari are stored into and how can we access them. The goal is to use same notification configuration defined in ambari to send other custom notifications from another application. Can anyone help me to understand the above. Thanks, Anil. ... View more

Hi, I defined a profile definition for my requirement to perform profile statistics. The calculation of profiler statistics for current profiling period will depend on the previous profile flushed stats. So, I am trying to read the previous stats during initialization ( “init” field in json ) part of profile for the current profiling period. The definition works fine when there are profile stats from the previous period, which means my profile definition working fine for first 15 minutes (my profile period is set to 15 mins). When profiler flushes the profile stats into hbase for first 15 minutes and starts for next period, I am getting the parse exception as seen in attached doc at Profiler topology and SplitterBolt. I have executed all lines of code in the stellar shell (instantiated with zookeeper config ./bin/stellat –z XXXX). It works fine without any issues in all the cases. Which means, I am able to fetch the previous profile stats, initialize for current record and execute all lines of code in profiler definition with expected result. I could not able to figure out what else is going wrong during SplitterBolt execution. I request someone to go through the attached logs and help me to troubleshoot the issue further. High Level Exception stack trace: *************************************** Exception seen at profiler topology logs ******************************* 2018-04-27 10:56:06.483 o.a.h.m.s.s.StormTimelineMetricsSink Thread-23 [WARN] Unable to send metrics to collector by address:http://null:6188/ws/v1/timeline/metrics 2018-04-27 10:57:16.575 o.a.m.p.b.ProfileSplitterBolt Thread-6-splitterBolt-executor[7 7] [ERROR] Unexpected failure: message='null', tuple='{value=[123,***********Lot of integer values here in the logs end with****34, 48, 34, 125]}' org.json.simple.parser.ParseException: null at org.json.simple.parser.Yylex.yylex(Yylex.java:610) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.parse(JSONParser.java:118) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.parse(JSONParser.java:81) ~[stormjar.jar:?] at org.json.simple.parser.JSONParser.parse(JSONParser.java:75) ~[stormjar.jar:?] at org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:108) ~[stormjar.jar:?] at org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:94) [stormjar.jar:?] at org.apache.storm.daemon.executor$fn__10250$tuple_action_fn__10252.invoke(executor.clj:730) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.daemon.executor$mk_task_receiver$fn__10171.invoke(executor.clj:462) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.disruptor$clojure_handler$reify__9685.onEvent(disruptor.clj:40) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.daemon.executor$fn__10250$fn__10263$fn__10316.invoke(executor.clj:849) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) [storm-core-1.1.0.2.6.3.0-235.jar:1.1.0.2.6.3.0-235] at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161] ****************************** Exception at SplitterBolt: *********************** Unexpected character (.) at position 885. at org.json.simple.parser.Yylex.yylex(Yylex.java:610) at org.json.simple.parser.JSONParser.nextToken(JSONParser.java:269) at org.json.simple.parser.JSONParser.parse(JSONParser.java:118) at org.json.simple.parser.JSONParser.parse(JSONParser.java:81) at org.json.simple.parser.JSONParser.parse(JSONParser.java:75) at org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:108) at org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:94) at org.apache.storm.daemon.executor$fn__10250$tuple_action_fn__10252.invoke(executor.clj:730) at org.apache.storm.daemon.executor$mk_task_receiver$fn__10171.invoke(executor.clj:462) at org.apache.storm.disruptor$clojure_handler$reify__9685.onEvent(disruptor.clj:40) at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472) at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451) at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73) at org.apache.storm.daemon.executor$fn__10250$fn__10263$fn__10316.invoke(executor.clj:849) at org.apache.storm.util$async_loop$fn__553.invoke(util.clj:484) at clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:748) @nallen @asubramanian your help is much appreciatedprofile-test-def.txtprofilertopology-exception-logs.txt Thanking you in advance. ... View more

We have a custom built-in oauth service provider. I would like to know if there is a way to integrate SSO with custom oauth for my hadoop services like ambari, zeppelin, kibana etc. So far my understanding, we can configure knox gateway for authentication which uses apache shiro. But apache shiro does not provide authentication for Oauth directly. It intern uses pacp4j to support other authentication mechanis like oauth, saml etc. But I am unable to figureout how to configure pacp4j to support authentication from our custom oauth service provider. May I get some help to understand if it is possible to configure pacp4j for custom oauth service such that we can integrate SSO for ambari, zeppelin etc services ... View more

I am trying to develop an usecase with metron. The end-to-end usecase is, we need to calculate the count of number of events happened for each ip’s. Based on the count reaching above the threshold count generate an alerts in the merton alerts-UI. For that, I have the events data coming to “indexing” kafka topic. I created a profile in ${METRON_HOME}/config/zookeeper/profiler.json as below. { "profiles": [ { "profile": "demo_iplogon_failed", "foreach": "ip_address", "onlyif": "source.type == 'demo_windowsnxlog' and event_id == '4625'", "init": { "count": "0" }, "update": { "count": "count + 1" }, "result": { "profile" : "count", "triage": { "logon_failed_count": "count" } } } ]} After reloading the zookeeper config files, I see the indices created in ElasticSearch with format profile_index_yyyy.MM.dd.HH and the snapshot of record is in attached snapshot I see all the records in the profiler_index_* pattern have field is_alert=”true”. But I don’t see these records turn up as alerts in the metron-alerts UI. I would like to figure out the reason for alerts not turning up in the metron-alerts UI ? I am seeing is_alerts=”true” for all the records under profiler_index_*. How can I configure to set is_alert=”true” only when the count exceeds the threshold value ? @nallen @asubramanian: tagging you as you can help me quickly. Please don't mind! ... View more

I am trying to convert the epoch in milliseconds to TimeStamp of specific timezone. I tried using "format" of nifi expression language as below in updateAttribute processor ${closed_epoch:format("yyyy", "GMT")} But updateAttribute processor is showing error saying "unexpected token ','. Is it the bug with Nifi expression language? and what is the timezone that Nifi uses by default when I try ${closed_epoch:format("yyyy")}? Is there any other way to convert epoch to timestamp of specific timezone ? ... View more

Hi, I have requirement where I need to publish messages to JMS and have multiple consumers to consume and process the same message. so, I created a topic and started posted messages to the topic using publishJms process in Nifi. I configured three JMSConsumer processors to consume the data from above topic. I observe that the messages that posted to the topic are getting consumed by any of the only one consumers. I would like to know if it is possible to have setup as per my requirement using publishJMS and ConsumeJMS or any other processors in Nifi. Required setup is: I will be posting the messages to some topic or queue and I would like to have multiple consumers to do different tasks for each message, so multiple consumers should be able to consume all the messages from above topic or queue. ... View more

I have configured Nifi putElasticSearch processor to store data to elasticSearch instance where shield plugin is enabled. I just tried the shield configuration for the putElasticSearch processor and stopped the processor as it was throwing errors. But I see that in the nifi-app.log the same error messages are being generated continuously (below kind of error messages and generating MB's of logs in 5 seconds approximately) ************************Error Message***************** 2017-10-23 12:55:02,730 INFO [elasticsearch[Martin Gold][generic][T#16]] org.elasticsearch.client.transport [Martin Gold] failed to get node info for {#transport#-1}{xx.x.xx.xx}{xx.x.xx.xx:9300}, disconnecting... org.elasticsearch.transport.RemoteTransportException: [ip-xx-x-xx-xx.ec2.internal][xx.x.xx.xx:9300][cluster:monitor/nodes/liveness] Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [cluster:monitor/nodes/liveness] at org.elasticsearch.shield.support.Exceptions.authenticationError(Exceptions.java:39) at org.elasticsearch.shield.authc.DefaultAuthenticationFailureHandler.missingToken(DefaultAuthenticationFailureHandler.java:70) at org.elasticsearch.shield.authc.InternalAuthenticationService.authenticateWithRealms(InternalAuthenticationService.java:282) at org.elasticsearch.shield.authc.InternalAuthenticationService.authenticate(InternalAuthenticationService.java:184) at org.elasticsearch.shield.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:112) at org.elasticsearch.shield.transport.ShieldServerTransportService$ProfileSecuredRequestHandler.messageReceived(ShieldServerTransportService.java:176) at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) at org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:245) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:114) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:75) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.handler.ipfilter.IpFilteringHandlerImpl.handleUpstream(IpFilteringHandlerImpl.java:154) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178) at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108) at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) ************************************* may I know what could be cause of huge continuous logs generation even though putElasticsearch processor is paused. ... View more

I would like to know if there is a way to encrypt the non-empty HDFS directory using RangerKMS?? I am getting below exception while trying to create encryptZone for non-empty directory. ************** hdfs crypto -createZone -keyName demokey -path /user/demo RemoteException: Attempt to create an encryption zone for a non-empty directory. *************** ... View more

Hi, I am observing the below behavior when there huge number of flow files are in processing by the data flow. All the nifi processors in the data flow are executing once per 10 secs. For example: I have a updateAttribute processor which is configured to perform 1 concurrent task. I see that it is processing 6 records per minutes even though there are thousands of flow files in the queue. At the start of loading huge data, the flow works fine. It processes thousands of records in just seconds. But I see the above kind of behavior after sometime. I am unable to understand what is happening in the backend of nifi that is causing for this kind of behavior. Can I get some help to understand/troubleshoot the issue. Thanks. ... View more

I have a requirement to establish connection with websocket using JettyWebSocketClient controller in Nifi. I need to pass authorization parameters as header to establish valid session with web socket. Example python code that works is as below ws = websocket.WebSocketApp("wss://<domain/url>/api-ws/v1/stream",header=['Authorization: Basic %s' % base64.encodestring(email + ':' + password)],on_message=on_message,on_error=on_error,on_close=on_close I am unable to figure out how to add the header attribute to JettyWebSocketClient configuration for establishing valid session with web socket. Can I get some help to understand if my case can be fulfilled with nifi JettyWebSocketClient controller. ... View more

Hi, I am facing an issue with consumeKafka_0_10 processor in my local nifi instance. I have seen the Jira created for my issue and was resolved long time back. So, I believe my local nifi instance does not have the fix for the ConsumeKafka_0_10 processor. I would like to know 1) where can I get the latest nar file for consumeKafka_0_10 processor 2) what is the process to update the consumeKafka_0_10 processor with latest in my local nifi instance ... View more