Member since
12-30-2015
21
Posts
4
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
6516 | 04-14-2016 07:41 PM |
04-21-2016
10:27 PM
Update: if just connect EvaluateJsonPath to RouteOnAttribute, configure it to route on ${msg:find('xyz'):equals('true')}. The result is still the same-- getting entire file instead of the records has xyz in msg. So, guess it is still RouteOnAttribbute is having problem. Any help on correcting it would be greatly appreciated. Thanks, Shannon
... View more
04-21-2016
10:15 PM
@jwitt, @Chris Gambino Would you be so kind to take a look at updated info above (the second part of the original posting)? I tried to simply the description in the beginning, which was a wrong call :). May be the problem is not in the processor I originally thought. May be UpdateAttribute was not configured right with the expression given? In the output file, I can't see the new attribute match_msg added, which was not what I expected either. Any ideas? Thanks! Shannon
... View more
04-21-2016
09:20 PM
2 Kudos
Hi Does anyone know if it is by design that Nifi RouteOnAttribute on look at an attribute value that is for the entire flowfile instead of attribute value that can be different for different record in a josn file? e.g., when RouteOnAttribute is set to find a word 'xyz' in a json element (say msg), e.g, properties configure to route on ${msg:find('xyz')}, it actually route entire file including records that msg does not contain xyz. If that how it works or if the RouteonAttribute properties configuration is missing something? Thanks, Shannon Update: Edit to add additional information based on @Chris Gambino and @jwitt replies: Data flow is design as: GetSourceData --> some process for parsing to get a flat json --> EvaluateJsonPath--> UpdateAttribute -- > RouteOnAttribute May be the problem is at UpdateAttribute, which has a property added to an attribute "matched_msg" using below so that this new attribute has 1 or 0 value. ${msg:find('xyz'):equals('true'):replace('true', '1'):or (${tmsg:find('xyz'):equals('false'):replace('false', '0')})} Above followed by RouteOnAttribute using ${matched_msg:equals(1)} Results is that it ended up with entire file instead of records with the match.
... View more
Labels:
- Labels:
-
Apache NiFi
04-14-2016
07:41 PM
Thanks to help from @Qi Wang, the problem is solved by moving the appTimeline server to the same server where history server is. Thank you, Qi!
... View more
04-12-2016
11:07 PM
@emaxwell Yes, that was the template I used as the base for development. I used flowfile-attibute. Here is the screenshot for the properties. None of those properties added works. Edit: May be this is what caused problem: Match has an array as its value. I tried $.response.Request.Actual.query, it worked fine. Do you know what to do to parse value from "Match"? (without going to a ExecuteScript). Thanks so much for helping! Shannon
... View more
04-12-2016
09:28 PM
@emaxwell That was what I did. but still end up pulling all attribute. Going to Provenance, it showed EvaluateJsonPath pulled empty strings on all specified attribute. Would you be so kind an tell me where I got it wrong? -- I tried to pull "subject" property using value $.response.Matches.M atch.Subject, $.Matches.Match.Subject, $.Match.Subject, and $.Subject. None of those worked. The source data looks like (see below):
Thanks ! Shannon
... View more
04-12-2016
12:48 AM
@emaxwell Hi emaxwell, I have a same task to extract attributes from json file via gethttp with EvaluateJsonPath processor. However, the resulting data from the processor do not limit to the attributes, it pulled all attributes the original json data has. Is this meant to be so? Thank you! Shannon
... View more
04-12-2016
12:34 AM
@Qi Wang Hi Qi, would you please email me at sluo(at)anexinet.com? I can send environment details to your email. Thanks! Shannon
... View more
04-07-2016
02:52 PM
@Emil Hi Emil,
Turn on Hive with Tez as execution engine. Still same problem. Where should I get this missing plugin, do you know? Thanks!
Shannon
... View more
04-01-2016
12:49 PM
@Emil Hi Emil, Thanks for replying. Sorry I was out of office. Found error in yarn-yarn-timelineserver-xxx.log only (see below). It seems to be missing a timeline plugin related to tez. Ambari used is the newest as of March 2016 and should not have any patch issue related to this, should it? Tez was not running as hive was not used. Do you know how this can be fixed? Thanks! Shannon --------------copied from log--------- 2016-03-10 20:57:02,239 INFO timeline.EntityGroupFSTimelineStore (EntityGroupFSTimelineStore.java:loadPlugIns(217)) - Error loading plugin org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl
org.apache.hadoop.yarn.exceptions.YarnException: Unable to load class: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl
at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getClazz(EntityGroupFSTimelineStore.java:187)
at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:214)
at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:171)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168)
at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178)
Caused by: java.lang.ClassNotFoundException: org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.getClazz(EntityGroupFSTimelineStore.java:183)
... 8 more 2016-03-10 20:57:02,245 FATAL applicationhistoryservice.ApplicationHistoryServer (ApplicationHistoryServer.java:launchAppHistoryServer(171)) - Error starting ApplicationHistoryServer
java.lang.RuntimeException: No class defined for org.apache.tez.dag.history.logging.ats.TimelineCachePluginImpl
at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.loadPlugIns(EntityGroupFSTimelineStore.java:223)
at org.apache.hadoop.yarn.server.timeline.EntityGroupFSTimelineStore.serviceInit(EntityGroupFSTimelineStore.java:171)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.serviceInit(ApplicationHistoryServer.java:104)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.launchAppHistoryServer(ApplicationHistoryServer.java:168)
at org.apache.hadoop.yarn.server.applicationhistoryservice.ApplicationHistoryServer.main(ApplicationHistoryServer.java:178)
... View more
- « Previous
-
- 1
- 2
- Next »