Member since
07-30-2019
944
Posts
197
Kudos Received
91
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 1503 | 10-05-2021 01:53 PM | |
| 16322 | 09-23-2019 06:03 AM | |
| 6816 | 05-04-2019 08:42 PM | |
| 1502 | 06-11-2018 12:45 PM | |
| 12617 | 06-04-2018 01:11 PM |
05-08-2017
02:08 PM
@umair ahmed Here is a template of the above flow. counter-to-delay-on-loop.xml
... View more
05-08-2017
01:34 PM
2 Kudos
@umair ahmed
You could use the UpdateAttribute and RouteOnAttribute processors to create additional attributes on the flow file that keep track on the number of times the file has looped on a particular relationship. Here is an example of a flow that keep track of every relationship but the original flow file in the InvokeHttp processor: Here is the configuration of the UpdateAttribute processor The advanced UI in the RouteOnAttribute can be used to set the delay time using an attribute Here is the configuration of the RouteOnAttribute processor The delay is done using the ExecuteStreamCommand processor and calling the sleep command with the appropriate value to wait for the next attempt in using the InvokeHttp processor. You can adjust for your need.
... View more
05-08-2017
12:24 PM
@HAR MAT Do you mean once a day, once an hour?
... View more
05-06-2017
09:39 PM
1 Kudo
@Pradhuman Gupta Max Select - represents the maximum number of files pulled in a single connection, in your example it will get two files each time it runs times the number of concurrent tasks Run Schedule - the amount of time to wait between each task of pulling files, in your example the processor will pul files every 30 seconds Polling Interval - how long to wait between getting listings of new files FYI, we refer to these as properties. So for the example above, the processor will run the first time and get a listing with the 10,000 files and pull two of them, then it will wait 30, seconds and then pull two more files and so on. Basically, the processor will have to run 5,000 times to pull the 10,000 files, with a 30 second wait between tasks, it will take 4,999 x 30 seconds between tasks, so, it will take 149,970 seconds, 41.66 hours, to pull all of the files. Or, about 4 files/minute, 20 files/5 minutes. If you don't write any new files to the directory, then the polling interval could be set even higher. Also, the listing needs a concurrent task and the pulling need a concurrent task, so I would give the processor at least 2 concurrent tasks and reduce the time on the run schedule. Consider increasing the Max selects, at least to 100 the default, because that will be more efficient and faster. Is there a reason you are pulling only four files per minute?
... View more
05-05-2017
08:42 PM
1 Kudo
@Pradhuman Gupta Yes, that is what Matt is saying. Here is a sample of writing the output from the UpdateAttribute processor into it's own log file named nifi-upattr.log in the same directory as the nifi-app.log file: <appender name="UPATT_FILE" class="ch.qos.logback.core.rolling.RollingFileAppender">
<file>${org.apache.nifi.bootstrap.config.log.dir}/nifi-upattr.log</file>
<rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
<fileNamePattern>${org.apache.nifi.bootstrap.config.log.dir}/nifi-upattr_%d.log.gz</fileNamePattern>
<!-- keep 30 log files worth of history -->
<maxHistory>30</maxHistory>
</rollingPolicy>
<encoder class="ch.qos.logback.classic.encoder.PatternLayoutEncoder">
<pattern>%date %level [%thread] %logger{40} %msg%n</pattern>
</encoder>
</appender>
<logger name="org.apache.nifi.processors.attributes.UpdateAttribute" level="INFO" additivity="false">
<appender-ref ref="UPATT_FILE"/>
</logger>
And then tail this log file and send an email when WARN and/or ERROR log messages are written to it. In this example, I just added this to the logback.xml section of Ambari
... View more
05-05-2017
05:39 PM
@Ray Deng If there anything else in the user log? Have you got past the issue?
... View more
05-04-2017
05:47 PM
@Akash S Are there any errors or warnings in the nifi-app.log file? Typically, when users describe NiFi as hung, it is usually a result of thread starvation in the flow. If you look at the upper right hand corner of the UI, you will see Left click, another menu appears Select Controller Settings, then this window will appear The default values are 10 and 5, as shown above. How many processors are in your overall graph? What volume of data is passing through the graph?
... View more
05-04-2017
01:47 PM
@hello hadoop What version of HDF are you using?
... View more
05-03-2017
10:11 PM
@hello hadoop My configuration was the other way, try switching yours to short name first.
... View more
05-03-2017
09:11 PM
1 Kudo
@hello hadoop What do the /etc/hosts files look like on your nodes? I had a similar issue, I had to put the FQDN of the nodes first in the /etc/hosts file on the nodes. For example, I had 12.34.56.78 node1 node1.domain 12.34.56.79 node2 node2.domain 12.34.56.77 node3 node3.domain When I switched them to
12.34.56.78 node1.domain node1 12.34.56.79 node2.domain node2 12.34.56.77 node3.domain node3 Everything started just fine.
... View more