Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

QueryDatabaseTable processor shutting down

Highlighted

Re: QueryDatabaseTable processor shutting down

Super Guru
@Sami Ahmad

You can do nifi.sh to analyse what does the threads are working on

./nifi.sh dump thread-dump.txt

Could you please clear state in the Querydatabase table processor and then run again the processor, we are encountering the issues that Matt Burgess mentioned in the comments,matt's comment "I recommend against using Maximum Number of Fragments, since the result set is not ordered, you can miss rows on the next execution if the largest current value for the max-value column has been encountered in the first set of "fragments". Instead, in NiFi 1.6.0 / HDF 3.2 (via NIFI-4836), you will be able to send out flow files while the result set is still being processed (via the Output Batch Size parameter). Note that when you use that parameter, the flow files won't have the total result set count information (because some flow files have already been transferred before the total count was ascertained), and it's not transactional (meaning an error halfway through the result set processing may end up with duplicate data being transmitted downstream on the next run)."

Please keep Maximum Number of Fragments to zero and re run the processor again then we are not going to miss any records.

Highlighted

Re: QueryDatabaseTable processor shutting down

Master Collaborator
ok I will clear everything and run the load again . in meantime the current flow is waiting for some reason
"FileSystemRepository Workers Thread-4" Id=59 WAITING  on java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d0
0fd51
        at sun.misc.Unsafe.park(Native Method)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
"Finalizer" Id=3 WAITING  on java.lang.ref.ReferenceQueue$Lock@59551962
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)





Highlighted

Re: QueryDatabaseTable processor shutting down

Master Collaborator

after clearing everything I started the flow again but its still doing nothing . the dump file shows the same waiting messages.

I had changed the concurrent tasks to 4 in the QDB-scheduling, maybe this is causing the issue , but I cant set it back to default value . ?

Btw my "max nr of fragments" is set to zero.

64858-capture.jpg

Highlighted

Re: QueryDatabaseTable processor shutting down

Super Guru
@Sami Ahmad

Could you please try to run QDT processor with the following configs after clearing the state in the processor

64861-querydatabasetable.png

For the first pull will take long time wait until it completes to pull all the 55 million records, after that processor needs to check for incremental records for every 10 sec.

if the processor doesn't seems to be working just threads are hanging for ever then

bin/nifi.sh dump thread-dump.txt

then share the

nifi-bootstrap.log

Re: QueryDatabaseTable processor shutting down

@Matt Burgess your comment on max fragment size feature if querydb processor. is this applicable to max rows fetch size too, i have set this in my flow and facing data loss issues. few records are missed by the processor working on 3 node cluster.

Don't have an account?
Coming from Hortonworks? Activate your account here