Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

problem using tailFile for over 5G file?

problem using tailFile for over 5G file?

Explorer

Hi

I have a python code to make a file with squid content (for example 50,000,000 and 5.9G)

I wanna transfer it HDFS durung runnig my code, so I use tailFile, mergeContent (Maximum Number of Entries = 10000) and putHDFS(block.size=64 MB)

cpu=8

ram=23G

heap size(in bootstrap.conf) = 4096MB

thread=6

After my code was finished, tailFile start to get all the content of my file but I am wanting to start when my code start .

I am using single NiFi

How can I solve it?

96403-tail.png

Don't have an account?
Coming from Hortonworks? Activate your account here