@stevenmatison  Here you go -    There is only 1 node (Standalone).   # Core Properties # nifi.flow.configuration.file=./conf/flow.xml.gz nifi.flow.configuration.archive.enabled=true nifi.flow.configuration.archive.dir=./conf/archive/ nifi.flow.configuration.archive.max.time=30 days nifi.flow.configuration.archive.max.storage=500 MB nifi.flow.configuration.archive.max.count= nifi.flowcontroller.autoResumeState=true nifi.flowcontroller.graceful.shutdown.period=10 sec nifi.flowservice.writedelay.interval=500 ms nifi.administrative.yield.duration=30 sec # If a component has no work to do (is "bored"), how long should we wait before checking again for work? nifi.bored.yield.duration=10 millis nifi.queue.backpressure.count=10000 nifi.queue.backpressure.size=1 GB nifi.authorizer.configuration.file=./conf/authorizers.xml nifi.login.identity.provider.configuration.file=./conf/login-identity-providers.xml nifi.templates.directory=./conf/templates nifi.ui.banner.text= nifi.ui.autorefresh.interval=30 sec nifi.nar.library.directory=./lib nifi.nar.library.autoload.directory=./extensions nifi.nar.working.directory=./work/nar/ nifi.documentation.working.directory=./work/docs/components #################### # State Management # #################### nifi.state.management.configuration.file=./conf/state-management.xml # The ID of the local state provider nifi.state.management.provider.local=local-provider # The ID of the cluster-wide state provider. This will be ignored if NiFi is not clustered but must be populated if running in a cluster. nifi.state.management.provider.cluster=zk-provider # Specifies whether or not this instance of NiFi should run an embedded ZooKeeper server nifi.state.management.embedded.zookeeper.start=false # Properties file that provides the ZooKeeper properties to use if <nifi.state.management.embedded.zookeeper.start> is set to true nifi.state.management.embedded.zookeeper.properties=./conf/zookeeper.properties # H2 Settings nifi.database.directory=./database_repository nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE # FlowFile Repository nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog nifi.flowfile.repository.directory=./flowfile_repository nifi.flowfile.repository.partitions=256 nifi.flowfile.repository.checkpoint.interval=2 mins nifi.flowfile.repository.always.sync=false nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager nifi.queue.swap.threshold=20000 nifi.swap.in.period=5 sec nifi.swap.in.threads=1 nifi.swap.out.period=5 sec nifi.swap.out.threads=4 # Content Repository nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository nifi.content.claim.max.appendable.size=1 MB nifi.content.claim.max.flow.files=100 nifi.content.repository.directory.default=./content_repository nifi.content.repository.archive.max.retention.period=12 hours nifi.content.repository.archive.max.usage.percentage=50% nifi.content.repository.archive.enabled=true nifi.content.repository.always.sync=false nifi.content.viewer.url=../nifi-content-viewer/ # Provenance Repository Properties nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository nifi.provenance.repository.debug.frequency=1_000_000 nifi.provenance.repository.encryption.key.provider.implementation= nifi.provenance.repository.encryption.key.provider.location= nifi.provenance.repository.encryption.key.id= nifi.provenance.repository.encryption.key= # Persistent Provenance Repository Properties nifi.provenance.repository.directory.default=./provenance_repository nifi.provenance.repository.max.storage.time=24 hours nifi.provenance.repository.max.storage.size=1 GB nifi.provenance.repository.rollover.time=30 secs nifi.provenance.repository.rollover.size=100 MB nifi.provenance.repository.query.threads=2 nifi.provenance.repository.index.threads=2 nifi.provenance.repository.compress.on.rollover=true nifi.provenance.repository.always.sync=false # Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are: # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, Relationship, Details nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship # FlowFile Attributes that should be indexed and made searchable. Some examples to consider are filename, uuid, mime.type nifi.provenance.repository.indexed.attributes= # Large values for the shard size will result in more Java heap usage when searching the Provenance Repository # but should provide better performance nifi.provenance.repository.index.shard.size=500 MB # Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from # the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved. nifi.provenance.repository.max.attribute.length=65536 nifi.provenance.repository.concurrent.merge.threads=2 ... View more

@Wynner  The file is 3 gb size.  1 concurrent task. ... View more

Even I have similar issue. My FetchSFTP processor doesn't download large files.    I am trying to download any file that is uploaded to FTP server.  This is what I came up with -     It works perfectly for small files. For large files, the process stops at FectchSFTP processor.    Here is the template -  https://drive.google.com/file/d/1nPz6uUEwdYEopTH_7N1pDsvlX5Qx4uHu/view?usp=sharing   Can someone help please.? @Wynner @denis_urbanavic    Thanks.     ... View more

@Shu_ashu  This is my dataflow -  https://drive.google.com/file/d/1SWtSAPKxRcgAWT7ca0dytePjfwMZIpgR/view?usp=sharing   The ListSFTP processor works fine. It lists all the files in the server. But FetchSFTP doesn't work as expected. I get comms.failure. My ListSFTP  and FetchSFTP configurations are the same.    Also I had one more doubt - How does FetchSFTP gets its state from LISTSFTP.? Is there any additional configuration that has to be done.? I am just joining ListSFTP and FetchSFTP with relationship Success. Not doing anything else.   Please help.      ... View more

@Shu Actually there is no password. Username is ec2-user. I am able to connect to the server using Filezilla. I did not enter any password when I connected using Filezilla. Just username and it worked. But I imported the pem file. I also created password for my ec2 instance and tried. Still no luck. ... View more

@Shu I am able to connect to the server using Filezilla but unable to connect using ListSFTP processor. I get this error - Here are my configurations - Is it because Private Key Property is blank.? If yes, can you please tell what has to be done. The pem file is in my desktop (local machine). I gave the path to it but it says it invalid. Thanks. ... View more

@Matt Clarke Thanks for replying. Can you please take a look at my log file and suggest some help please. It shows some exception like --> java.net.SocketException: Unresolved address nifi-bootstrap.txt nifi-bootstrap.txt ... View more

@Shu It worked man! Thank you so very much:) LoL. One last thing, What if the my table has more number of columns lets say 15-20 columns. Do we still have to hard code all the column names in the processors.? eg - In the insert statement (in replaceText processor) where we do insert into table_name value (${column1},${column2}................) Is there a way that nifi can do this dynamically? I mean if there are 20-25 columns in my table then it would be a pain to mention all the column names in the insert statement. Can I use regex or something.? ... View more

@Shu Sure man. Here are the screenshots of all my processors - 1) GetFTP processor - (contd) 2) InferAROSchema - 3) ConvertCSVtoAVRO 4) AVROToJSON - (I'll post the rest it in the next answer since I allowed to post only 5 attachments) ... View more

@Shu I am not able to attach my CSV file. My CSV file has these two records. policyID,statecode,county 498960,498960,498960 498960,498960,792148 name of my CSV file - FL_insurance_sample.csv I created a table in my sql server with these three columns names all being varchar data type. ... View more

@Shu This is the output of my EVALUATEJSONPATH Processor - Guess this is fine. This is the output of my REPLACETEXT processor Don't really much about this. This is the output of my PUTSQL processor If you see the values section is blank with just two commas. Can you please tell me what is going wrong. My insert statement in REPLACETEXT processor insert into [test_shailesh].[dbo].[nifiTest](policyID,statecode,county) values (${policyID},${statecode},${county}) my overall flow - ... View more

@Shu Can you please help me fix this error. It says Incorrect syntax near','. Not able to figure out where I am making this error. This is my CSV file. My insert statement is insert into [test_shailesh].[dbo].[nifiTest](policyID,statecode,county) values (${policyID},${statecode},${county}) ... View more

@Shu I've been hard-coding the replacement value property in REPLACE TEXT processor ( eg - "insert into table_name values (..)" ). Is there a way to do it dynamically.? Can I use reg-ex.? I tried a couple if things. It did not work. Can you please tell me what Search Value is.? ... View more

Hi.. i was following article to install sandbox on docker. I was successfully able to complete step 3. when I execute the command docker images I am able to see - Sandbox-hdp. I am stuck in step 4. I ran the command that you have in step 4. I get this error. am I doing it correctly? Should I change the hostname.? I am a beginner. Please help. ... View more

@Yash Hi..I changed the relationship in all my processors. I get this error in all of them. Please take a look. I am getting the relationship error in all my processors. This is what I did in the bridge that connects "InferAvroSchema" and "convert CSV to AVRO".(screenshot - 2) This is my "inferavro" configuration . When I select success in "Automatically terminate relationships" it gets selected and after I connect it to the next processor it gets un-selected automatically. Please take a look. ... View more

@Yash Hi.. Your understanding is right..I am trying to insert it into the table only. I just wanted to see what the output is at the end of the flow..That's why I added PUTFILE. I'll change it. Again thank you so much for replying. I'll make the changes and try again. ... View more

@Yash Hi man. I was following your answer step by step. I am getting error. Can you please help me fix it. I am getting error "Convert CSV to AVRO" and "Convert "AVRO to JSON "processors.(Ignore the 3 processors in the 1st row) Screenshot - 1 shows the errors. At Convert CSV to AVRO it says cannot find Schema. My Configuration of "CSV to AVRO" processor. At "AVRO to JSON "processor. "AVRO to JSON " configuration I was trying to get a text file which has CSV's from filezilla. That part worked. and then the second step infer Avro schema also worked. Is it because my CSV's are in text file, these errors occur.? The only difference between your input and mine is. You hard coded the headers($is,$ name..) and I set take header from the first line of the file. Is it because of that.? Please have a look man. Thanks ... View more

Thanks you so much man @Yash. I'll try this out.:) ... View more

Hi @Yash. Thank you so much for taking your time to reply to my Question. I am sorry I do not understand your answer. I do not know what nodes are. What I was trying to do is download a CSV file from filezilla and insert it into a table in sql server. I used FetchFTP to get a CSV file from Filezilla. Before inserting I though I would try writing the file into my local machine. That's where i got struck. Can you teach me how to insert it into a table in sql server. If you know any good tutorials. Please suggest me. Thanks ... View more