@Ravikiran Dasari: do you have any experience in shell scripts? Do you know how to write a bash script, or maybe a Python script? Do you have a special scheduler that you use in your environment, or will you use cron? If you don't know, I guess it will become cron. Try if you are able to edit the crontab via entering this command on the shell crontab -e Either you get a list of cron jobs or an error message like 'You (<<userid>>) are not allowed to use this program (crontab)' Now when you want to write a shell script the starting point is a simple text files, containing the commands you otherwise enter on the shell. The script file should start with a line aka 'shebang' providing the script interpreter to be used. I.e. on RedHat bash: #!/bin/bash python: #!/usr/bin/python perl: #!usr/bin/perl If you decide to go for bash script, just create a file like this (you can use a different editor if you like): vi ~/mycopyscript enter in that script all your command #!/bin/bash dir = `date +%Y-%m-%d` sftp ayosftpuser@IPaddredss << __MY_FTP_COMMANDS__ password cd /sourcedir get -Pr ${dir} bye __MY_FTP_COMMANDS__ #at this point the files should already be locally copied hadoop fs -put -f ${dir} /destination Save the script ( by entering <ESC>:wq in vi) Next make the script executable, and only allow access from the owner (you) chmod 700 ~/mycopyscript You should be able to execute it now: ~/mycopyscript This script is just a starting point, and done plain simple, no error handling and no security, whoever reads the script also has access to the password, and no parameter (you must execute it at the date that the dir uses) Still it should provide you with the basic idea of a shell script. ... View more

yes, that's correct. The only interesting point is, how your client works. If it is a proper hadoop client, it will run directly and in parallel on the nodes storing the file blocks. If you have a non-hadoop client it will really retrieve the full file from hdfs, process it and write back to hdfs. In a Spark application each of the steps a-d will be executed in parallel on different nodes, while the hadoop framework takes care to bring the execution to the data. And if the stripes are unluckily distributed on the blocks (and therefore hdfs nodes), the data tranfer between the nodes is much higher, than if the stripes are well distributed. But this is exactly because the stripes are created independent from the blocks. And it also is the key to optimize the stripe size (together with your use pattern). ... View more

posted my answer in parallel, without noticing yours. sorry for the redundant info. ... View more

Are you looking for the scheduling or how to script it? Shall the files be copied to hdfs as soon as they arrive, or in a special frequency, ie. daily, hourly, etc... The best way depends on the tools and knowledge you have. It could be done with a plain shell script, but also with nifi. Spark has also a FTP connector. Here is a post on how to solve it with nifi: https://community.hortonworks.com/questions/70261/how-to-read-data-from-a-file-from-remote-ftp-serve.html ... View more

On the functional level you will have the logic for the ORC file (stripes and indexes) independent from the distribution of the blocks in hdfs. So on application level you simply read the ORC file and write the data to the next ORC file. The application handling the ORC format knows at which position/byte in the file the data is stored, while hdfs nows which bytes of the files are stored on which node. If you now read the complete file, all Nodes will deliver their block. If this data is then just stored again in a hdfs file hdfs will typically decide how many blocks it will use and distribute it on the nodes. So the data gets transfered from the source nodes to the target nodes. This is typically transparent to the user. The application will have to take care if the stripes are differently defined to write the data correctly into the ORC file. ... View more

Perhaps some more details could help. Do you want to query the data after you created a table in hive? ... View more