Answers by @Sagar Shimpi and @Lester Martin look pretty good to me. Some further explanations: How does snapshots help for Disaster Recovery? What are the best practices around using snapshots for DR purposes? Especially trying to understand when data is directly stored on HDFS, Hive data and HBase data If you're using the current distcp for DR (i.e., using distcp copying data from one cluster to your backup cluster), you have an option to utilize snapshot to do incremental backup so as to improve the distcp performance/efficiency. More specifically, you can choose to take snapshots in both the source and the backup cluster and use -diff option for the distcp command. Then instead of blindly copying all the data, the distcp will first compute the difference between the given snapshots, and only copy the difference to the backup cluster. As I understand, no data is copied for snapshots, but only metadata is maintained for the blocks added/ modified / deleted. If that’s the case, just wondering what happens when the comamnd hdfs dfs -rm /data/snapshot-dir/file1 is run. Will the file be moved to the trash? If so, will the snapshot maintain the reference to the entry in trash? Will trach eviction has any impact in this case? Yes, if you have not skipped the trash, the file will be moved to the trash, and in the meanwhile, you can still access the file using the corresponding snapshot path. How does snapshots work along with HDFS quotas. For example, assume a directory with a quota of 1GB with snapshotting enabled. Assume the directory is closer to its full quota and a user deleted a large file to store some other dataset. Will the new data be allowed to be saved to the directory or will the operation be stopped because the quota limits have been exceeded? No, if the file belongs to the snapshot (i.e., the file was created before a snapshot was taken), you will not release quota by deleting it. You may have to delete some old snapshots or increase your quota limit. Also in some old hadoop versions you may find the snapshots also affect the namespace quota usage in a strange way, i.e., sometimes deleting a file can increase the quota usage. This has been fixed by the latest version of HDP. ... View more

We kind of built a data warehouse around the same idea that you have talked about in your article. Integrating Salesforce and Google analytics as data-warehouse @infocaptor http://www.infocaptor.com The benefit is you can also co-relate with your financial data When you design using GA api, you need to load the initial historical data for a certain date range. This has its own complications as you might run into segmentation issues, loss of data etc. You need to handle pagination etc. Once the initial data load is complete, you then run it in incremental mode where you just bring new data only. This data gets appended to the same Data warehouse tables and does not cause duplicate with overlapping dates. The minimum you would need to design is some kind of background daemon that runs everyday or at some frequency. You will need job tables to monitor the success and failure of the extracts so that it can resume from where the error occurred. Some of the other considerations 1. What happens if you run the extract for the same data range 2. What if a job fails for certain dates It is important to set your primary keys for your DW target tables. The extracted data is stored as CSV files and these can be easily pushed to Hadoop file system. ... View more

Other very good ways to load data into HDFS is using Flume or Nifi. "Hadoop fs put" is good but it has some limitation or lack of flexibility that might make it difficult to use it in a production environment. If you look at the documentation of the Flume HDFS sink for instance ( http://flume.apache.org/FlumeUserGuide.html#hdfs-sink ), you'll see that Flume lets you define how to rotate the files, how to write the file names etc. Other options can be defined for the source (your local text files) or for the channel. "Hadoop fs put" is more basic and doesn't offer those possibilities. ... View more

Is this method correct to specify the path of the hdfs file. Is this same for everyone ... View more

Can someone please help me with the below error while moving a file from local drive to hdp? [root@sandbox-hdp ~]# ssh root@127.0.0.1 -p 22 The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established. RSA key fingerprint is 9c:3a:83:80:2c:d5:1f:a9:41:48:68:96:4d:0f:bb:ed. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '127.0.0.1' (RSA) to the list of known hosts. root@127.0.0.1's password: Last login: Fri Dec 15 17:45:44 2017 from 127.0.0.1 [root@sandbox-hdp ~]# scp -P 22 /Users/me/test/Input/movies_initial.csv.gz root@sandbox-hdp.hortonworks.com:root/Input root@sandbox-hdp.hortonworks.com's password: /Users/me/test/Input/movies_initial.csv.gz: No such file or directory , I am facing the below error while moving a file from local directory(Mac) to Hortonworks. Can someone please help? [root@sandbox-hdp ~]# ssh root@127.0.0.1 -p 22 The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established. RSA key fingerprint is 9c:3a:83:80:2c:d5:1f:a9:41:48:68:96:4d:0f:bb:ed. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '127.0.0.1' (RSA) to the list of known hosts. root@127.0.0.1's password: Last login: Fri Dec 15 17:45:44 2017 from 127.0.0.1 [root@sandbox-hdp ~]# scp -P 22 /Users/me/Hadoop/Input/movies_initial.csv.gz root@sandbox-hdp.hortonworks.com:root/Input root@sandbox-hdp.hortonworks.com's password: /Users/me/Hadoop/Input/movies_initial.csv.gz: No such file or directory ... View more

Well, it seems 600 characters wasn't enough for to supply enough of the back story @Peter Coates, so I added some high-level details in my original answer. I'd be glad to share more fine-grained info via email if you'd like. ... View more

If you keep using `globals()` you will eventually get an error, as it keeps adding itself to itself and you eventually get one of the following errors: RuntimeError: maximum recursion depth exceeded while getting the repr of a list or RuntimeError: dictionary changed size during iteration ... View more