Are there any concerns with turning on HDFS snapshots on the Apps/Hive/Warehouse directory? I want to make sure the directory is recoverable if someone were to accidentally rmr it (or if anything else happened). https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-hdfs/HdfsSnapshots.html
... View more
@Saptak Sen Why try Spark 1.5.1 when you can try 1.5.2? Spark is sort of like a client, so you can easily run multiple versions of Spark on one cluster. Simply download Spark from their the apache website - https://spark.apache.org/downloads.html - and invoke it to run it (can configure Zeppelin to use it or just run via the CLI).
... View more
@snukavarapu It depends on how big the files are you are loading into HDFS. If the files are very big, a bigger block size would provide higher throughput. Bigger blocks would also equate to less blocks in HDFS which would reduce the load on the namenode. You can specify blocksize for particular files too: hadoop fs -D fs.local.block.size=134217728 -put local_name remote_location
... View more
Hi All, Does anyone have any steps for installing Flink on HDP? And any tutorials/examples of using Flink? Maybe with Hive ORC tables? Thanks,
... View more
Hi All, While ORC and Parquet are both columnar data stores that are supported in HDP, I was wondering if there was additional guidance on when to use one over the other? Or things to consider before choosing which format to use? Thanks, Andrew
... View more
