Sorry, could not focus on this, was busy with production activities.   Finally, I could able to run it successfully with below configurations,   Jar / py name   :  ${nameNode}/user/solution.jar Main Class       :  Module.final_solution Options List     : --conf spark.yarn.jar=local:/opt/cloudera/parcels/CDH/lib/spark/lib/spark-assembly.jar   Properties: Spark Master : yarn Mode               : cluster App name       : Final Solution ... View more

The Cloudera ODBC connector is available for Windows, and .NET does support ODBC: http://www.cloudera.com/downloads/connectors/hive/odbc.html There's no direct client for HDFS in .NET, but HDFS offers a REST API via its WebHDFS component. This REST API is documented at http://archive.cloudera.com/cdh5/cdh/5/hadoop/hadoop-project-dist/hadoop-hdfs/WebHDFS.html, and you can use your .NET HTTP client to make use of it. ... View more

Your understanding is correct. You either need to ensure flume can write to that directory, or create a directory that flume owns and can write to. -pd ... View more

The flume http source is for creating a REST API that you can post data to from the upstream sender. If you are looking for a source that will consume from SQL server via accessing the SQL server API, you'll need to write a custom source for that, or possibly try this: https://github.com/keedio/flume-ng-sql-source Additionally, if you don't need real time processing, you may want to consider using sqoop to import data via batch processing, and it can handle incremental updates. -pd ... View more

This documentation goes over stopping and starting flume when not using Cloudera Manager. This assumes you are running packages and not parcels on this edge node: http://www.cloudera.com/documentation/enterprise/latest/topics/cdh_ig_flume_run.html ... View more