Thanks a lot Tamil Selvan K for this article, it's very useful. I have a question please and I hope find the best answer, I should do a permission to a database and not a tables. I have not kerberos cluster Hadoop . I manage the permission hive, hdfs via Ranger . The Resource Path in Ranger for HDFS are: /user/myLogin /apps/hive/warehouse/mylogin_* /apps/hive/warehouse I can create a database in hive ( via console) also in Ambari. But when I remove the permission /apps/hive/warehouse I can't create a database in Hive (Console) but in Ambari I can create it. This following the error: hive> create database database_tesst; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTa sk. MetaException(message:org.apache.hadoop.security.AccessControlException: Permission denied: user=AAAAA, access=EXECUTE, inode="/apps/hive/warehouse/database_tesst.db":hdfs:hdfs:d--------- at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPe How can I create a database or runing a request in hive (console) without the permission /apps/hive/warehouse ? Because I should remove this permission from Ranger to allow access users only to there data. ... View more

@Constantin Stanca Thank you for your answer, I'm sorry for this deplay in response. In fact, the port 8080 is reserved for Ambari, http://localhost:8080, this is a print-screen: ... View more

Thak you for your answer. Yes I checked on availability of the port 4040 is reserved: the result of checking is the following: Active connexion local address Remote address state TCP 127.0.0.1:4040 DESKTOP-FF4U.....:0 Listening ... View more

I logged by http://localhost:18080/, it display this result, I think that I should change the logging directory. 1.4.1 History Server Timeline Service Location: http://sandbox.hortonworks.com:8188/ Last Updated: Dec 24, 2016 7:22:57 PM UTC Service Started: Dec 24, 2016 12:22:54 PM UTC Current Time: Dec 24, 2016 7:23:01 PM UTC Timeline Service: Timeline service is enabled History Provider: Apache Hadoop YARN Timeline Service No completed applications found! Did you specify the correct logging directory? Please verify your setting of spark.history.fs.logDirectory and whether you have the permissions to access it. It is also possible that your application did not run to completion or did not stop the SparkContext. Show incomplete applications ... View more

@Rajkumar Singh I checked it in cmd : ping 10.0.2.15, this is the result: Send a request 'Ping' 10.0.2.15 with 32 bytes of data Waiting time exceeded Waiting time exceeded Waiting time exceeded Waiting time exceeded Ping statistics for 10.0.2.15: Packages: sent = 4, received = 0, lost = 4 (100% loss) I can't access by this address. ... View more

@Rajkumar Singh in the running it display like this: INFO BlockManagerInfo: Added broadcast_20_piece0 in memory on 10.0.2.15:44895 (size: 6.7 KB, free: 348.1 MB) INFO SparkUI: Stopped Spark web UI at http://10.0.2.15:4040 My code is running succesfully just it take a lot time. I tried by http://10.0.15:4040 and also I changed the port 4041 and 4042 to acces on the application logs, but the page is inaccessible. you told me can be access from this URL : http://:18080/>:18080/ but I don't have the port 18080 and how I can know the <hostname_spark_history_server> ? Thank you. ... View more

Sorry no, this small code take nearly 30sec in running without Thread.sleep(...) . You can see in this following the result of the running, I took just the half or more, execuse me for this reply I think it's very long, but really I can't understand why all this display for a small code, may be an error in the configuration in Hadoop or Spark? Please can you tell me why all this, I remark that the executor it run many times, it's normaly ? Thank you. 16/12/24 16:29:21 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 16/12/24 16:29:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 16/12/24 16:29:21 INFO Executor: Finished task 1.0 in stage 11.0 (TID 23). 1689 bytes result sent to driver 16/12/24 16:29:21 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 20 ms 16/12/24 16:29:23 INFO DAGScheduler: looking for newly runnable stages 16/12/24 16:29:23 INFO DAGScheduler: running: Set() 16/12/24 16:29:23 INFO DAGScheduler: waiting: Set(ShuffleMapStage 15, ShuffleMapStage 12, ShuffleMapStage 16, ShuffleMapStage 13, ShuffleMapStage 17, ResultStage 18, ShuffleMapStage 14) 16/12/24 16:29:24 INFO Executor: Running task 0.0 in stage 12.0 (TID 26) 16/12/24 16:29:24 INFO Executor: Running task 1.0 in stage 12.0 (TID 27) 16/12/24 16:29:24 INFO Executor: Running task 2.0 in stage 12.0 (TID 28) 16/12/24 16:29:24 INFO Executor: Running task 3.0 in stage 12.0 (TID 29) 16/12/24 16:29:24 INFO BlockManager: Found block rdd_16_1 locally 16/12/24 16:29:24 INFO BlockManager: Found block rdd_15_1 locally 16/12/24 16:29:24 INFO BlockManager: Found block rdd_16_2 locally 16/12/24 16:29:24 INFO BlockManager: Found block rdd_15_2 locally 16/12/24 16:29:24 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 16/12/24 16:29:24 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 16/12/24 16:29:24 INFO BlockManager: Found block rdd_16_3 locally 16/12/24 16:29:24 INFO Executor: Running task 1.0 in stage 13.0 (TID 31) 16/12/24 16:29:24 INFO Executor: Running task 0.0 in stage 13.0 (TID 30) 16/12/24 16:29:24 INFO Executor: Running task 2.0 in stage 13.0 (TID 32) 16/12/24 16:29:24 INFO Executor: Running task 3.0 in stage 13.0 (TID 33) 16/12/24 16:29:24 INFO BlockManager: Found block rdd_21_0 locally 16/12/24 16:29:24 INFO BlockManager: Found block rdd_20_0 locally 16/12/24 16:29:24 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 16/12/24 16:29:24 INFO BlockManager: Found block rdd_20_2 locally 16/12/24 16:29:24 INFO Executor: Finished task 3.0 in stage 13.0 (TID 33). 1689 bytes result sent to driver 16/12/24 16:29:24 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 16/12/24 16:29:25 INFO TaskSetManager: Starting task 1.0 in stage 14.0 (TID 35, localhost, partition 1, PROCESS_LOCAL, 5377 bytes) 16/12/24 16:29:25 INFO Executor: Running task 0.0 in stage 14.0 (TID 34) 16/12/24 16:29:25 INFO Executor: Running task 2.0 in stage 14.0 (TID 36) 16/12/24 16:29:25 INFO Executor: Running task 3.0 in stage 14.0 (TID 37) 16/12/24 16:29:25 INFO BlockManager: Found block rdd_16_1 locally sk 3.0 in stage 16.0 (TID 45, localhost, partition 3, PROCESS_LOCAL, 5377 bytes) 16/12/24 16:29:26 INFO Executor: Running task 0.0 in stage 16.0 (TID 42) 16/12/24 16:29:26 INFO Executor: Running task 1.0 in stage 16.0 (TID 43) 16/12/24 16:29:26 INFO Executor: Running task 2.0 in stage 16.0 (TID 44) 16/12/24 16:29:29 INFO Executor: Running task 3.0 in stage 16.0 (TID 45) 16/12/24 16:29:29 INFO BlockManagerInfo: Removed broadcast_13_piece0 on 10.0.2.15:46654 in memory (size: 6.0 KB, free: 348.1 MB) in memory (size: 5.5 KB, free: 348.1 MB) 16/12/24 16:29:29 INFO Executor: Running task 0.0 in stage 18.0 (TID 50) 16/12/24 16:29:29 INFO Executor: Running task 2.0 in stage 18.0 (TID 52) 16/12/24 16:29:29 INFO Executor: Running task 3.0 in stage 18.0 (TID 53) 16/12/24 16:29:29 INFO Executor: Running task 1.0 in stage 18.0 (TID 51) 16/12/24 16:29:30 INFO BlockManager: Found block rdd_15_3 locally 16/12/24 16:29:30 INFO BlockManager: Found block rdd_15_3 locally 16/12/24 16:29:30 INFO BlockManager: Found block rdd_15_1 locally 16/12/24 16:29:30 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 16/12/24 16:29:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 1 ms 16/12/24 16:29:30 INFO BlockManager: Found block rdd_15_1 locally 16/12/24 16:29:30 INFO BlockManager: Found block rdd_15_2 locally 16/12/24 16:29:30 INFO BlockManager: Found block rdd_15_2 locally 16/12/24 16:29:30 INFO ShuffleBlockFetcherIterator: Getting 1 non-empty blocks out of 4 blocks 16/12/24 16:29:30 INFO Executor: Running task 1.0 in stage 30.0 (TID 55) 16/12/24 16:29:30 INFO Executor: Running task 2.0 in stage 30.0 (TID 56) 16/12/24 16:29:30 INFO Executor: Running task 0.0 in stage 30.0 (TID 54) 16/12/24 16:29:30 INFO Executor: Running task 3.0 in stage 30.0 (TID 57) 16/12/24 16:29:30 INFO BlockManager: Found block rdd_20_1 locally 16/12/24 16:29:30 INFO BlockManager: Found block rdd_20_1 locally 16/12/24 16:29:30 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 0 ms 16/12/24 16:29:30 INFO MemoryStore: Block rdd_107_1 stored as values in memory (estimated size 16.0 B, free 347.7 MB) 16/12/24 16:29:38 INFO MemoryStore: MemoryStore cleared 16/12/24 16:29:38 INFO BlockManager: BlockManager stopped 16/12/24 16:29:38 INFO BlockManagerMaster: BlockManagerMaster stopped 16/12/24 16:29:38 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 16/12/24 16:29:39 INFO SparkContext: Successfully stopped SparkContext [success] Total time: 164 s, completed Dec 24, 2016 4:29:42 PM 16/12/24 16:29:42 INFO ShutdownHookManager: Shutdown hook called 16/12/24 16:29:42 INFO ShutdownHookManager: Deleting directory /tmp/spark-1d6bc2e8-5756-4107-82ee-8950cb6c5875 [root@sandbox projectFilms]# ... View more

@Rajkumar Singh this following my small code: /* My first application in Spark scala has an idea to build a system of recommandation for the users */ import java.io.File import org.apache.spark.SparkContext import org.apache.spark.SparkContext._ import org.apache.spark.SparkConf import org.apache.spark.rdd._ import org.apache.spark.mllib.recommendation.ALS import org.apache.spark.mllib.recommendation.Rating import org.apache.spark.mllib.recommendation.MatrixFactorizationModel object appFilms { val conf = new SparkConf().setAppName("system of recommandation").setMaster("local[4]") val sc = new SparkContext(conf) def main(args: Array[String]) { // Load file rating and parse it val Ratingfiles = sc.textFile("hdfs://sandbox.hortonworks.com:8020/tmp/ProjetFilm/ratings.dat").map {line => val splitRating = line.split("::").toString Rating(splitRating(0).toInt, splitRating(1).toInt, splitRating(2).toDouble) } //load file rating and parse it val Moviefile = sc.textFile("hdfs://sandbox.hortonworks.com:8020/tmp/ProjetFilm/movies.dat").map { line => val splitMovie = line.split("::").toString (splitMovie(0).toInt, splitMovie(1).toString) } //Split the Rating files into three parties val training_RDD = Ratingfiles.filter(x => x.rating < 6).cache() val validation_RDD = Ratingfiles.filter(x => x.rating >= 6 && x.rating < 8).cache() val test_RDD = Ratingfiles.filter(x => x.rating > 8).cache() // the model of training val rank = 10 val numIteration = 4 val model = ALS.train(test_RDD, rank, numIteration, 0.01) //val result = model.save(sc, "hdfs://sandbox.hortonworks.com:8020/tmp/ProjetFilm/Model") println("helooooooooooo") sc.stop() } } ... View more

Thank you for your answer. I'm a student and i'm beginner in Spark (not just I use Spark 1.4.1 but I am running in HDP 2.3.4 -:) ). Can you tell me please how I can set the number of cores on my machine, this is from Ambari ? I run my code in a console when I write : ./sbin/start-history-server.sh it display: -bash: cd: ./sbin/start-history-server.sh: No such file or directory ... View more