Member since
12-10-2014
25
Posts
0
Kudos Received
0
Solutions
06-04-2016
08:53 AM
Hi, I just wanted to add more information about the problem that I have, please go to http://community.cloudera.com/t5/Hadoop-101-Training-Quickstart/I-cannot-access-programmatically-a-file-within-a-CDH-5-4-2-0/td-p/41464 The real problem is to get the same result than i got within the spark-shell, i mean, when i run the code using setMaster(local[4]) and ratingFile is set to "file:///home/cloudera/Downloads/ratings.csv", i am getting this message: Parsed file:///home/cloudera/Downloads/ratings.csv. Kept 0 ratings out of 568454 because if i run the exact code in the spark-shell, i got this message: Parsed file:///home/cloudera/Downloads/ratings.csv. Kept 73279 ratings out of 568454 If i use setMaster("local[4]"), it does not matter if i use the /home/cloudera/Download/ratings.csv or the one loaded within the hdfs, i am always getting the message: Parsed file:///home/cloudera/Downloads/ratings.csv. Kept 0 ratings out of 568454 I thought that using the code submitting the job to the pseudo cluster, the problem will gone, but it isn't. Please help. How can i get this spark job up and running properly (with the same results than i am getting using the spark-shell) in the pseudo distributed cdh 5.7.0 image? ¿How can i use YARN programmatically using sbt-pack?
... View more
06-04-2016
08:20 AM
Hi everyone, I have running a vmware cdh 5.7.0 with centos 6.8 image in my osx laptop, the image uses 4 cores and 8 GB of RAM, i want to run a direct spark streaming job connected to a kafka which is running within the osx host. I know that YARN is necessary to run the code within the pseudo distributed cdh 5.7.0 image and i want to use sbt-pack to create the unix command. Someone on the spark user list told me to delete the setMaster line in order to let YARN to schedule the necessary resources in order to run the process, because if i use this config: val sparkConf = new SparkConf(false).setAppName("AmazonKafkaConnector")
//.setMaster("local[4]")
.setMaster("spark://192.168.30.137:7077")
.set("spark.cores.max", "2")
.set("spark.driver.allowMultipleContexts", "true")
val sc = new SparkContext(sparkConf) I got this warning: 16/06/04 17:03:42 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.30.137:59231 (size: 2.6 KB, free: 945.5 MB)
16/06/04 17:03:42 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1006
16/06/04 17:03:42 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 0 (MapPartitionsRDD[3] at groupBy at Recommender.scala:28)
16/06/04 17:03:42 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks
16/06/04 17:03:57 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
16/06/04 17:04:12 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources the process does not work because of resources. But if i comment the setMaster line, i got this one: org.apache.spark.SparkException: A master URL must be set in your configuration
at org.apache.spark.SparkContext.<init>(SparkContext.scala:402)
at example.spark.AmazonKafkaConnector$.main(AmazonKafkaConnectorWithMongo.scala:93)
at example.spark.AmazonKafkaConnector.main(AmazonKafkaConnectorWithMongo.scala) When i see spark-master page, this is what i see: Spark Master at spark://192.168.30.137:7077
URL: spark://192.168.30.137:7077
REST URL: spark://192.168.30.137:6066 (cluster mode)
Alive Workers: 0
Cores in use: 0 Total, 0 Used
Memory in use: 0.0 B Total, 0.0 B Used
Applications: 2 Running, 2 Completed
Drivers: 0 Running, 0 Completed
Status: ALIVE
Workers
Worker Id Address State Cores Memory
Running Applications
Application ID Name Cores Memory per Node Submitted Time User State Duration
app-20160604170643-0003
(kill)
AmazonKafkaConnector 0 1024.0 MB 2016/06/04 17:06:43 cloudera WAITING 23 s
app-20160604170642-0002
(kill)
AmazonKafkaConnector 0 1024.0 MB 2016/06/04 17:06:42 cloudera WAITING 23 s And this is what i see within spark-worker page: Spark Worker at 192.168.30.137:7078
ID: worker-20160604170606-192.168.30.137-7078
Master URL:
Cores: 4 (0 Used)
Memory: 6.7 GB (0.0 B Used)
Back to Master
Running Executors (0)
ExecutorID Cores State Memory Job Details Logs I am using 1.6.0-cdh5.7.0 spark version, the exact same version than my cloudera image. This is github project, the develop tree, my question is, ¿how can i use sbt-pack to run programmatically this spark streaming process?, by the way, the code of the process looks like this: //This is part of AmazonKafkaConnectorWithMongo
val sparkConf = new SparkConf(false).setAppName("AmazonKafkaConnector")
//.setMaster("local[4]")
//Commented or decommented, i cannot have this code up and running within the cloudera image
.setMaster("spark://192.168.30.137:7077")
.set("spark.cores.max", "2")
.set("spark.driver.allowMultipleContexts", "true")
val sc = new SparkContext(sparkConf)
val sqlContext = new SQLContext(sc)
val ssc = new StreamingContext(sparkConf, Seconds(2))
//this checkpointdir should be in a conf file, for now it is hardcoded!
val streamingCheckpointDir = "/home/cloudera/my-recommendation-spark-engine/checkpoint"
ssc.checkpoint(streamingCheckpointDir)
// Create direct kafka stream with brokers and topics
val topicsSet = topics.split(",").toSet
val kafkaParams = Map[String, String]("metadata.broker.list" -> brokers)
val messages = KafkaUtils.createDirectStream[String, String, StringDecoder, StringDecoder](ssc, kafkaParams, topicsSet)
println("Initialized Streaming Spark Context and kafka connector...")
//create recomendation module
println("Creating rating recommender module...")
val ratingFile= "hdfs://192.168.30.137:8020/user/cloudera/ratings.csv"
val recommender = new Recommender(sc,ratingFile)
...
This is part of Recommender class:
val NumRecommendations = 10
val MinRecommendationsPerUser = 10
val MaxRecommendationsPerUser = 20
val MyUsername = "myself"
val NumPartitions = 20
@transient val random = new Random() with Serializable
println("Using this ratingFile: " + ratingFile)
// first create an RDD out of the rating file
val rawTrainingRatings = sc.textFile(ratingFile).map {
line =>
val Array(userId, productId, scoreStr) = line.split(",")
AmazonRating(userId, productId, scoreStr.toDouble)
}
// only keep users that have rated between MinRecommendationsPerUser and
//THIS IS WHAT PROVOKES THE WARNING ABOUT LACK OF RESOURCES!
MaxRecommendationsPerUser products
val trainingRatings = rawTrainingRatings.groupBy(_.userId)
.filter(r => MinRecommendationsPerUser <= r._2.size && r._2.size < MaxRecommendationsPerUser)
.flatMap(_._2)
.repartition(NumPartitions)
.cache()
println(s"Parsed $ratingFile. Kept ${trainingRatings.count()} ratings out of ${rawTrainingRatings.count()}")
I already test this code within a spark scala shell and it works!, but it does not work when running using sbt-pack unix command. Please help, i am stuck with this issue during the last week, the file that i am trying to use as ratingFile is only 16MB, it cannot be a matter of provided resources, isnt ? Thank yoy for reading until here.
... View more
06-04-2016
07:49 AM
Thank you Sean, but the link that you have provided are returning this message: The core node you are trying to access was not found, it may have been deleted. Please refresh your original page and try the operation again. EDIT, now i have noticed that there is an extra ")".
... View more
06-03-2016
02:40 AM
Hi Sean, doing this provokes that spark-worker doesn't run and Hue does not work properly, i think that it needs internally to use quickstart.cloudera. If i restart again the vmware image and i redo again what i did within /etc/init.d/cloudera-quickstart-init with the call to cloudera-quickstart-ip, i can login again to Hue, and spark-history-manager runs properly: I have noticed that running a ps xa | grep spark... [cloudera@quickstart ~]$ ps xa | grep spark
6330 ? Sl 0:03 /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/lib/spark/conf/:/usr/lib/spark/lib/spark-assembly-1.6.0-cdh5.7.0-hadoop2.6.0-cdh5.7.0.jar:/etc/hadoop/conf/:/usr/lib/spark/lib/spark-assembly.jar:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hive/lib/*:/usr/lib/flume-ng/lib/*:/usr/lib/paquet/lib/*:/usr/lib/avro/lib/* -Dspark.deploy.defaultCores=4 -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.master.Master
6499 ? Sl 0:04 /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /usr/lib/spark/conf/:/usr/lib/spark/lib/spark-assembly-1.6.0-cdh5.7.0-hadoop2.6.0-cdh5.7.0.jar:/etc/hadoop/conf/:/usr/lib/spark/lib/spark-assembly.jar:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hive/lib/*:/usr/lib/flume-ng/lib/*:/usr/lib/paquet/lib/*:/usr/lib/avro/lib/* -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.worker.Worker spark://quickstart.cloudera:7077
6674 ? Sl 0:05 /usr/java/jdk1.7.0_67-cloudera/bin/java -cp /etc/spark/conf/:/usr/lib/spark/lib/spark-assembly-1.6.0-cdh5.7.0-hadoop2.6.0-cdh5.7.0.jar:/etc/hadoop/conf/:/usr/lib/spark/lib/spark-assembly.jar:/usr/lib/hadoop/lib/*:/usr/lib/hadoop/*:/usr/lib/hadoop-hdfs/lib/*:/usr/lib/hadoop-hdfs/*:/usr/lib/hadoop-mapreduce/lib/*:/usr/lib/hadoop-mapreduce/*:/usr/lib/hadoop-yarn/lib/*:/usr/lib/hadoop-yarn/*:/usr/lib/hive/lib/*:/usr/lib/flume-ng/lib/*:/usr/lib/paquet/lib/*:/usr/lib/avro/lib/* -Dspark.history.fs.logDirectory=hdfs:///user/spark/applicationHistory -Dspark.history.ui.port=18088 -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.history.HistoryServer
6915 pts/0 R+ 0:00 grep spark As you can see, spark-master runs with 4 cores (-Dspark.deploy.defaultCores=4) with no dedicated cores to worker, is it normal? [cloudera@quickstart ~]$ sudo service spark-worker status
Spark worker is running [ OK ]
[cloudera@quickstart ~]$ sudo service spark-master status
Spark master is running [ OK ]
[cloudera@quickstart ~]$ sudo service spark-history-server status
Spark history-server is running [ OK ] As you can see, it looks normal, but examining http://quickstart.cloudera:18080/ , the spark-master, i can see that URL: spark://192.168.30.137:7077
REST URL: spark://192.168.30.137:6066 (cluster mode)
Alive Workers: 0
Cores in use: 0 Total, 0 Used
Memory in use: 0.0 B Total, 0.0 B Used
Applications: 0 Running, 0 Completed
Drivers: 0 Running, 0 Completed
Status: ALIVE zero cores in use from zero in total! with no memory, that´s strange, because spark-master tries to use 4 cores (-Dspark.deploy.defaultCores=4) and 1GB (-Xms1g -Xmx1g -XX:MaxPermSize=256m) Then, you can see this output from spark-worker: ID: worker-20160603111341-192.168.30.137-7078
Master URL:
Cores: 4 (0 Used)
Memory: 6.7 GB (0.0 B Used) the master URL is not setup and it is using 4 cores with 6.7 GB when spark-worker is running with this setup: -Xms1g -Xmx1g -XX:MaxPermSize=256m org.apache.spark.deploy.worker.Worker spark://quickstart.cloudera:7077 What do you think, what can i do in order to continue developing my project? because, that is what i want, use this vmware image to develop this project using the hdfs to load a tiny file, only 16MB. What most annoys me is that the code works perfectly in the spark-shell of the virtual image but when I try to make it run programmatically creating the unix with SBT-pack command does not work. Regards Alonso
... View more
06-03-2016
02:11 AM
Hi Sean, that change does not have any effect, spark-worker doesn't run, even if i try to restart it manually with sudo service spark-worker restart, it failed as soon as i ask by its status. Also, Hue does not work, i think it happens because internally uses quickstart.cloudera in order to talk with another component... I am starting to think that this vmware image is useless to develop something related with spark, i cannot run anything...
... View more
06-03-2016
01:44 AM
Hi Sean this is how it looks /etc/hosts by default, when the image is restarted: [cloudera@quickstart ~]$ cat /etc/hosts 127.0.0.1 quickstart.cloudera quickstart localhost localhost.domain
... View more
06-02-2016
11:56 AM
I wanted to say, i do not know what to do next. I have readed this blog post, http://www.datastax.com/dev/blog/common-spark-troubleshooting, using that post i figure out to modify programmatically this variable: .set("spark.cores.max", "2") and modify /etc/spark/conf/spark-env.sh to put this export: export SPARK_MASTER_OPTS="-Dspark.deploy.defaultCores=4" Is there anything i can do in order to run a simple spark job with a file of 16MB? another thing i noticed is that i HAVE to put this programmatically in the sparkconf in order to run the job: .set("spark.driver.allowMultipleContexts", "true") Please help...
... View more
06-02-2016
11:51 AM
Hi Sean, i am adding more research about this issue. I have added programmatically this line to SparkConf .set("spark.cores.max", "2") and this line to /etc/spark/conf/spark-env.sh ##ADDED!!
export SPARK_MASTER_OPTS="-Dspark.deploy.defaultCores=4" with no luck, if i i check /var/log/spark-worker.out file, these are last 50 lines: 16/06/02 20:47:43 INFO worker.Worker: Retrying connection to master (attempt # 11)
16/06/02 20:47:43 INFO worker.Worker: Connecting to master quickstart.cloudera:7077...
16/06/02 20:47:43 WARN worker.Worker: Failed to connect to master quickstart.cloudera:7077
java.io.IOException: Failed to connect to quickstart.cloudera/127.0.0.1:7077
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:199)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:190)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:186)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: quickstart.cloudera/127.0.0.1:7077
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) the worker cannot talk to master. I starting to think that this vmware image running with 8GB and 4 cores is not enough to run a simple spark job programmatically, but why i can run the same code within a spark-shell? Please, help, i do not want what to do...
... View more
06-02-2016
09:45 AM
Hi Sean, if i update /etc/hosts removing the line with 127.0.0.1 and using a line with the assigned ip instead of 127.0.0.1, i am getting this exception: Caused by: java.io.IOException: org.apache.spark.SparkException: Failed to get broadcast_1_piece0 of broadcast_1
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1223)
at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165)
at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64)
at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88)
at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:65)
at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:89)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.SparkException: Failed to get broadcast_1_piece0 of broadcast_1
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1$$anonfun$2.apply(TorrentBroadcast.scala:138)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply$mcVI$sp(TorrentBroadcast.scala:137)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$org$apache$spark$broadcast$TorrentBroadcast$$readBlocks$1.apply(TorrentBroadcast.scala:120)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.apache.spark.broadcast.TorrentBroadcast.org$apache$spark$broadcast$TorrentBroadcast$$readBlocks(TorrentBroadcast.scala:120)
at org.apache.spark.broadcast.TorrentBroadcast$$anonfun$readBroadcastBlock$1.apply(TorrentBroadcast.scala:175)
at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1220) I I am pretty sure that i can reach to my host machine and i can reach to vmware image from my host. host -> vmware image MacBook-Pro-Retina-de-Alonso:my-recommendation-spark-engine aironman$ ping 192.168.30.139
PING 192.168.30.139 (192.168.30.139): 56 data bytes
64 bytes from 192.168.30.139: icmp_seq=0 ttl=64 time=0.462 ms
64 bytes from 192.168.30.139: icmp_seq=1 ttl=64 time=0.353 ms
64 bytes from 192.168.30.139: icmp_seq=2 ttl=64 time=0.311 ms
64 bytes from 192.168.30.139: icmp_seq=3 ttl=64 time=0.310 ms
64 bytes from 192.168.30.139: icmp_seq=4 ttl=64 time=0.273 ms vmware image -> host [cloudera@quickstart bin]$ ping 192.168.1.35
PING 192.168.1.35 (192.168.1.35) 56(84) bytes of data.
64 bytes from 192.168.1.35: icmp_seq=1 ttl=128 time=0.272 ms
64 bytes from 192.168.1.35: icmp_seq=2 ttl=128 time=0.416 ms
64 bytes from 192.168.1.35: icmp_seq=3 ttl=128 time=0.348 ms
64 bytes from 192.168.1.35: icmp_seq=4 ttl=128 time=0.367 ms
64 bytes from 192.168.1.35: icmp_seq=5 ttl=128 time=0.432 ms It is not a problem of the software trying to talk with the kafka server located in host machine, i can do a telnet to host machine and kafka port: [cloudera@quickstart bin]$ telnet 192.168.1.35 9092
Trying 192.168.1.35...
Connected to 192.168.1.35.
Escape character is '^]'.
Connection closed by foreign host. Please, help me Sean, i am desperate, i cannot run nothing in this vmware image...
... View more
06-01-2016
08:50 AM
Updating thread with relevant update (06/01/2016): I have a vmware cloudera image, cdh-5.7 running with centos6.8, i am using OS X as my development machine, and the cdh image to run the code. UPDATE This is the build.sbt that i am currently using, i just have updated spark version from official (1.6.1) to 1.6.0-cdh5.7.0: [cloudera@quickstart awesome-recommendation-engine]$ cat build.sbt
name := "my-recommendation-spark-engine"
version := "1.0-SNAPSHOT"
scalaVersion := "2.10.4"
val sparkVersion = "1.6.0-cdh5.7.0"
val akkaVersion = "2.3.11" // override Akka to be this version to match the one in Spark
libraryDependencies ++= Seq(
"org.apache.kafka" % "kafka_2.10" % "0.8.1"
exclude("javax.jms", "jms")
exclude("com.sun.jdmk", "jmxtools")
exclude("com.sun.jmx", "jmxri"),
// HTTP client to request data to Amazon
"net.databinder.dispatch" %% "dispatch-core" % "0.11.1",
// HTML parser
"org.jodd" % "jodd-lagarto" % "3.5.2",
"com.typesafe" % "config" % "1.2.1",
"com.typesafe.play" % "play-json_2.10" % "2.4.0-M2",
"org.scalatest" % "scalatest_2.10" % "2.2.1" % "test",
"org.twitter4j" % "twitter4j-core" % "4.0.2",
"org.twitter4j" % "twitter4j-stream" % "4.0.2",
"org.codehaus.jackson" % "jackson-core-asl" % "1.6.1",
"org.scala-tools.testing" % "specs_2.8.0" % "1.6.5" % "test",
"org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.0-cdh5.7.0",
"org.apache.spark" % "spark-core_2.10" % "1.6.0-cdh5.7.0",
"org.apache.spark" % "spark-streaming_2.10" % "1.6.0-cdh5.7.0",
"org.apache.spark" % "spark-sql_2.10" % "1.6.0-cdh5.7.0",
"org.apache.spark" % "spark-mllib_2.10" % "1.6.0-cdh5.7.0",
"com.google.code.gson" % "gson" % "2.6.2",
"commons-cli" % "commons-cli" % "1.3.1",
"com.stratio.datasource" % "spark-mongodb_2.10" % "0.11.1",
// Akka
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion,
// MongoDB
"org.reactivemongo" %% "reactivemongo" % "0.10.0"
)
packAutoSettings
resolvers ++= Seq(
"JBoss Repository" at "http://repository.jboss.org/nexus/content/repositories/releases/",
"Spray Repository" at "http://repo.spray.cc/",
"Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/",
"Akka Repository" at "http://repo.akka.io/releases/",
"Twitter4J Repository" at "http://twitter4j.org/maven2/",
"Apache HBase" at "https://repository.apache.org/content/repositories/releases",
"Twitter Maven Repo" at "http://maven.twttr.com/",
"scala-tools" at "https://oss.sonatype.org/content/groups/scala-tools",
"Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/",
"Second Typesafe repo" at "http://repo.typesafe.com/typesafe/maven-releases/",
"Mesosphere Public Repository" at "http://downloads.mesosphere.io/maven",
Resolver.sonatypeRepo("public")
) This is my /etc/hosts file located in the cdh image with a line like this: 127.0.0.1 quickstart.cloudera quickstart localhost localhost.domain The cloudera version that i am running is: [cloudera@quickstart bin]$ cat /usr/lib/hadoop/cloudera/cdh_version.properties
# Autogenerated build properties
version=2.6.0-cdh5.7.0
git.hash=c00978c67b0d3fe9f3b896b5030741bd40bf541a
cloudera.hash=c00978c67b0d3fe9f3b896b5030741bd40bf541a
cloudera.cdh.hash=e7465a27c5da4ceee397421b89e924e67bc3cbe1
cloudera.cdh-packaging.hash=8f9a1632ebfb9da946f7d8a3a8cf86efcdccec76
cloudera.base-branch=cdh5-base-2.6.0
cloudera.build-branch=cdh5-2.6.0_5.7.0
cloudera.pkg.version=2.6.0+cdh5.7.0+1280
cloudera.pkg.release=1.cdh5.7.0.p0.92
cloudera.cdh.release=cdh5.7.0
cloudera.build.time=2016.03.23-18:30:29GMT I can do a ls command in the vmware machine: [cloudera@quickstart ~]$ hdfs dfs -ls /user/cloudera/ratings.csv
-rw-r--r-- 1 cloudera cloudera 16906296 2016-05-30 11:29 /user/cloudera/ratings.csv I can read its content: [cloudera@quickstart ~]$ hdfs dfs -cat /user/cloudera/ratings.csv | wc -l
568454 The code is quite simple, just trying to map its content: val ratingFile="hdfs://192.168.1.40:8020/user/cloudera/ratings.csv"
//where 192.168.1.40 is the eth0 assigned ip of cloudera image
case class AmazonRating(userId: String, productId: String, rating: Double)
val NumRecommendations = 10
val MinRecommendationsPerUser = 10
val MaxRecommendationsPerUser = 20
val MyUsername = "myself"
val NumPartitions = 20
println("Using this ratingFile: " + ratingFile)
// first create an RDD out of the rating file
val rawTrainingRatings = sc.textFile(ratingFile).map {
line =>
val Array(userId, productId, scoreStr) = line.split(",")
AmazonRating(userId, productId, scoreStr.toDouble)
}
// only keep users that have rated between MinRecommendationsPerUser and MaxRecommendationsPerUser products
val trainingRatings = rawTrainingRatings.groupBy(_.userId).filter(r => MinRecommendationsPerUser <= r._2.size && r._2.size < MaxRecommendationsPerUser).flatMap(_._2).repartition(NumPartitions).cache()
println(s"Parsed $ratingFile. Kept ${trainingRatings.count()} ratings out of ${rawTrainingRatings.count()}") I am getting this message: **16/06/01 17:20:04 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources** because if i run the exact code within the spark-shell, i got this message: Parsed hdfs://quickstart.cloudera:8020/user/cloudera/ratings.csv. Kept 73279 ratings out of 568454 Why is it working fine within the spark-shell but it is not programmatically running in the vmware image? UPDATE I am running the code using sbt-pack plugin to generate unix commands and run them within the vmware image which has the spark pseudocluster, This is the code i use to instantiate the sparkconf: val sparkConf = new SparkConf().setAppName("AmazonKafkaConnector")
.setMaster("spark://192.168.1.40:7077") .set("spark.driver.allowMultipleContexts", "true")
val sc = new SparkContext(sparkConf)
val sqlContext = new SQLContext(sc)
val ssc = new StreamingContext(sparkConf, Seconds(2))
//this checkpointdir should be in a conf file, for now it is hardcoded!
val streamingCheckpointDir = "/home/cloudera/my-recommendation-spark-engine/checkpoint"
ssc.checkpoint(streamingCheckpointDir) I think that this must be a misconfiguration in a cloudera configuration file, but which one? UPDATE2 06/01/2016 Ok, changing the ip (192.168.1.40) instead of the fully qualified name (quickstart.cloudera) now eliminates the previous exception but now this warning arises: **16/06/01 17:20:04 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources** If i run the next commands: [cloudera@quickstart awesome-recommendation-engine]$ sudo service spark-master status
Spark master is running [ OK ]
[cloudera@quickstart awesome-recommendation-engine]$ sudo service spark-worker status
Spark worker is running [ OK ] I can see that spark-master and spark-worker are running, but when i check in 192.168.1.40:18081, the web page that checks spark-worker status, i see: ID: worker-20160601173323-192.168.1.40-7078
Master URL:
Cores: 1 (0 Used)
Memory: 2.7 GB (0.0 B Used)
Back to Master
Running Executors (0)
ExecutorID Cores State Memory Job Details Logs Nothing to show!, no ExecutorID, no Cores, no State, no Memory, nothing at all! In fact, after few minutes or less, when i run the next status command: [cloudera@quickstart awesome-recommendation-engine]$ sudo service spark-worker status
Spark worker is dead and pid file exists [FAILED]
[cloudera@quickstart awesome-recommendation-engine]$ sudo service spark-master status
Spark master is running [ OK ] I guess that i have to increase resources to the vmware image, more ram and maybe another available core from host machine, isn't? Thank you very much for reading until here.
... View more
06-01-2016
02:35 AM
Hi Sean, thanks for the answer, but, ¿what are you exactly suggesting i have to do? I mean, i have a vmware process running a cloudera image simulating a spark cluster inside of a OS X host. In this host i am running a single instance of kafka and the spark process has not any problem trying to reach to that kafka process when i use setMaster("local[*]"), but the software crashes quickly when i try to use setMaster("spark://quickstart.cloudera:7077"). An exception rises... java.io.IOException: Failed to connect to quickstart.cloudera/127.0.0.1:7077
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:200)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:183)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: quickstart.cloudera/127.0.0.1:7077
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) I was thinking that this could be related with used dependencies, this is my build.sbt: [cloudera@quickstart awesome-recommendation-engine]$ cat build.sbt
name := "my-recommendation-spark-engine"
version := "1.0-SNAPSHOT"
scalaVersion := "2.10.4"
val sparkVersion = "1.6.1"
val akkaVersion = "2.3.11" // override Akka to be this version to match the one in Spark
libraryDependencies ++= Seq(
"org.apache.kafka" % "kafka_2.10" % "0.8.1"
exclude("javax.jms", "jms")
exclude("com.sun.jdmk", "jmxtools")
exclude("com.sun.jmx", "jmxri"),
//not working play module!! check
//jdbc,
//anorm,
//cache,
// HTTP client
"net.databinder.dispatch" %% "dispatch-core" % "0.11.1",
// HTML parser
"org.jodd" % "jodd-lagarto" % "3.5.2",
"com.typesafe" % "config" % "1.2.1",
"com.typesafe.play" % "play-json_2.10" % "2.4.0-M2",
"org.scalatest" % "scalatest_2.10" % "2.2.1" % "test",
"org.twitter4j" % "twitter4j-core" % "4.0.2",
"org.twitter4j" % "twitter4j-stream" % "4.0.2",
"org.codehaus.jackson" % "jackson-core-asl" % "1.6.1",
"org.scala-tools.testing" % "specs_2.8.0" % "1.6.5" % "test",
"org.apache.spark" % "spark-streaming-kafka_2.10" % "1.6.1",
"org.apache.spark" % "spark-core_2.10" % "1.6.1",
"org.apache.spark" % "spark-streaming_2.10" % "1.6.1",
"org.apache.spark" % "spark-sql_2.10" % "1.6.1",
"org.apache.spark" % "spark-mllib_2.10" % "1.6.1",
"com.google.code.gson" % "gson" % "2.6.2",
"commons-cli" % "commons-cli" % "1.3.1",
"com.stratio.datasource" % "spark-mongodb_2.10" % "0.11.1",
// Akka
"com.typesafe.akka" %% "akka-actor" % akkaVersion,
"com.typesafe.akka" %% "akka-slf4j" % akkaVersion,
// MongoDB
"org.reactivemongo" %% "reactivemongo" % "0.10.0"
)
packAutoSettings
resolvers ++= Seq(
"JBoss Repository" at "http://repository.jboss.org/nexus/content/repositories/releases/",
"Spray Repository" at "http://repo.spray.cc/",
"Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/",
"Akka Repository" at "http://repo.akka.io/releases/",
"Twitter4J Repository" at "http://twitter4j.org/maven2/",
"Apache HBase" at "https://repository.apache.org/content/repositories/releases",
"Twitter Maven Repo" at "http://maven.twttr.com/",
"scala-tools" at "https://oss.sonatype.org/content/groups/scala-tools",
"Typesafe repository" at "http://repo.typesafe.com/typesafe/releases/",
"Second Typesafe repo" at "http://repo.typesafe.com/typesafe/maven-releases/",
"Mesosphere Public Repository" at "http://downloads.mesosphere.io/maven",
Resolver.sonatypeRepo("public")
) Could it be that i have to use the spark jars from cloudera? With this build.sbt, i am getting that exception...
... View more
05-31-2016
07:09 AM
Hi Sean, thank you for the answer, the real problem is described here, thank you very much: https://community.cloudera.com/t5/Apache-Hadoop-Concepts-and/I-cannot-access-programmatically-a-file-within-a-CDH-5-4-2-0/m-p/41464/highlight/false#M1742
... View more
05-31-2016
02:20 AM
Updating the thread with better info, why i cant modify the previous thread like in StackOverflow? I have a vmware cloudera image, cdh-5.7 running with centos6.8, i am using OS X as my development machine, and the cdh image to run the code, i have modified my /etc/hosts file located in the cdh image with a line like this: 127.0.0.1 quickstart.cloudera quickstart localhost localhost.domain
192.168.30.138 quickstart.cloudera quickstart localhost localhost.domain The cloudera version that i am running is: [cloudera@quickstart bin]$ cat /usr/lib/hadoop/cloudera/cdh_version.properties
# Autogenerated build properties
version=2.6.0-cdh5.7.0
git.hash=c00978c67b0d3fe9f3b896b5030741bd40bf541a
cloudera.hash=c00978c67b0d3fe9f3b896b5030741bd40bf541a
cloudera.cdh.hash=e7465a27c5da4ceee397421b89e924e67bc3cbe1
cloudera.cdh-packaging.hash=8f9a1632ebfb9da946f7d8a3a8cf86efcdccec76
cloudera.base-branch=cdh5-base-2.6.0
cloudera.build-branch=cdh5-2.6.0_5.7.0
cloudera.pkg.version=2.6.0+cdh5.7.0+1280
cloudera.pkg.release=1.cdh5.7.0.p0.92
cloudera.cdh.release=cdh5.7.0
cloudera.build.time=2016.03.23-18:30:29GMT I can do a ls command in the vmware machine: [cloudera@quickstart ~]$ hdfs dfs -ls /user/cloudera/ratings.csv
-rw-r--r-- 1 cloudera cloudera 16906296 2016-05-30 11:29 /user/cloudera/ratings.csv
I can read its content:
[cloudera@quickstart ~]$ hdfs dfs -cat /user/cloudera/ratings.csv | wc -l
568454 The code is quite simple, just trying to map its content: val ratingFile="hdfs://quickstart.cloudera:8020/user/cloudera/ratings.csv"
case class AmazonRating(userId: String, productId: String, rating: Double)
val NumRecommendations = 10
val MinRecommendationsPerUser = 10
val MaxRecommendationsPerUser = 20
val MyUsername = "myself"
val NumPartitions = 20
println("Using this ratingFile: " + ratingFile)
// first create an RDD out of the rating file
val rawTrainingRatings = sc.textFile(ratingFile).map {
line =>
val Array(userId, productId, scoreStr) = line.split(",")
AmazonRating(userId, productId, scoreStr.toDouble)
}
// only keep users that have rated between MinRecommendationsPerUser and MaxRecommendationsPerUser products
val trainingRatings = rawTrainingRatings.groupBy(_.userId).filter(r => MinRecommendationsPerUser <= r._2.size && r._2.size < MaxRecommendationsPerUser).flatMap(_._2).repartition(NumPartitions).cache()
println(s"Parsed $ratingFile. Kept ${trainingRatings.count()} ratings out of ${rawTrainingRatings.count()}") I am getting this message: Parsed hdfs://quickstart.cloudera:8020/user/cloudera/ratings.csv. Kept 0 ratings out of 568454 because if i run the exact code within the spark-shell, i got this message: Parsed hdfs://quickstart.cloudera:8020/user/cloudera/ratings.csv. Kept 73279 ratings out of 568454 Why is it working fine within the spark-shell but it is not programmatically running in the vmware image? UPDATE I am running the code using sbt-pack plugin to generate unix commands and run them within the vmware image which has the spark pseudocluster, This is the code i use to instantiate the sparkconf: val sparkConf = new SparkConf().setAppName("AmazonKafkaConnector")
.setMaster("local[4]") .set("spark.driver.allowMultipleContexts", "true")
val sc = new SparkContext(sparkConf)
val sqlContext = new SQLContext(sc)
val ssc = new StreamingContext(sparkConf, Seconds(2))
//this checkpointdir should be in a conf file, for now it is hardcoded!
val streamingCheckpointDir = "/home/cloudera/my-recommendation-spark-engine/checkpoint"
ssc.checkpoint(streamingCheckpointDir) I have tried to use this way of setting spark master, but an exception raises, i suspect that this is symptomatic of my problem. //.setMaster("spark://quickstart.cloudera:7077") The exception when i try to use the fully qualified domain name: .setMaster("spark://quickstart.cloudera:7077") java.io.IOException: Failed to connect to quickstart.cloudera/127.0.0.1:7077
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:216)
at org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:167)
at org.apache.spark.rpc.netty.NettyRpcEnv.createClient(NettyRpcEnv.scala:200)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:187)
at org.apache.spark.rpc.netty.Outbox$$anon$1.call(Outbox.scala:183)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.ConnectException: Connection refused: quickstart.cloudera/127.0.0.1:7077
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:224)
at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:289)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) I can ping to quickstart.cloudera in the cloudera terminal, so why i can't use .setMaster("spark://quickstart.cloudera:7077") instead of .setMaster("local[*]"): [cloudera@quickstart bin]$ ping quickstart.cloudera
PING quickstart.cloudera (127.0.0.1) 56(84) bytes of data.
64 bytes from quickstart.cloudera (127.0.0.1): icmp_seq=1 ttl=64 time=0.019 ms
64 bytes from quickstart.cloudera (127.0.0.1): icmp_seq=2 ttl=64 time=0.026 ms
64 bytes from quickstart.cloudera (127.0.0.1): icmp_seq=3 ttl=64 time=0.026 ms
64 bytes from quickstart.cloudera (127.0.0.1): icmp_seq=4 ttl=64 time=0.028 ms
64 bytes from quickstart.cloudera (127.0.0.1): icmp_seq=5 ttl=64 time=0.026 ms
64 bytes from quickstart.cloudera (127.0.0.1): icmp_seq=6 ttl=64 time=0.020 ms And the port 7077 is listening to incoming calls: [cloudera@quickstart bin]$ netstat -nap | grep 7077
(Not all processes could be identified, non-owned process info
will not be shown, you would have to be root to see it all.)
tcp 0 0 192.168.30.138:7077 0.0.0.0:* LISTEN
[cloudera@quickstart bin]$ ping 192.168.30.138
PING 192.168.30.138 (192.168.30.138) 56(84) bytes of data.
64 bytes from 192.168.30.138: icmp_seq=1 ttl=64 time=0.023 ms
64 bytes from 192.168.30.138: icmp_seq=2 ttl=64 time=0.026 ms
64 bytes from 192.168.30.138: icmp_seq=3 ttl=64 time=0.028 ms
^C
--- 192.168.30.138 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2810ms
rtt min/avg/max/mdev = 0.023/0.025/0.028/0.006 ms
[cloudera@quickstart bin]$ ifconfig
eth2 Link encap:Ethernet HWaddr 00:0C:29:6F:80:D2
inet addr:192.168.30.138 Bcast:192.168.30.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:8612 errors:0 dropped:0 overruns:0 frame:0
TX packets:8493 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2917515 (2.7 MiB) TX bytes:849750 (829.8 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:57534 errors:0 dropped:0 overruns:0 frame:0
TX packets:57534 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:44440656 (42.3 MiB) TX bytes:44440656 (42.3 MiB) I think that this must be a misconfiguration in a cloudera configuration file, but which one? Thank you very much for reading until here.
... View more
05-30-2016
10:51 AM
I have a vmware cloudera image, cdh-5.4.2.0 running with centos6, i am using OS X as my host machine, i have modified my /etc/hosts file with a line like this: MacBook-Pro-Retina-de-Alonso:bin aironman$ cat /etc/hosts
127.0.0.1 localhost
127.0.0.1 my-cassandra-node-001
255.255.255.255 broadcasthost
::1 localhost
192.168.30.137 quickstart.cloudera quickstart You can see that i can reach to the vmware machine from the host machine: $:bin aironman$ ping quickstart.cloudera
PING quickstart.cloudera (192.168.30.137): 56 data bytes
64 bytes from 192.168.30.137: icmp_seq=0 ttl=64 time=0.293 ms
64 bytes from 192.168.30.137: icmp_seq=1 ttl=64 time=0.273 ms
64 bytes from 192.168.30.137: icmp_seq=2 ttl=64 time=0.207 ms
64 bytes from 192.168.30.137: icmp_seq=3 ttl=64 time=0.240 ms
64 bytes from 192.168.30.137: icmp_seq=4 ttl=64 time=0.402 ms
^C
--- quickstart.cloudera ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.207/0.283/0.402/0.066 ms And i can reach to 8020 port from that machine: $:bin aironman$ telnet quickstart.cloudera 8020
Trying 192.168.30.137...
Connected to quickstart.cloudera.
Escape character is '^]'. I can do a ls command in the vmware machine: [cloudera@quickstart ~]$ hdfs dfs -ls /user/cloudera/ratings.csv
-rw-r--r-- 1 cloudera cloudera 16906296 2016-05-30 11:29 /user/cloudera/ratings.csv I can read its content: [cloudera@quickstart ~]$ hdfs dfs -cat /user/cloudera/ratings.csv | wc -l
568454 The code is quite simple, just trying to map its content: val ratingFile="hdfs://quickstart.cloudera:8020/user/cloudera/ratings.csv"
case class AmazonRating(userId: String, productId: String, rating: Double)
val NumRecommendations = 10
val MinRecommendationsPerUser = 10
val MaxRecommendationsPerUser = 20
val MyUsername = "myself"
val NumPartitions = 20
println("Using this ratingFile: " + ratingFile)
// first create an RDD out of the rating file
val rawTrainingRatings = sc.textFile(ratingFile).map {
line =>
val Array(userId, productId, scoreStr) = line.split(",")
AmazonRating(userId, productId, scoreStr.toDouble)
}
// only keep users that have rated between MinRecommendationsPerUser and MaxRecommendationsPerUser products
val trainingRatings = rawTrainingRatings.groupBy(_.userId).filter(r => MinRecommendationsPerUser <= r._2.size && r._2.size < MaxRecommendationsPerUser).flatMap(_._2).repartition(NumPartitions).cache()
println(s"Parsed $ratingFile. Kept ${trainingRatings.count()} ratings out of ${rawTrainingRatings.count()}") I am getting this message: Parsed hdfs://quickstart.cloudera:8020/user/cloudera/ratings.csv. Kept 0 ratings out of 568454 because if i run the exact code in the spark-shell, i got this message: Parsed hdfs://quickstart.cloudera:8020/user/cloudera/ratings.csv. Kept 73279 ratings out of 568454 Why is it working fine within the spark-shell but it is not programmatically running in the vmware image? Thank your for reading until here.
... View more
05-30-2016
03:36 AM
Hi again Sean, i did not see your reply until now when i have a similar problem trying to access a file programmatically. I have a vmware cloudera image, cdh-5.4.2.0 running with centos6, i am using OS X as my host machine, i have modified my /etc/hosts file with a line like this: MacBook-Pro-Retina-de-Alonso:bin aironman$ cat /etc/hosts
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting. Do not change this entry.
##
127.0.0.1 localhost
127.0.0.1 my-cassandra-node-001
255.255.255.255 broadcasthost
::1 localhost
192.168.30.137 quickstart.cloudera quickstart You can see that i can reach to the vmware machine from the host machine: MacBook-Pro-Retina-de-Alonso:bin aironman$ ping quickstart.cloudera
PING quickstart.cloudera (192.168.30.137): 56 data bytes
64 bytes from 192.168.30.137: icmp_seq=0 ttl=64 time=0.293 ms
64 bytes from 192.168.30.137: icmp_seq=1 ttl=64 time=0.273 ms
64 bytes from 192.168.30.137: icmp_seq=2 ttl=64 time=0.207 ms
64 bytes from 192.168.30.137: icmp_seq=3 ttl=64 time=0.240 ms
64 bytes from 192.168.30.137: icmp_seq=4 ttl=64 time=0.402 ms
^C
--- quickstart.cloudera ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.207/0.283/0.402/0.066 ms And i can reach to 8020 port from that machine: MacBook-Pro-Retina-de-Alonso:bin aironman$ telnet quickstart.cloudera 8020
Trying 192.168.30.137...
Connected to quickstart.cloudera.
Escape character is '^]'. I can do a ls command in the vmware machine: [cloudera@quickstart ~]$ hdfs dfs -ls /user/cloudera/ratings.csv
-rw-r--r-- 1 cloudera cloudera 16906296 2016-05-30 11:29 /user/cloudera/ratings.csv I can read its content: [cloudera@quickstart ~]$ hdfs dfs -cat /user/cloudera/ratings.csv | wc -l
568454 But, when i try to access the file programmatically i get a IOException: Exception in thread "main" java.io.IOException: Incomplete HDFS URI, no host: hdfs:/quickstart.cloudera:8020/ratings.csv
at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:133)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2433)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:88)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2467)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2449)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:367)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:287)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:221)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:270)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:35)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:237)
at org.apache.spark.Partitioner$.defaultPartitioner(Partitioner.scala:65)
at org.apache.spark.rdd.RDD$$anonfun$groupBy$1.apply(RDD.scala:620)
at org.apache.spark.rdd.RDD$$anonfun$groupBy$1.apply(RDD.scala:620)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.groupBy(RDD.scala:619)
at example.utils.Recommender.<init>(Recommender.scala:35)
at example.spark.AmazonKafkaConnector$.main(AmazonKafkaConnectorWithMongo.scala:109)
at example.spark.AmazonKafkaConnector.main(AmazonKafkaConnectorWithMongo.scala) The code is : val ratingFile= "hdfs:////quickstart.cloudera:8020/ratings.csv"
val rawTrainingRatings = sc.textFile(ratingFile).map {
line =>
val Array(userId, productId, scoreStr) = line.split(",")
AmazonRating(userId, productId, scoreStr.toDouble)
} I also tried to use: val ratingFile= "hdfs:////quickstart.cloudera:8020/user/cloudera/ratings.csv" with the same result, IOException I also tried to disable my local firewall with the same result. How can i access the hdfs running in a vmware image from an external machine (my OS X laptop), what files do i have to modify in the pseudo distributed image in order to get read and write access on files located in the image? please, help.
... View more
09-07-2015
09:11 AM
Hi, i have this scala code to write to hdfs in my laptop using this image: package org.glassfish.samples
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import java.io.PrintWriter;
object AnotherApp {
def write(uri: String, filePath: String, data: Array[Byte]): Unit = {
//System.setProperty("HADOOP_USER_NAME", "root")
System.setProperty("HADOOP_USER_NAME","cloudera")
val path = new Path(filePath)
val conf = new Configuration()
conf.set("fs.defaultFS", uri)
val fs = FileSystem.get(conf)
val os = fs.create(path)
os.write(data)
fs.close()
}
def main(args: Array[String]) {
//val conf = ConfigFactory.load()
// write(conf.getString("hdfs.uri"), conf.getString("hdfs.result_path"), "Hello World".getBytes)
write("hdfs://192.168.30.147:8020", "/tmp/helloworld.txt", "Hello World".getBytes)
}
} I ran it and i can see the new file /tmp/helloworld.txt without content, so i suspect it is matter of ports. My question is, what is it the port to write to hdfs in pseudo distributed mode. How can i see the used ports in Hue? thank you
... View more
08-14-2015
03:39 AM
Hi, i am learning the apache spark framework using the cloudera-quickstart-vm-5.4.2-0-virtualbox image. This image comes with apache spark 1.3.0 and i spark 1.4.1 is already bundled to be used, so my question is: which is the best way to upgrade this virtual box image to use spark 1.4.1? sudo yum update and restarting the image? Thanks in advance Alonso
... View more
- Tags:
- upgrading spark
08-07-2015
03:05 AM
Ok, i am going to download cloudera-quickstart-vm-5.4.2-0-virtualbox image. Thank you very much folks Alonso
... View more
08-06-2015
09:15 AM
I am going to update this thread in order that another people with the same problem know what to do. I restarted the image with sudo halt command, then i updated the imageś config applying 2 cores and 8 GB of ram instead of 1 core and 4GB (default options) After the restart, i am not able to login to HUE, and services looks offline: [cloudera@quickstart ~]$ service --status-all | grep FAILED
Flume NG agent is not running [FAILED]
Hadoop datanode is not running [FAILED]
Hadoop journalnode is not running [FAILED]
Hadoop namenode is not running [FAILED]
Hadoop secondarynamenode is not running [FAILED]
Hadoop httpfs is not running [FAILED]
Hadoop historyserver is not running [FAILED]
Hadoop nodemanager is not running [FAILED]
Hadoop proxyserver is not running [FAILED]
Hadoop resourcemanager is not running [FAILED]
HBase master daemon is not running [FAILED]
HBase rest daemon is not running [FAILED]
HBase Solr Indexer is not running [FAILED]
HBase thrift daemon is not running [FAILED]
Hive Metastore is not running [FAILED]
Hive Server2 is not running [FAILED]
Impala Catalog Server is not running [FAILED]
Impala Server is not running [FAILED]
Impala State Store Server is not running [FAILED]
/etc/init.d/kdump: line 48: /var/lock/kdump: Permission denied
Sentry DB Store Service is not running [FAILED]
Solr server daemon agent is not running [FAILED]
Spark history-server is not running [FAILED]
Spark master is not running [FAILED]
Spark worker is not running [FAILED]
Sqoop Server is not running [FAILED]
/etc/init.d/sshd: line 33: /etc/sysconfig/sshd: Permission denied
[cloudera@quickstart ~]$ pwd
/home/cloudera
[cloudera@quickstart ~]$ ls
cloudera-manager Downloads Music Templates
cm_api.sh eclipse notes from spark course Videos
datasets labfiles notes from spark course~ workspace
Desktop labfiles.zip Pictures
Documents lib Public
[cloudera@quickstart ~]$ ./cloudera-manager --force
You must run this script as root. Try 'sudo ./cloudera-manager '.
[cloudera@quickstart ~]$ sudo ./cloudera-manager --force
[QuickStart] Shutting down CDH services via init scripts...
[QuickStart] Disabling CDH services on boot...
[QuickStart] Starting Cloudera Manager services...
[QuickStart] Deploying client configuration...
[QuickStart] Starting CM Management services...
[QuickStart] Enabling CM services on boot...
[QuickStart] Starting CDH services... ________________________________________________________________________________ Success! You can now log into Cloudera Manager from the QuickStart VM's browser: http://quickstart.cloudera:7180 Username: cloudera Password: cloudera [cloudera@quickstart ~]$ As you can see, the script located on desktop is trying to launch every component, but i can see errors when i launch service --status-all command: [cloudera@quickstart ~]$ sudo service --status-all | grep FAILED
Flume NG agent is not running [FAILED]
Hadoop datanode is not running [FAILED]
Hadoop journalnode is not running [FAILED]
Hadoop namenode is not running [FAILED]
Hadoop secondarynamenode is not running [FAILED]
Hadoop httpfs is not running [FAILED]
Hadoop historyserver is not running [FAILED]
Hadoop nodemanager is not running [FAILED]
Hadoop proxyserver is not running [FAILED]
Hadoop resourcemanager is not running [FAILED]
HBase master daemon is not running [FAILED]
HBase rest daemon is not running [FAILED]
HBase Solr Indexer is not running [FAILED]
HBase thrift daemon is not running [FAILED]
Hive Metastore is not running [FAILED]
Hive Server2 is not running [FAILED]
Impala Catalog Server is not running [FAILED]
Impala Server is not running [FAILED]
Impala State Store Server is not running [FAILED]
Sentry DB Store Service is not running [FAILED]
Solr server daemon agent is not running [FAILED]
Spark history-server is not running [FAILED]
Spark master is not running When i try to use HUE and examine the data i wrote yesterday, i found this: Cannot access: /user/cloudera. The HDFS REST service is not available. Note: You are a Hue admin but not a HDFS superuser (which is "hdfs").
('Connection aborted.', error(111, 'Connection refused'))
[06/Aug/2015 09:02:40 -0700] webhdfs ERROR Failed to determine superuser of WebHdfs at http://quickstart.cloudera:50070/webhdfs/v1: ('Connection aborted.', error(111, 'Connection refused'))
Traceback (most recent call last):
File "/usr/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 149, in superuser
sb = self.stats('/')
File "/usr/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 236, in stats
res = self._stats(path)
File "/usr/lib/hue/desktop/libs/hadoop/src/hadoop/fs/webhdfs.py", line 230, in _stats
raise ex
WebHdfsException: ('Connection aborted.', error(111, 'Connection refused')) the terminal commands output looks like: [cloudera@quickstart ~]$ hdfs dfs -ls /
ls: Call From quickstart.cloudera/127.0.0.1 to quickstart.cloudera:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
[cloudera@quickstart ~]$ hadoop fs -ls /
ls: Call From quickstart.cloudera/127.0.0.1 to quickstart.cloudera:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused
[cloudera@quickstart ~]$ hadoop fs -ls /
ls: Call From quickstart.cloudera/127.0.0.1 to quickstart.cloudera:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see: http://wiki.apache.org/hadoop/ConnectionRefused Configuration Issues on Cloudera manager is telling me this: All Configuration Issues
Cloudera QuickStart
zookeeper: Service zookeeper has 1 Server. Cloudera suggests at least 3 Servers for ZooKeeper.
hdfs: Service hdfs has 1 DataNode. Cloudera suggests at least 3 DataNodes for HDFS.
quickstart.cloudera: Memory Overcommit Validation Threshold
Memory on host quickstart.cloudera is overcommitted. The total memory allocation is 68.7 GiB bytes but there are only 7.7 GiB bytes of RAM (1.5 GiB bytes of which are reserved for the system). Visit the Resources tab on the Host page for allocation details. Reconfigure the roles on the host to lower the overall memory allocation. Note: Java maximum heap sizes are multiplied by 1.3 to approximate JVM overhead. I do not understand this, this monocluster works well the first time with 1 core and 4GB of ram, i mean, i was able to run some spark process and access the HDFS, now, with 2 cores and 8GB i cant? Please, give me some advice. Alonso
... View more
08-06-2015
08:14 AM
Hi, i have just noticed that my cloudera-quickstart-vm-5.3.0-0-virtualbox image is not running properly, i mean that hdfs and other components are offline. I cannot list any files using the terminal, hue is not working and so on so forth. [cloudera@quickstart labfiles]$ service --status-all | grep FAILED
Flume NG agent is not running [FAILED]
Hadoop namenode is dead and pid file exists [FAILED]
Hadoop historyserver is dead and pid file exists [FAILED]
Hadoop proxyserver is dead and pid file exists [FAILED]
HBase master daemon is dead and pid file exists [FAILED]
HBase Solr Indexer is not running [FAILED]
Hive Metastore is dead and pid file exists [FAILED]
Hive Server2 is dead and pid file exists [FAILED]
Impala Catalog Server is dead and pid file exists [FAILED]
Impala Server is dead and pid file exists [FAILED]
/etc/init.d/kdump: line 48: /var/lock/kdump: Permission denied
Sentry DB Store Service is dead and pid file exists [FAILED]
Spark history-server is dead and pid file exists [FAILED]
Spark master is dead and pid file exists [FAILED]
Spark worker is dead and pid file exists [FAILED]
Sqoop Server is dead and pid file exists [FAILED]
/etc/init.d/sshd: line 33: /etc/sysconfig/sshd: Permission denied
[cloudera@quickstart labfiles]$ I have noticed this behaviour after an update of the centos system, i do not want to restart the virtualbox image, so, is there any script that launch again the components in order? Thank you.
... View more
08-06-2015
07:49 AM
I have the same problem, tail command is getting this output: 2015-08-06 07:47:26,459 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: 0.0.0.0/0.0.0.0:8022
2015-08-06 07:47:32,462 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8022. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-08-06 07:47:33,463 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8022. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-08-06 07:47:34,464 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8022. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-08-06 07:47:35,465 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8022. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-08-06 07:47:36,466 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8022. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-08-06 07:47:37,467 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8022. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-08-06 07:47:38,468 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8022. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-08-06 07:47:39,469 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8022. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-08-06 07:47:40,471 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8022. Already tried 8 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-08-06 07:47:41,472 INFO org.apache.hadoop.ipc.Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8022. Already tried 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2015-08-06 07:47:41,473 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Problem connecting to server: 0.0.0.0/0.0.0.0:8022 Before i notice this problem i updated the centos image doing sudo yum update, 3GB of new data... Is there any way to see whats going on with a graphical user interface?
... View more
08-06-2015
02:30 AM
Hi srowen, i am using a cloudera-quickstart-vm-5.3.0-virtualbox image, without any aditional settings from me, just unzip it and ready to go. Only one node running on my laptop. Thank you for the assistance
... View more
08-05-2015
08:55 AM
scala> val README = sc.textFile("hdfs:///user/cloudera/README.md") 2015-08-05 08:53:26,278 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - ensureFreeSpace(246950) called with curMem=4845564, maxMem=280248975 2015-08-05 08:53:26,278 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_36 stored as values in memory (estimated size 241.2 KB, free 262.4 MB) 2015-08-05 08:53:26,296 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - ensureFreeSpace(19465) called with curMem=5092514, maxMem=280248975 2015-08-05 08:53:26,297 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_36_piece0 stored as bytes in memory (estimated size 19.0 KB, free 262.4 MB) 2015-08-05 08:53:26,297 INFO [sparkDriver-akka.actor.default-dispatcher-14] storage.BlockManagerInfo (Logging.scala:logInfo(59)) - Added broadcast_36_piece0 in memory on localhost:46000 (size: 19.0 KB, free: 266.9 MB) 2015-08-05 08:53:26,297 INFO [main] storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Updated info of block broadcast_36_piece0 2015-08-05 08:53:26,298 INFO [main] spark.SparkContext (Logging.scala:logInfo(59)) - Created broadcast 36 from textFile at <console>:12 README: org.apache.spark.rdd.RDD[String] = hdfs:///user/cloudera/README.md MappedRDD[45] at textFile at <console>:12 scala> README.count() 2015-08-05 08:53:32,262 INFO [sparkDriver-akka.actor.default-dispatcher-6] storage.BlockManager (Logging.scala:logInfo(59)) - Removing broadcast 30 2015-08-05 08:53:32,262 INFO [sparkDriver-akka.actor.default-dispatcher-6] storage.BlockManager (Logging.scala:logInfo(59)) - Removing block broadcast_30_piece0 2015-08-05 08:53:32,262 INFO [sparkDriver-akka.actor.default-dispatcher-6] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_30_piece0 of size 1614 dropped from memory (free 275138610) 2015-08-05 08:53:32,263 INFO [sparkDriver-akka.actor.default-dispatcher-14] storage.BlockManagerInfo (Logging.scala:logInfo(59)) - Removed broadcast_30_piece0 on localhost:46000 in memory (size: 1614.0 B, free: 266.9 MB) 2015-08-05 08:53:32,263 INFO [sparkDriver-akka.actor.default-dispatcher-6] storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Updated info of block broadcast_30_piece0 2015-08-05 08:53:32,263 INFO [sparkDriver-akka.actor.default-dispatcher-6] storage.BlockManager (Logging.scala:logInfo(59)) - Removing block broadcast_30 2015-08-05 08:53:32,263 INFO [sparkDriver-akka.actor.default-dispatcher-6] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_30 of size 2560 dropped from memory (free 275141170) 2015-08-05 08:53:32,263 INFO [Spark Context Cleaner] spark.ContextCleaner (Logging.scala:logInfo(59)) - Cleaned broadcast 30 2015-08-05 08:53:32,263 INFO [sparkDriver-akka.actor.default-dispatcher-15] storage.BlockManager (Logging.scala:logInfo(59)) - Removing broadcast 32 2015-08-05 08:53:32,263 INFO [sparkDriver-akka.actor.default-dispatcher-15] storage.BlockManager (Logging.scala:logInfo(59)) - Removing block broadcast_32 2015-08-05 08:53:32,264 INFO [sparkDriver-akka.actor.default-dispatcher-15] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_32 of size 2560 dropped from memory (free 275143730) 2015-08-05 08:53:32,264 INFO [sparkDriver-akka.actor.default-dispatcher-15] storage.BlockManager (Logging.scala:logInfo(59)) - Removing block broadcast_32_piece0 2015-08-05 08:53:32,264 INFO [sparkDriver-akka.actor.default-dispatcher-15] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_32_piece0 of size 1614 dropped from memory (free 275145344) 2015-08-05 08:53:32,264 INFO [sparkDriver-akka.actor.default-dispatcher-3] storage.BlockManagerInfo (Logging.scala:logInfo(59)) - Removed broadcast_32_piece0 on localhost:46000 in memory (size: 1614.0 B, free: 266.9 MB) 2015-08-05 08:53:32,264 INFO [sparkDriver-akka.actor.default-dispatcher-15] storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Updated info of block broadcast_32_piece0 2015-08-05 08:53:32,264 INFO [Spark Context Cleaner] spark.ContextCleaner (Logging.scala:logInfo(59)) - Cleaned broadcast 32 2015-08-05 08:53:32,264 INFO [sparkDriver-akka.actor.default-dispatcher-14] storage.BlockManager (Logging.scala:logInfo(59)) - Removing broadcast 35 2015-08-05 08:53:32,265 INFO [sparkDriver-akka.actor.default-dispatcher-14] storage.BlockManager (Logging.scala:logInfo(59)) - Removing block broadcast_35 2015-08-05 08:53:32,265 INFO [sparkDriver-akka.actor.default-dispatcher-14] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_35 of size 2560 dropped from memory (free 275147904) 2015-08-05 08:53:32,265 INFO [sparkDriver-akka.actor.default-dispatcher-14] storage.BlockManager (Logging.scala:logInfo(59)) - Removing block broadcast_35_piece0 2015-08-05 08:53:32,265 INFO [sparkDriver-akka.actor.default-dispatcher-14] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_35_piece0 of size 1614 dropped from memory (free 275149518) 2015-08-05 08:53:32,265 INFO [sparkDriver-akka.actor.default-dispatcher-16] storage.BlockManagerInfo (Logging.scala:logInfo(59)) - Removed broadcast_35_piece0 on localhost:46000 in memory (size: 1614.0 B, free: 266.9 MB) 2015-08-05 08:53:32,265 INFO [sparkDriver-akka.actor.default-dispatcher-14] storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Updated info of block broadcast_35_piece0 2015-08-05 08:53:32,266 INFO [Spark Context Cleaner] spark.ContextCleaner (Logging.scala:logInfo(59)) - Cleaned broadcast 35 java.io.IOException: Incomplete HDFS URI, no host: hdfs:/user/cloudera/README.md at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:141) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296) at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:256) at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228) at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313) at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205) at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203) at scala.Option.getOrElse(Option.scala:120) at org.apache.spark.rdd.RDD.partitions(RDD.scala:203) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1328) at org.apache.spark.rdd.RDD.count(RDD.scala:910) at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15) at $iwC$$iwC$$iwC.<init>(<console>:20) at $iwC$$iwC.<init>(<console>:22) at $iwC.<init>(<console>:24) at <init>(<console>:26) at .<init>(<console>:30) at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:852) at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1125) at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:674) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:705) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:669) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:828) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:873) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:785) at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:628) at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:636) at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:641) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:968) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916) at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916) at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:916) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1011) at org.apache.spark.repl.Main$.main(Main.scala:31) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) scala> It is necessary to use the full hdfs path: scala> val README = sc.textFile("hdfs://quickstart.cloudera:8020/user/cloudera/README.md") 2015-08-05 08:54:22,840 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - ensureFreeSpace(246950) called with curMem=5099457, maxMem=280248975 2015-08-05 08:54:22,841 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_37 stored as values in memory (estimated size 241.2 KB, free 262.2 MB) 2015-08-05 08:54:22,856 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - ensureFreeSpace(19465) called with curMem=5346407, maxMem=280248975 2015-08-05 08:54:22,858 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_37_piece0 stored as bytes in memory (estimated size 19.0 KB, free 262.1 MB) 2015-08-05 08:54:22,858 INFO [sparkDriver-akka.actor.default-dispatcher-5] storage.BlockManagerInfo (Logging.scala:logInfo(59)) - Added broadcast_37_piece0 in memory on localhost:46000 (size: 19.0 KB, free: 266.9 MB) 2015-08-05 08:54:22,858 INFO [main] storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Updated info of block broadcast_37_piece0 2015-08-05 08:54:22,859 INFO [main] spark.SparkContext (Logging.scala:logInfo(59)) - Created broadcast 37 from textFile at <console>:12 README: org.apache.spark.rdd.RDD[String] = hdfs://quickstart.cloudera:8020/user/cloudera/README.md MappedRDD[47] at textFile at <console>:12 Thanks again
... View more
08-05-2015
08:51 AM
Yea, it works! thank you Sir. scala> val README = sc.textFile("hdfs://quickstart.cloudera:8020/user/cloudera/README.md") scala> README.count() 2015-08-05 08:51:09,639 INFO [main] mapred.FileInputFormat (FileInputFormat.java:listStatus(247)) - Total input paths to process : 1 2015-08-05 08:51:09,643 INFO [main] spark.SparkContext (Logging.scala:logInfo(59)) - Starting job: count at <console>:15 2015-08-05 08:51:09,643 INFO [sparkDriver-akka.actor.default-dispatcher-3] scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Got job 15 (count at <console>:15) with 1 output partitions (allowLocal=false) 2015-08-05 08:51:09,643 INFO [sparkDriver-akka.actor.default-dispatcher-3] scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Final stage: Stage 15(count at <console>:15) 2015-08-05 08:51:09,643 INFO [sparkDriver-akka.actor.default-dispatcher-3] scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Parents of final stage: List() 2015-08-05 08:51:09,644 INFO [sparkDriver-akka.actor.default-dispatcher-3] scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Missing parents: List() 2015-08-05 08:51:09,644 INFO [sparkDriver-akka.actor.default-dispatcher-3] scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Submitting Stage 15 (hdfs://quickstart.cloudera:8020/user/cloudera/README.md MappedRDD[39] at textFile at <console>:12), which has no missing parents 2015-08-05 08:51:09,646 INFO [sparkDriver-akka.actor.default-dispatcher-3] storage.MemoryStore (Logging.scala:logInfo(59)) - ensureFreeSpace(2560) called with curMem=4304386, maxMem=280248975 2015-08-05 08:51:09,646 INFO [sparkDriver-akka.actor.default-dispatcher-3] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_32 stored as values in memory (estimated size 2.5 KB, free 263.2 MB) 2015-08-05 08:51:09,647 INFO [sparkDriver-akka.actor.default-dispatcher-3] storage.MemoryStore (Logging.scala:logInfo(59)) - ensureFreeSpace(1614) called with curMem=4306946, maxMem=280248975 2015-08-05 08:51:09,647 INFO [sparkDriver-akka.actor.default-dispatcher-3] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_32_piece0 stored as bytes in memory (estimated size 1614.0 B, free 263.2 MB) 2015-08-05 08:51:09,647 INFO [sparkDriver-akka.actor.default-dispatcher-14] storage.BlockManagerInfo (Logging.scala:logInfo(59)) - Added broadcast_32_piece0 in memory on localhost:46000 (size: 1614.0 B, free: 267.0 MB) 2015-08-05 08:51:09,647 INFO [sparkDriver-akka.actor.default-dispatcher-3] storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Updated info of block broadcast_32_piece0 2015-08-05 08:51:09,648 INFO [sparkDriver-akka.actor.default-dispatcher-3] spark.SparkContext (Logging.scala:logInfo(59)) - Created broadcast 32 from broadcast at DAGScheduler.scala:838 2015-08-05 08:51:09,649 INFO [sparkDriver-akka.actor.default-dispatcher-3] scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Submitting 1 missing tasks from Stage 15 (hdfs://quickstart.cloudera:8020/user/cloudera/README.md MappedRDD[39] at textFile at <console>:12) 2015-08-05 08:51:09,649 INFO [sparkDriver-akka.actor.default-dispatcher-3] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(59)) - Adding task set 15.0 with 1 tasks 2015-08-05 08:51:09,649 INFO [sparkDriver-akka.actor.default-dispatcher-14] scheduler.TaskSetManager (Logging.scala:logInfo(59)) - Starting task 0.0 in stage 15.0 (TID 15, localhost, ANY, 1319 bytes) 2015-08-05 08:51:09,650 INFO [Executor task launch worker-7] executor.Executor (Logging.scala:logInfo(59)) - Running task 0.0 in stage 15.0 (TID 15) 2015-08-05 08:51:09,657 INFO [Executor task launch worker-7] rdd.HadoopRDD (Logging.scala:logInfo(59)) - Input split: hdfs://quickstart.cloudera:8020/user/cloudera/README.md:0+4811 2015-08-05 08:51:09,665 INFO [Executor task launch worker-7] executor.Executor (Logging.scala:logInfo(59)) - Finished task 0.0 in stage 15.0 (TID 15). 1757 bytes result sent to driver 2015-08-05 08:51:09,670 INFO [sparkDriver-akka.actor.default-dispatcher-14] scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Stage 15 (count at <console>:15) finished in 0.019 s 2015-08-05 08:51:09,670 INFO [main] scheduler.DAGScheduler (Logging.scala:logInfo(59)) - Job 15 finished: count at <console>:15, took 0.027246 s res30: Long = 141 scala> 2015-08-05 08:51:09,670 INFO [task-result-getter-3] scheduler.TaskSetManager (Logging.scala:logInfo(59)) - Finished task 0.0 in stage 15.0 (TID 15) in 20 ms on localhost (1/1) 2015-08-05 08:51:09,671 INFO [task-result-getter-3] scheduler.TaskSchedulerImpl (Logging.scala:logInfo(59)) - Removed TaskSet 15.0, whose tasks have all completed, from pool
... View more
08-05-2015
08:46 AM
I am trying to load a file from HDFS using scala-shell. I have load into HDFS the file using the next command: [cloudera@quickstart labfiles]$ hadoop fs -copyFromLocal README.md /user/cloudera/README.md
[cloudera@quickstart labfiles]$ hadoop fs -ls /user/cloudera
Found 3 items
-rw-r--r-- 1 cloudera cloudera 4811 2015-08-05 08:05 /user/cloudera/README.md
-rw-r--r-- 1 cloudera cloudera 6538420 2015-08-04 09:34 /user/cloudera/downloadFiles.mp4
-rw-r--r-- 1 cloudera cloudera 1770 2015-08-04 10:02 /user/cloudera/readme.txt but when i try to load the file from HDFS using scala... scala> val README = sc.textFile("/user/cloudera/README.md")
2015-08-05 08:43:13,897 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - ensureFreeSpace(246950) called with curMem=3500967, maxMem=280248975
2015-08-05 08:43:13,897 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_28 stored as values in memory (estimated size 241.2 KB, free 263.7 MB)
2015-08-05 08:43:13,934 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - ensureFreeSpace(19465) called with curMem=3747917, maxMem=280248975
2015-08-05 08:43:13,935 INFO [main] storage.MemoryStore (Logging.scala:logInfo(59)) - Block broadcast_28_piece0 stored as bytes in memory (estimated size 19.0 KB, free 263.7 MB)
2015-08-05 08:43:13,935 INFO [sparkDriver-akka.actor.default-dispatcher-3] storage.BlockManagerInfo (Logging.scala:logInfo(59)) - Added broadcast_28_piece0 in memory on localhost:46000 (size: 19.0 KB, free: 267.0 MB)
2015-08-05 08:43:13,935 INFO [main] storage.BlockManagerMaster (Logging.scala:logInfo(59)) - Updated info of block broadcast_28_piece0
2015-08-05 08:43:13,937 INFO [main] spark.SparkContext (Logging.scala:logInfo(59)) - Created broadcast 28 from textFile at <console>:12
README: org.apache.spark.rdd.RDD[String] = /user/cloudera/README.md MappedRDD[35] at textFile at <console>:12
scala> README.count()
org.apache.hadoop.mapred.InvalidInputException: Input path does not exist: file:/user/cloudera/README.md
at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:285)
at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:313)
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:205)
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:203)
at scala.Option.getOrElse(Option.scala:120)
at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1328)
at org.apache.spark.rdd.RDD.count(RDD.scala:910)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:15)
at $iwC$$iwC$$iwC.<init>(<console>:20)
at $iwC$$iwC.<init>(<console>:22)
at $iwC.<init>(<console>:24)
at <init>(<console>:26)
at .<init>(<console>:30)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:852)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1125)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:674)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:705)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:669)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:828)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:873)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:785)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:628)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:636)
at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:641)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:968)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916)
at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:916)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:916)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1011)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
scala> I can see the file using HUE and i can access it using hdfs fs -ls /user/cloudera, so what is going on? I am using cloudera-quickstart-vm-5.3.0.0-virtualbox in a macbook pro. Thank you
... View more
