Member since
06-18-2015
55
Posts
34
Kudos Received
2
Solutions
My Accepted Solutions
| Title | Views | Posted |
|---|---|---|
| 54 | 03-04-2016 02:39 AM | |
| 100 | 12-29-2015 09:42 AM |
04-21-2016
02:44 AM
1 Kudo
Hi,
I used below code to insert data from Spark1.5.2 to Phoenix 4.4
df.write.format("org.apache.phoenix.spark").mode(SaveMode.Overwrite).options(collection.immutable.Map(
"zkUrl" -> "localhost:2181/hbase-unsecure",
"table" -> "TEST")).save();
Spark job run parameters
spark-shell --properties-file /TestDivya/Spark/Phoenix.properties --jars /usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-spark-4.4.0.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/phoenix/phoenix-client.jar --driver-class-path /usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-spark-4.4.0.2.3.4.0-3485.jar,/usr/hdp/2.3.4.0-3485/hbase/lib/phoenix-client-4.4.0.jar --conf "spark.executor.extraClassPath=/usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-core-4.4.0.2.3.4.0-3485.jar" --conf "spark.executor.extraClassPath=/usr/hdp/2.3.4.0-3485/phoenix/lib/phoenix-core-4.4.0.2.3.4.0-3485.jar" --packages com.databricks:spark-csv_2.10:1.4.0 --master yarn-client -i /TestDivya/Spark/WriteToPheonix.scala
I am getting below error when I am running the spark job
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 3.0 failed 4 times, most recent failure: Lost task 0.3 in stage 3.0 (TID 411, ip-xxxxx-xx-xxx.ap-southeast-1.compute.internal): java.lang.RuntimeException: java.sql.SQLException: No suitable driver found for jdbc:phoenix:localhost:2181:/hbase-unsecure;
at org.apache.phoenix.mapreduce.PhoenixOutputFormat.getRecordWriter(PhoenixOutputFormat.java:58)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopDataset$1$anonfun$12.apply(PairRDDFunctions.scala:1030)
at org.apache.spark.rdd.PairRDDFunctions$anonfun$saveAsNewAPIHadoopDataset$1$anonfun$12.apply(PairRDDFunctions.scala:1014)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Has any body face this issue ?
Would really appreciate the help.
Thanks
... View more
Labels:
04-12-2016
06:47 AM
Hi, I am on HDP 2.3.4 ( 3 node cluster) , My HBase scans are slow after inserting a million row data As I am new bee to HBase, Any suggestions experts can provide me to tune performance. Would really appreciate the help. Thanks, Divya
... View more
Labels:
03-17-2016
08:52 AM
As mentioned in my comment I already tried ,but it didn't work .
... View more
03-15-2016
06:13 AM
1 Kudo
I downloaded Hive-Hbase-Handler1.2.jar and renamed to hive-hbase-1.2.1.2.3.4.0-3485.jar and uploaded to usr/hdp/hive/lib Now when I try to create table based on hbase , I am getting error ,Unable to connect to Thrift Server.
... View more
03-15-2016
05:29 AM
1 Kudo
Hi, Mistakenly I deleted(did delete forever) the hive-hbase handler jar from my HDP 2.3.4 cluster . From where can I download now Thanks, Divya
... View more
Labels:
03-04-2016
02:39 AM
2 Kudos
Able to resolve it as missing one of the jars files hbase-hadoop-compat.jar:
... View more
03-02-2016
03:09 AM
3 Kudos
Hi, I have registered the hive external table on hbase table. When I try to access that through hiveContext getting below error org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in sta ge 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3, i p-172-31-29-201.ap-southeast-1.compute.internal): java.lang.RuntimeException: hb ase-default.xml file seems to be for an older version of HBase (null), this vers ion is 1.1.2.2.3.4.0-3485
at org.apache.hadoop.hbase.HBaseConfiguration.checkDefaultsVersion(HBase Configuration.java:71)
at org.apache.hadoop.hbase.HBaseConfiguration.addHbaseResources(HBaseCon figuration.java:81) I have already placed hbase-default.xml ,hbase-site.xml in spark/conf by setting the below property to true <property>
<name>hbase.defaults.for.version.skip</name>
<value>true</value>
<description>Set to true to skip the 'hbase.defaults.for.version' check.
Setting this to true can be useful in contexts other than
the other side of a maven generation; i.e. running in an
IDE. You'll want to set this boolean to true to avoid
seeing the RuntimeException complaint: "hbase-default.xml file
seems to be for and old version of HBase (\${hbase.version}), this
version is X.X.X-SNAPSHOT"</description>
</property> Spark code : import org.apache.spark.sql.hive.HiveContextval hiveContext = new HiveContext(sc)val df = hiveContext.sql("select * from test")df.show Adding these jars while submitting starting spark shell /usr/hdp/2.3.4.0-3485/hive/lib/guava-14.0.1.jar
/usr/hdp/2.3.4.0-3485/hive/lib/hive-hbase-handler-1.2.1.2.3.4.0-3485.jar
/usr/hdp/2.3.4.0-3485/hive/lib/htrace-core-3.1.0-incubating.jar,
/usr/hdp/2.3.4.0-3485/hive/lib/zookeeper-3.4.6.2.3.4.0-3485.jar,
/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-client-1.1.2.2.3.4.0-3485.jar
/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-common-1.1.2.2.3.4.0-3485.jar
/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-protocol-1.1.2.2.3.4.0-3485.jar
/usr/hdp/2.3.4.0-3485/hbase/lib/hbase-server-1.1.2.2.3.4.0-3485.jar
... View more
Labels:
02-19-2016
02:40 AM
1 Kudo
@asinghal Why doesn't it throw error when I run the same command HDP 2.3.2 sandbox ? In sandbox it works fine and I couldn't see Jackson dependencies conflicts error .
... View more
02-18-2016
09:09 AM
2 Kudos
Hi,I am getting following error while starting spark shell with phoenix clients spark-shell --jars /usr/hdp/current/phoenix-
client/phoenix-4.4.0.2.3.4.0-3485-client.jar --driver-class-path /usr/hdp/current/phoenix-client/phoenix-4.4.0.2.3.4.0-3485-client.jar --master yarn-client StackTrace: INFO TimelineClientImpl: Timeline service address: http://ip-xxx-xx-xx-xxx.ap-southeast-1.compute.internal:8188/ws/v1/timeline/
java.lang.NoSuchMethodError: org.codehaus.jackson.map.ObjectMapper.setSerializationInclusion(Lorg/codehaus/jackson/map/annotate/JsonSerialize$Inclusion;)Lorg/codehaus/jackson/map/ObjectMapper;
at org.apache.hadoop.yarn.webapp.YarnJacksonJaxbJsonProvider.configObjectMapper(YarnJacksonJaxbJsonProvider.java:59)
at org.apache.hadoop.yarn.util.timeline.TimelineUtils.<clinit>(TimelineUtils.java:50)
at org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.serviceInit(YarnClientImpl.java:172)
at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:108)
at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:144)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:523)
at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017)
at $iwC$iwC.<init>(<console>:9)
at $iwC.<init>(<console>:18)
at <init>(<console>:20)
at .<init>(<console>:24)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:125)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply$mcZ$sp(SparkILoop.scala:991)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:685)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
java.lang.NullPointerException
at org.apache.spark.sql.execution.ui.SQLListener.<init>(SQLListener.scala:34)
at org.apache.spark.sql.SQLContext.<init>(SQLContext.scala:77)
at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:74)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028)
at $iwC$iwC.<init>(<console>:9)
at $iwC.<init>(<console>:18)
at <init>(<console>:20)
at .<init>(<console>:24)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:132)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply$mcZ$sp(SparkILoop.scala:991)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:685)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
<console>:10: error: not found: value sqlContext
import sqlContext.implicits._
^
<console>:10: error: not found: value sqlContext
import sqlContext.sql
Googled and found there is Jackson dependency is not available for Hadoop 2.x version(SPARK-5108)
Is the above errors related to above mentioned issue . Thanks,
... View more
Labels:
02-18-2016
08:43 AM
1 Kudo
As SparkUI port was internal ip of Ec2 ,that's the reason I wasn't able to view the current running jobs in Spark UI. To resolve this issue ,had to configure SSH Tunnel.
... View more
02-18-2016
08:40 AM
1 Kudo
@Artem Ervits : I had to configure SSH Tunnel as my cluster was running on EC2. Thanks a lot.
... View more
02-15-2016
08:14 AM
1 Kudo
@Artem Ervits : Can you please help me with issue. What settings should I change to view the logs in Web UI ? Thanks
... View more
02-15-2016
08:08 AM
1 Kudo
@Neeraj Sabharwal :I am able to view jobs in Spark history server but unable to view current running jobs in Spark Web UI even I though I am specifying --master yarn-client or --master yarn When the job finishes I could see the " This webpage is not available ERR_CONNECTION_REFUSED " And When job running it displays the page but doesnt show any running jobs . Am I missing any settings ?
... View more
02-15-2016
04:24 AM
1 Kudo
@Neeraj Sabharwal : I am running my spark below : spark-shell --properties-file /TestDivya/Spark/Oracle.properties --jars /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --driver-class-path /usr/hdp/2.3.4.0-3485/spark/lib/ojdbc6.jar --packages com.databricks:spark-csv_2.10:1.1.0 --master yarn-client -i /TestDivya/Spark/Test.scala Do I need to modify my options to view jobs in Spark UI ?
... View more
02-15-2016
02:58 AM
1 Kudo
Hi, I am running HDP-2.3.4 cluster on EC2.When I go to http://ec2-xx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com:4041/jobs/ ,it doesnt display any currently running jobs. Whereas in spark-shell ,it shows the spark UI started at 4041 16/02/14 21:45:13 INFO AbstractConnector: Started SelectChannelConnector@0.0.0.0:4041
16/02/14 21:45:13 INFO Utils: Successfully started service 'SparkUI' on port 4041.
16/02/14 21:45:13 INFO SparkUI: Started SparkUI at http://xxx.xx.xx.xxx:4041 But if I go to history server at It shows all the old jobs ran in the past . I am new bee to Spark. P.S. Attaching the screen shots of Spark Web UI and History UI Thanks, Divya
... View more
Labels:
02-15-2016
02:37 AM
1 Kudo
Thanks Ali for the help
... View more
02-04-2016
04:07 AM
1 Kudo
16/02/03 22:55:25 WARN AbstractLifeCycle: FAILED SelectChannelConnector@0.0.0.0:4040: java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$connect$1(JettyUtils.scala:237)
at org.apache.spark.ui.JettyUtils$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.ui.JettyUtils$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.util.Utils$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:247)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
at org.apache.spark.SparkContext$anonfun$13.apply(SparkContext.scala:474)
at org.apache.spark.SparkContext$anonfun$13.apply(SparkContext.scala:474)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:474)
at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017)
at $line3.$read$iwC$iwC.<init>(<console>:9)
at $line3.$read$iwC.<init>(<console>:18)
at $line3.$read.<init>(<console>:20)
at $line3.$read$.<init>(<console>:24)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.<init>(<console>:7)
at $line3.$eval$.<clinit>(<console>)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:125)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply$mcZ$sp(SparkILoop.scala:991)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:685)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/02/03 22:55:25 WARN AbstractLifeCycle: FAILED org.spark-project.jetty.server.Server@7509bf4e: java.net.BindException: Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:444)
at sun.nio.ch.Net.bind(Net.java:436)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.spark-project.jetty.server.nio.SelectChannelConnector.open(SelectChannelConnector.java:187)
at org.spark-project.jetty.server.AbstractConnector.doStart(AbstractConnector.java:316)
at org.spark-project.jetty.server.nio.SelectChannelConnector.doStart(SelectChannelConnector.java:265)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.spark-project.jetty.server.Server.doStart(Server.java:293)
at org.spark-project.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:64)
at org.apache.spark.ui.JettyUtils$.org$apache$spark$ui$JettyUtils$connect$1(JettyUtils.scala:237)
at org.apache.spark.ui.JettyUtils$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.ui.JettyUtils$anonfun$3.apply(JettyUtils.scala:247)
at org.apache.spark.util.Utils$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1920)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1911)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:247)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:136)
at org.apache.spark.SparkContext$anonfun$13.apply(SparkContext.scala:474)
at org.apache.spark.SparkContext$anonfun$13.apply(SparkContext.scala:474)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:474)
at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017)
at $line3.$read$iwC$iwC.<init>(<console>:9)
at $line3.$read$iwC.<init>(<console>:18)
at $line3.$read.<init>(<console>:20)
at $line3.$read$.<init>(<console>:24)
at $line3.$read$.<clinit>(<console>)
at $line3.$eval$.<init>(<console>:7)
at $line3.$eval$.<clinit>(<console>)
at $line3.$eval.$print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:125)
at org.apache.spark.repl.SparkILoopInit$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324)
at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124)
at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974)
at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159)
at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108)
at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply$mcZ$sp(SparkILoop.scala:991)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:685)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:120)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null}
16/02/03 22:55:25 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/st
[ec2-user@ip-172-31-29-201 ~]$ netstat -pnat | grep 404 (No info could be read for "-p": geteuid()=1000 but you should be root.)
tcp 1 0 xxx.xx.xx.xxx:34047 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp 1 0 xxx.xx.xx.xxx:34049 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp 1 0 xxx.xx.xx.xxx:34046 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp 1 0 xxx.xx.xx.xxx:34048 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp6 0 0 :::4040 :::* LISTEN -
tcp6 0 0 :::4041 :::* LISTEN -
tcp6 1 0 xxx.xx.xx.xxx:34043 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp6 1 0 xxx.xx.xx.xxx:34045 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp6 1 0 xxx.xx.xx.xxx:34042 xxx.xx.xx.xxx:6188 CLOSE_WAIT -
tcp6 1 0 xxx.xx.xx.xxx:34044 xxx.xx.xx.xxx:6188 CLOSE_WAIT Any pointers Appreciated .
... View more
Labels:
02-01-2016
04:08 AM
2 Kudos
@Pradeep Allu If your parameter parameter file is in hdfs then make sure your pig script should also be in hdfs not in local file system. Try placing your pig script also in HDFS .
... View more
01-28-2016
07:21 AM
1 Kudo
@Artem Ervits I tried doing that but still getting page cannot be displayed . Do I need to make any settings in config files aka YARN-default.xml ,etc.. I try to look up for port 8042 in configuration files of YARN but could not find the property itself arn.nodemanager.webapp.address ${yarn.nodemanager.hostname}:8042
... View more
01-28-2016
07:18 AM
Hi, Where can I find port 8042 in HDP 2.3.4 configuration files ? My Hadoop Hortonworks cluster si set up in Amazon EC2. When I visit
http://ec2-xx-xx-xxx-xxx.ap-southeast-1.compute.amazonaws.com:8088/cluster to view the application logs, the page gets displayed with all the the applications When I click of any of the application also page get displayed(Attached screenshot) Now when I click on the logs,it shows the internal IP of one of the data node and <code>http://ip-xx-xx-xx-xxx.ap-southeast-1.compute.internal:8042/node/containerlogs/container_e06_1453856711181_0026_02_000001/hdfs
shows : page not displayed .. (Attached screenshot) I could not find the port 8042 in configuration file,yarn--default.xml <code>yarn.nodemanager.webapp.address ${yarn.nodemanager.hostname}:8042
... View more
01-21-2016
03:07 AM
1 Kudo
Hi, Would really appreciate if somebody can give me pointers how can I store the pig output using DBStorage to Oracle. Below is my sample script : REGISTER /TestDivya/Pig/ojdbc6.jar ;
REGISTER /TestDivya/Pig/piggybank-0.15.0.jar ;
A = LOAD '/tmp/TestDivya/Pig/PigTest.txt' using PigStorage(',') as (name: chararray, age: int);
STORE A into 'dummy' using org.apache.pig.piggybank.storage.DBStorage('oracle.jdbc.driver.OracleDriver','jdbc:oracle:thin:@xx.xxx.xxx.xx:1521:db_name','db_username','db_password','insert into TEST values(?,?)');
... View more
Labels:
01-21-2016
02:32 AM
1 Kudo
Hi, I am newbee to EC2 and HDP 2.3.4 . When I try to view the application logs as its taking the internal IP of EC2 of one of the datanode,I get "web page cant be displayed error". When I searched the yarn-site.xml I couldnt find the port 8042 where I can make changes for log files. Can somebody could help me configuring the Application log address. Refer to the screenshot attached
... View more
Labels:
01-18-2016
07:36 AM
Hi, Can somebody provide me steps/guidance to install Apache Impala on HDP 2.3.2 cluster. Would really appreciate the help. Thanks,
... View more
Labels:
01-18-2016
05:36 AM
Hi, HDP 2.3.2 cluster is set up on EC2 . When try to view the application logs for particular job it says " web page is not available" the application job url is mapped to internal IP rather than external . When try to view the logs of the application as its is mapped to internal IP . How can I map application logs to external IPs . P.S. - Using elastic IP (EC2) for the name node . Thanks, Divya
... View more
12-29-2015
09:42 AM
Finally resolved the issue the input data was not correct format so when
I was using Timestamp/DateType its was returning empty result set.
... View more
12-29-2015
09:41 AM
Finally resolved the issue the input data was not correct format so when I was using Timestamp/DateType its was returning empty result set.
... View more
12-29-2015
07:27 AM
Hi,
I configured below properties SPARK_LOCAL_IP IP address of the machine to bind to. SPARK_PUBLIC_DNS Hostname your Spark program will advertise to other machines.Now In spark-shell console log
15/12/29 02:11:18 INFO Utils: Successfully started service 'SparkUI' on port 4040.
15/12/29 02:11:18 INFO SparkUI: Started SparkUI at http://public-IP:4040
Now when I go this address http://public-IP:4040
Its redirecting to http://ip-private-ip.ap-southeast-1.compute.internal:8088/proxy/application_1451351202944_0007/ which is mapped to
spark.org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.param.PROXY_URI_BASES
in Spark shell environment variable
spark.driver.appUIAddress http://private-ip:4040
Am I mising any configuration ?
... View more
12-29-2015
05:56 AM
1 Kudo
Hi, I have HDP2.3.2 cluster installed in Amazon EC2. I want to update the IP adress of spark.driver.appUIAddress,which is currently mapped to private IP of EC2. Searched in spark config in ambari,could find spark.driver.appUIAddress property. Because of this private IP mapping I am unable to connect to Spark web UI. Would really appreciate the help. Attached the screen shot for reference Thanks, Divya
... View more
Labels:
12-28-2015
08:57 AM
Below code returns empty resullt set as I used TimeStamp as one of the StructField
15/12/28 03:34:27 INFO SparkILoop: Created sql context (with Hive support)..
SQL context available as sqlContext.
scala> import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.HiveContext
scala> import org.apache.spark.sql.hive.orc._
import org.apache.spark.sql.hive.orc._
scala> val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
15/12/28 03:34:57 WARN SparkConf: The configuration key 'spark.yarn.applicationMaster.waitTries' has been deprecated as of Spark 1.3 and and may be removed in the future. Please use the new key 'spark.yarn.am.waitTime' instead.
15/12/28 03:34:57 INFO HiveContext: Initializing execution hive, version 0.13.1
hiveContext: org.apache.spark.sql.hive.HiveContext = org.apache.spark.sql.hive.HiveContext@3413fbe
scala> import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType,FloatType ,LongType ,TimestampType,NullType };
import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType, FloatType, LongType, TimestampType, NullType}
scala> val loandepoSchema = StructType(Seq(
| StructField("COLUMN1", StringType, true),
| StructField("COLUMN2", StringType , true),
| StructField("COLUMN3", TimestampType , true),
| StructField("COLUMN4", TimestampType , true),
| StructField("COLUMN5", StringType , true),
| StructField("COLUMN6", StringType, true),
| StructField("COLUMN7", IntegerType, true),
| StructField("COLUMN8", IntegerType, true),
| StructField("COLUMN9", StringType, true),
| StructField("COLUMN10", IntegerType, true),
| StructField("COLUMN11", IntegerType, true),
| StructField("COLUMN12", IntegerType, true),
| StructField("COLUMN13", StringType, true),
| StructField("COLUMN14", StringType, true),
| StructField("COLUMN15", StringType, true),
| StructField("COLUMN16", StringType, true),
| StructField("COLUMN17", StringType, true),
| StructField("COLUMN18", StringType, true),
| StructField("COLUMN19", StringType, true),
| StructField("COLUMN20", StringType, true),
| StructField("COLUMN21", StringType, true),
| StructField("COLUMN22", StringType, true)))
loandepoSchema: org.apache.spark.sql.types.StructType = StructType(StructField(COLUMN1,StringType,true), StructField(COLUMN2,StringType,true), StructField(COLUMN3,TimestampType,true), StructField(COLUMN4,TimestampType,true), StructField(COLUMN5,StringType,true), StructField(COLUMN6,StringType,true), StructField(COLUMN7,IntegerType,true), StructField(COLUMN8,IntegerType,true), StructField(COLUMN9,StringType,true), StructField(COLUMN10,IntegerType,true), StructField(COLUMN11,IntegerType,true), StructField(COLUMN12,IntegerType,true), StructField(COLUMN13,StringType,true), StructField(COLUMN14,StringType,true), StructField(COLUMN15,StringType,true), StructField(COLUMN16,StringType,true), StructField(COLUMN17,StringType,true), StructField(COLUMN18,StringType,true), StructField(COLUMN19,Strin...
scala> val lonadepodf = hiveContext.read.format("com.databricks.spark.csv").option("header", "true").schema(loandepoSchema).load("/tmp/TestDivya/loandepo_10K.csv")
15/12/28 03:37:52 INFO HiveContext: Initializing HiveMetastoreConnection version 0.13.1 using Spark classes.
lonadepodf: org.apache.spark.sql.DataFrame = [COLUMN1: string, COLUMN2: string, COLUMN3: timestamp, COLUMN4: timestamp, COLUMN5: string, COLUMN6: string, COLUMN7: int, COLUMN8: int, COLUMN9: string, COLUMN10: int, COLUMN11: int, COLUMN12: int, COLUMN13: string, COLUMN14: string, COLUMN15: string, COLUMN16: string, COLUMN17: string, COLUMN18: string, COLUMN19: string, COLUMN20: string, COLUMN21: string, COLUMN22: string]
scala> lonadepodf.select("COLUMN1").show(10)
15/12/28 03:38:01 INFO MemoryStore: ensureFreeSpace(216384) called with curMem=0, maxMem=278302556
15/12/28 03:38:01 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 211.3 KB, free 265.2 MB)
...............................................................................
15/12/28 03:38:07 INFO DAGScheduler: ResultStage 2 (show at <console>:33) finished in 0.653 s
15/12/28 03:38:07 INFO YarnScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool
15/12/28 03:38:07 INFO DAGScheduler: Job 2 finished: show at <console>:33, took 0.669388 s
+-------+
|COLUMN1|
+-------+
+-------+
Once Timestamp StructField is removed . Result set is returned
scala> val loandepoSchema = StructType(Seq(
| StructField("COLUMN1", StringType, true),
| StructField("COLUMN2", StringType , true),
| StructField("COLUMN3", StringType , true),
| StructField("COLUMN4", StringType , true),
| StructField("COLUMN5", StringType , true),
| StructField("COLUMN6", StringType, true),
| StructField("COLUMN7", IntegerType, true),
| StructField("COLUMN8", IntegerType, true),
| StructField("COLUMN9", StringType, true),
| StructField("COLUMN10", IntegerType, true),
| StructField("COLUMN11", IntegerType, true),
| StructField("COLUMN12", IntegerType, true),
| StructField("COLUMN13", StringType, true),
| StructField("COLUMN14", StringType, true),
| StructField("COLUMN15", StringType, true),
| StructField("COLUMN16", StringType, true),
| StructField("COLUMN17", StringType, true),
| StructField("COLUMN18", StringType, true),
| StructField("COLUMN19", StringType, true),
| StructField("COLUMN20", StringType, true),
| StructField("COLUMN21", StringType, true),
| StructField("COLUMN22", StringType, true)))
loandepoSchema: org.apache.spark.sql.types.StructType = StructType(StructField(COLUMN1,StringType,true), StructField(COLUMN2,StringType,true), StructField(COLUMN3,StringType,true), StructField(COLUMN4,StringType,true), StructField(COLUMN5,StringType,true), StructField(COLUMN6,StringType,true), StructField(COLUMN7,IntegerType,true), StructField(COLUMN8,IntegerType,true), StructField(COLUMN9,StringType,true), StructField(COLUMN10,IntegerType,true), StructField(COLUMN11,IntegerType,true), StructField(COLUMN12,IntegerType,true), StructField(COLUMN13,StringType,true), StructField(COLUMN14,StringType,true), StructField(COLUMN15,StringType,true), StructField(COLUMN16,StringType,true), StructField(COLUMN17,StringType,true), StructField(COLUMN18,StringType,true), StructField(COLUMN19,StringType,...
scala> val lonadepodf = hiveContext.read.format("com.databricks.spark.csv").option("header", "true").schema(loandepoSchema).load("/tmp/TestDivya/loandepo_10K.csv")
lonadepodf: org.apache.spark.sql.DataFrame = [COLUMN1: string, COLUMN2: string, COLUMN3: string, COLUMN4: string, COLUMN5: string, COLUMN6: string, COLUMN7: int, COLUMN8: int, COLUMN9: string, COLUMN10: int, COLUMN11: int, COLUMN12: int, COLUMN13: string, COLUMN14: string, COLUMN15: string, COLUMN16: string, COLUMN17: string, COLUMN18: string, COLUMN19: string, COLUMN20: string, COLUMN21: string, COLUMN22: string]
scala> lonadepodf.select("COLUMN1").show(10)
15/12/28 03:39:48 INFO BlockManagerInfo: Removed broadcast_8_piece0 on 172.31.20.85:40013 in memory (size: 4.2 KB, free: 265.3 MB)
15/12/28 03:39:49 INFO YarnScheduler: Removed TaskSet 6.0, whose tasks have all completed, from pool
15/12/28 03:39:49 INFO DAGScheduler: Job 6 finished: show at <console>:33, took 0.223277 s
+-------+
|COLUMN1|
+-------+
| CTR0|
| CTR1|
| CTR2|
| CTR3|
| CTR4|
| CTR5|
| CTR6|
| CTR7|
| CTR8|
| CTR9|
+-------+
... View more
Labels:
12-28-2015
02:39 AM
1 Kudo
Hi,
Sample code
spark-shell --packages com.databricks:spark-csv_2.10:1.1.0 --master yarn-client --driver-memory 512m --executor-memory 512m
import org.apache.spark.sql.hive.HiveContext
import org.apache.spark.sql.hive.orc._
val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc)
import org.apache.spark.sql.types.{StructType, StructField, StringType, IntegerType,FloatType ,LongType ,TimestampType };
val firstSchema = StructType(Seq(StructField("COLUMN1", StringType, true),StructField("COLUMN2", StringType, true),StructField("COLUMN2", StringType, true),StructField("COLUMN3", StringType, true)
StructField("COLUMN4", StringType, true),StructField("COLUMN5", StringType, true)))
val file1df = hiveContext.read.format("com.databricks.spark.csv").option("header", "true").schema(firstSchema).load("/tmp/File1.csv")
val secondSchema = StructType(Seq(
StructField("COLUMN1", StringType, true),
StructField("COLUMN2", NullType , true),
StructField("COLUMN3", TimestampType , true),
StructField("COLUMN4", TimestampType , true),
StructField("COLUMN5", NullType , true),
StructField("COLUMN6", StringType, true),
StructField("COLUMN7", IntegerType, true),
StructField("COLUMN8", IntegerType, true),
StructField("COLUMN9", StringType, true),
StructField("COLUMN10", IntegerType, true),
StructField("COLUMN11", IntegerType, true),
StructField("COLUMN12", IntegerType, true)))
val file2df = hiveContext.read.format("com.databricks.spark.csv").option("header", "false").schema(secondSchema).load("/tmp/file2.csv")
val joineddf = file1df.join(file2df, file1df("COLUMN1") === file2df("COLUMN6"))
val selecteddata = joineddf.select(file1df("COLUMN2"),file2df("COLUMN10"))
//displaying the joined data in console
joineddf.collect.foreach(println)
//Saving just the column names ,no joined data inside
selecteddata.write.format("com.databricks.spark.csv").option("header", "true").save("/tmp/JoinedData.csv")
... View more
Labels:
