About _Umesh

_Umesh · ‎07-30-2019

Since the "list" commands gets the apps from the ResourceManager and doesn't set any explicit filters and limits (except those provided with it) on the request, technically it returns all the applications which are present with RM at the moment. That number is controlled by "yarn.resourcemanager.max-completed-applications" config. Hope that clarifies.

_Umesh · ‎06-07-2019

As your intent seems to capture the driver logs in a separate file while executing the app in the cluster mode, make sure that ' /some/path/to/edgeNode/' dir is present on all of the NodeManager essentially as in cluster mode the driver will be running in the Yarn app's application master. If you can't make sure that follow a general practice to provide log file path to some pre-existing paths e.g. "/var/log/SparkDriver.log".

_Umesh · ‎05-14-2019

Please check if numpy is actually installed on all of the nodemanagers, if not, install it using below command (for python2.x) : pip install numpy If already installed, let us know the following: 1) Can you execute the same command outside of hue i.e. using Spark2-submit ? Mention the full command here. 2) What spark command you use in Hue?

_Umesh · ‎08-20-2018

Thanks for reporting 404 for that parcel URL, apologies for the inconvenience caused. However, I can see that the mentioned JIRA ( SPARK-22306 ) fix is present in below CDS releases : SPARK2-2.3.0-CLOUDERA1 SPARK2-2.3.0-CLOUDERA2 SPARK2-2.3.0-CLOUDERA3 So feel free to use below link to download the CDS 2.3 release 3 parcels in the meantime: http://archive.cloudera.com/spark2/parcels/2.3.0.cloudera3/

_Umesh · ‎03-06-2018

I believe you can achieve this by following the below sequence: spark.sql("SET spark.sql.shuffle.partitions=12") Execute operations on small table spark.sql("SET spark.sql.shuffle.partitions=500") Execute operations on larger table

_Umesh · ‎02-19-2018

It would be good if you can attach full stacktrace of the error which you are seeing. As a side note, make sure that you have added Hive gateway role on the host from which you are submitting spark app.

_Umesh · ‎08-11-2017

Gopa, There is no need to pool the client connections as KafkaProducer and KafkaConsumer usually keeps the broker connection open and allow you to produce/consume the events bypassing the connection creation overhead. The connection can be closed by calling close() method of each clients. Also as a side-note, t he producer is thread safe and sharing a single producer instance across threads will generally be faster than having multiple instances. However the Kafka consumer is NOT thread-safe. All network I/O happens in the thread of the application making the call. It is the responsibility of the user to ensure that multi-threaded access is properly synchronized.

_Umesh · ‎04-14-2017

It is the below line which is setting the data types for both the fields as StringType: val schema = StructType( schemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true))) You can define your custom schema as follows : val customSchema = StructType(Array( StructField("name", StringType, true), StructField("age", IntegerType, true))) You can add additional fields as well in the above schema definition. And then you can use this customSchema while creating the dataframe as follows: val peopleDataFrame = sqlContext.createDataFrame(rowRDD, customSchema) Also for details, please see this page.

_Umesh · ‎03-06-2017

By Spark 2.1 do you mean Cloudera Spark 2.0 Release 1 or Apache Spark 2.1 ? Regarding Cloudera Spark 2.0 Release 1 or Release 2, I would like to tell that minimum required CDH version is CDH 5.7.x but you are on CDH 5.5.4.

_Umesh · ‎11-23-2016

Spark SQL in Spark 1.6 prints the physical plan which is supposed to list all the available dirs (InputPaths) rather actual directories to scan while fetching the results for the queries. But while reading the data, it reads only relevant partition file and the same can be seen from "INFO" log messages. You can turn on the "INFO" level logging by setting: sc.setLogLevel("INFO") in spark-shell.

_Umesh · ‎08-30-2016

The valuable information is at very bottom: NameError: name 'master' is not defined Please make sure you have defined variable "master" in your code. Or if you are specifying master via spark-submit, you should not set it in code.

_Umesh · ‎08-29-2016

Well, it should work, but you can try --driver-class-path as you are using yarn-client mode and for sure Cloudera has a simmiler implementation. Please check here and this blog.

_Umesh · ‎08-28-2016

Try passing the conf files as below --conf spark.driver.extraClassPath=<your config path here> --conf spark.executor.extraClassPath=<your config path here>

_Umesh · ‎08-03-2016

You don't need to export as a JAR for unit testing. You can do : SparkConf().setMaster(local[2]) and run the program as usual Java application in IDE. Also make sure that you have all the dependent libraries in the classpath.

_Umesh · ‎08-03-2016

You are getting this exception because "sc.testFile" r eads a text file from HDFS, a local file system (available on all nodes), or any Hadoop-supported file system URI. You said that you want to get the data from URL and want to save it to HDFS, then you should do: val data = scala.io.Source.fromURL("http://10.3.9.34:9900/messages").mkString val list = data.split("\n").filter(_ != "") val rdds = sc.parallelize(list) rdds.saveAsTextFile(outputDirectory)

_Umesh · ‎06-22-2016

The attached log indicates that application is accepted by cluster manager (YARN) but unable to execute due to resource crunch. Please make sure there are enough resources available in your cluster while submitting the job. Do check following and configure based on hosts: yarn.nodemanager.resource.memory-mb yarn.nodemanager.resource.cpu-vcores yarn.scheduler.maximum-allocation-mb yarn.scheduler.minimum-allocation-mb yarn-scheduler.maximum-allocation-vcores yarn.scheduler.minimum-allocation-vcores

_Umesh · ‎06-21-2016

This looks strange. Your console output listed the below lines com.databricks#spark-avro_2.10 added as a dependency org.apache.avro#avro-mapred added as a dependency Can you try once with : --packages com.databricks:spark-avro_2.10:1.0.0,org.apache.avro:avro-mapred:1.6.3 I can sense some version compatibility issues of avro-mapred with spark-avro.

_Umesh · ‎06-15-2016

Try starting spark-shell with following packages: --packages com.databricks:spark-avro_2.10:2.0.1,org.apache.avro:avro-mapred:1.7.7

_Umesh · ‎06-10-2016

What Python version you are using. You may want to refer : http://www.cloudera.com/documentation/enterprise/5-5-x/topics/spark_ipython.html

_Umesh · ‎05-30-2016

See the Environment tab of Job History UI and locate " spark.local.dir". Yes that is the expected behaviour as JAR is required to the executors.

_Umesh · ‎05-30-2016

This looks weird. And can you confirm that http://192.168.88.28:55310/jars/phoenix-1.2.0-client.jar is still not present? Spark keeps all JARs specified by --jars option in job's temp directory on each executor nodes [1]. There must be some sort of OS settings which lead the deletion of existing phoenix jar from temp and when Spark Context is unable to find it at its usual location it tries to download it from the given location. However this should not happen until the temp directory is actively accessed by the job or process. You can try bundling that JAR with your Spark JAR and then refer it in spark-submit. I suspect, you will need again 20 odd days to test this workaround 🙂

_Umesh · ‎05-11-2016

You are messing with createPollingStream method. Give 198.168.1.31 as sink address as below and it should work. FlumeUtils.createPollingStream(ssc,"198.168.1.31",8020)

_Umesh · ‎05-11-2016

Add below dependency as well: groupId = org.apache.spark artifactId = spark-streaming-flume_2.10 version = 1.6.1 See here for pull based configuration.

_Umesh · ‎05-04-2016

Kafka in CDH 5.7 is v2.0 (v0.9.0) and there is a known issue which tells you should use Kafka not less than v0.9.0 with Spark in CDH 5.7. You can find it here .

_Umesh · ‎05-01-2016

Yes you can!

_Umesh · ‎04-27-2016

Its because the version of Spark SQL you are using (1.0.0) has not the toJSON() method. Is there any specific reason behind using these old versions? When I do: Umeshs-MacBook-Pro:~ umesh$ javap -classpath .m2/repository/org/apache/spark/spark-sql_2.10/1.2.1/spark-sql_2.10-1.2.1.jar org/apache/spark/sql/SchemaRDD | grep 'toJSON' public org.apache.spark.rdd.RDD<java.lang.String> toJSON(); Umeshs-MacBook-Pro:~ umesh$ I can see toJSON method in SparkSQL v 1.2.1 but when I do: Umeshs-MacBook-Pro:~ umesh$ javap -classpath Downloads/spark-sql_2.10-1.0.0.jar org/apache/spark/sql/SchemaRDD | grep 'toJSON' Umeshs-MacBook-Pro:~ umesh$ I cant see the method. So start using SparkSQL v 1.2.1 at least for getting this method and it will comply you Spark Core version also.

_Umesh · ‎04-20-2016

CM is supporting single version for Spark on YARN and single version for Standalone installation (Single version is common requirement). For supporting multiple versions of Spark you need to install it manually on a single node and copy the config files for YARN and Hive inside its conf directory. And when you refer the spark-submit of that version, it will distribute the Spark-core binary on each YARN nodes to execute your code. You don't need to install Spark on each YARN nodes.

_Umesh · ‎04-20-2016

For debugging your application in IDE, you need to setup remote debugging, this article is suitable for you. I can see in your logs : 16/04/19 07:08:41 INFO client.AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@10.0.2.15:7077/user/Master... I think there is a typo which you have made from my last reply, because of this your application is unable to connect to the master. Use your master's address instead and which spark-submit script you have used?

_Umesh · ‎04-19-2016

Yes, YARN provides this flexibility. Here you can find the detailed answer. For CDH there is a "Spark" service, which meant for YARN and another is "Spark Standalone" service which runs it's daemons standalone on the specified nodes. YARN will do the work for you if you want to test the multiple versions simultaneously. You should have your multiple versions on Gateway Host and then you can launch Spark applications from there.

_Umesh · ‎04-19-2016

My code: SparkConf conf = new SparkConf().setAppName( "HdfsToSolr" ) ; JavaSparkContext sc = new JavaSparkContext(conf) ; HiveContext hiveContext = new HiveContext(JavaSparkContext. toSparkContext (sc)) ; hiveContext.sql( "select * from test1" ).show() ; And it displayed me the records in the table . You should not specify "master" in the program, instead try to pass it with spark-submit. spark-submit --class SparkSQLTest /my/jar/path.jar --master spark://10.0.2.15:7077 for your case I can see: 16/04/18 16:10:21 INFO MetaStoreDirectSql: MySQL check failed, assuming we are not on mysql: Lexical error at line 1, column 5. Encountered: "@" (64), after : "". Check your configuration for hive: Hive Metastore Database Type , ideally it should be MySql. Try to submit the above code from the shell where VM is running.

Online	Offline
Last Visited	‎05-12-2021 11:29 PM

Date Registered	‎04-05-2016 11:32 PM
Last Visited	‎05-12-2021 11:29 PM
Total Messages Posted	36
Total Kudos Received	8

Re: How deep does command "yarn application -list ...

Re: Config log4j in Spark - Driver Logs

Re: How to define datatype when creating dataframe...

Re: create Analytics from http usng spark streamin...

Re: not able to launch PYSPARK CDH 5.5.0 and spark...

Re: How deep does command "yarn application -list ...

Re: Config log4j in Spark - Driver Logs

Re: ImportError: No module named numpy

Re: CDS Spark2.2 Release 3

Re: How to set multiple spark configurations for s...

Re: Getting error in Spark-Shell while using hive ...

Re: Kafka Producer Connection Pool is required or ...

Re: How to define datatype when creating dataframe...

Re: PySpark - Error initializing SparkContext

Re: sparkSQL not pruning directiories

Re: Issue running spark application in Yarn-cluste...

Re: Issues with Spark On Hbase Connector

Re: Issues with Spark On Hbase Connector

Re: Spark program in eclipse

Re: create Analytics from http usng spark streamin...

Re: not able to launch PYSPARK CDH 5.5.0 and spark...

Re: NoClassDefFoundError when using avro in spark-...

Re: NoClassDefFoundError when using avro in spark-...

Re: Unrecognized alias: '--profile=pyspark'

Re: Spark Streaming: FileNotFoundException on file...

Re: Spark Streaming: FileNotFoundException on file...

Re: Integration of Spark Streaming with Flume

Re: Integration of Spark Streaming with Flume

Re: run-example streaming.KafkaWordCount fails on ...

Re: toJSON is not a member of org.apache.spark.sql...

Re: toJSON is not a member of org.apache.spark.sql...

Re: Multiple Spark version on the same cluster

Re: Spark hive program in eclipse

Re: Multiple Spark version on the same cluster

Re: Spark hive program in eclipse