About gsthina

muslihuddin · ‎03-28-2021

Hi @jake_allston , do you find the culprit? I've got similar issue. Took 5 minutes to load spark-shell. Its a new cluster. Doen not happen to my other cluster

gsthina · ‎02-10-2021

It looks like you are running Spark shell on a Windows machine, maybe your local laptop. Is there anywhere in the code you are mentioning the hostname, "dclvmsbigdmd01"? If not, where is your 172.30.294.196 (hive.metastore.uris)? Does this IP resolve the name dclvmsbigdmd01? Can you review if the host/domain is reachable from your local?

gsthina · ‎07-14-2020

Hello, AFAIK, the Stanford CoreNLP wrapper for Apache Spark should not be a bottleneck in terms of parallel processing. Spark would take care of running it parallelly on multiple documents. Regardless of the number of documents, the number of API requests to the CoreNLP server would remain the same.

gsthina · ‎07-14-2020

Hey, could you share the exact trace of output that you receive? If the issue is on the WebUI, could you also share the Screenshot of what you see?

gsthina · ‎05-08-2020

Okay, let me know if changing HiveContext to SparkContext makes any difference. It could give a lead to resolution.

gsthina · ‎05-08-2020

Hi @clvi, Try adding the --appOwner <username> to the yarn logs command. However, I think the application states are erased from the RM state store, probably due to an RM State Restore.

gsthina · ‎03-23-2020

Hello RIshab, Can you please mention the error you are facing exactly?

gnish · ‎11-14-2019

@gsthina .. Thanks for your help. Yeah I have checked it already. Utility present on [1] says that it can convert Zeppelin to Jupyter but my requirement is to convert jupyter to zeppelin. And my requirement is to convert python code (jupyter notebook) to pyspark (zeppelin notebook) which is also not supported by this converter. You can check in the description.

gsthina · ‎11-14-2019

Hey @avengers, Just thought, this could add some more value to this question here. Spark SQL uses a Hive Metastore to manage the metadata of persistent relational entities (e.g. databases, tables, columns, partitions) in a relational database (for fast access) [1]. Also, I don't think there would be a MetaStore crash if we use it along with HiveOnSpark. [1] https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-hive-metastore.html

gsthina · ‎11-05-2019

Hi @wret_1311, Thanks for your response and I appreciate for confirming the solution. I'm glad, it helped you 🙂

Online	Offline
Last Visited	‎11-22-2024 03:18 AM

Member Since	‎08-29-2018 12:11 AM
Last Visited	‎11-22-2024 03:18 AM
Posts	109
Kudos received	2

Cloudera Community

Re: Can I use SparkSQL on a cluster using Hive on ...

Re: Yarn container logs

Re: Loading spark shell takes like 6 mins from my ...

Re: Py4JJavaError: An error occurred while calling...

Re: CoreNLP Server on Spark

Re: mapreduce job does not have tracking url

Re: Quering hive table from spark-shell

Re: Difference between yarn ui running application...

Re: Failed to receive heartbeat from agent.

Re: How to convert jupyter notebook to Zeppelin no...

Re: Can I use SparkSQL on a cluster using Hive on ...

Re: Yarn container logs