Member since
08-29-2018
109
Posts
3
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
3283 | 11-14-2019 02:54 AM | |
10799 | 11-05-2019 07:51 PM |
02-10-2021
09:41 AM
It looks like you are running Spark shell on a Windows machine, maybe your local laptop. Is there anywhere in the code you are mentioning the hostname, "dclvmsbigdmd01"? If not, where is your 172.30.294.196 (hive.metastore.uris)? Does this IP resolve the name dclvmsbigdmd01? Can you review if the host/domain is reachable from your local?
... View more
07-14-2020
10:56 AM
Hey, Are there any parameters used in the spark-shell command? Usually, this delay happens for a lot of reasons from a connection time to resource availability. However, we cannot confirm anything with just the driver logs. In order to narrow this down, could you share the yarn log of the application for this application using the command, "yarn logs -applicationId application_1594337770867_0003"? We will have more clarity on what has been happening during the delay. Thanks
... View more
07-14-2020
06:11 AM
Hello, AFAIK, the Stanford CoreNLP wrapper for Apache Spark should not be a bottleneck in terms of parallel processing. Spark would take care of running it parallelly on multiple documents. Regardless of the number of documents, the number of API requests to the CoreNLP server would remain the same.
... View more
07-14-2020
04:02 AM
Hey, could you share the exact trace of output that you receive? If the issue is on the WebUI, could you also share the Screenshot of what you see?
... View more
05-08-2020
10:55 PM
Okay, let me know if changing HiveContext to SparkContext makes any difference. It could give a lead to resolution.
... View more
05-08-2020
02:20 AM
Hi @clvi, Try adding the --appOwner <username> to the yarn logs command. However, I think the application states are erased from the RM state store, probably due to an RM State Restore.
... View more
04-12-2020
06:36 AM
Hey @hicha , what is the version of Spark you are using? What do you receive as the output, when using the `Spark Session`, instead of `Hive Context`?
... View more
03-23-2020
05:29 AM
Hello RIshab, Can you please mention the error you are facing exactly?
... View more
11-14-2019
03:06 AM
Hey @gnish, Thanks for asking. I haven't tried it before, but hope you have come across the Conversion utility from Zeppelin notes to Jupyter notebooks [1]. However, I notice that the JIRA - ZEPPELIN-2616 [2] for documentation for this feature seems yet to be resolved. [1] https://github.com/rdblue/jupyter-zeppelin [2] https://issues.apache.org/jira/browse/ZEPPELIN-2616
... View more
11-14-2019
02:54 AM
Hey @avengers, Just thought, this could add some more value to this question here. Spark SQL uses a Hive Metastore to manage the metadata of persistent relational entities (e.g. databases, tables, columns, partitions) in a relational database (for fast access) [1]. Also, I don't think there would be a MetaStore crash if we use it along with HiveOnSpark. [1] https://jaceklaskowski.gitbooks.io/mastering-spark-sql/spark-sql-hive-metastore.html
... View more