Member since
09-06-2017
14
Posts
1
Kudos Received
0
Solutions
01-29-2019
08:28 AM
Hi, I am running a spark job to compress a file, most of the times its successful, but every once in a while this issue occurs and sometimes its stuck indefinitely. Since the job is still in running phase , we don't find any error in the yarn logs. It gets stuck right after the driver requests to kill the executor. In this particular job it got stuck in this phase for about 6 hours although it easily completed for data much smaller in size. Also, dynamic allocation is enabled in our cluster. 19/01/28 09:19:40 INFO YarnAllocator: Driver requested a total number of 1 executor(s).
19/01/28 09:19:40 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 2, 1.
19/01/28 09:19:41 INFO YarnAllocator: Driver requested a total number of 0 executor(s).
19/01/28 09:19:41 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 3.
19/01/28 15:14:39 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. 19/01/28 15:14:39 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down.
... View more
Labels:
11-15-2018
10:19 PM
Please look at import API. http://ranger.apache.org/apidocs/resource_ServiceREST.html#resource_ServiceREST_importPoliciesFromFile_POST
... View more
10-16-2018
02:37 PM
@Vani Deeppak Q: If the service is disabled, will the policies in the service not work anymore? I Ans: Correct, if service is disabled this means it does not work with ranger authorization. Therefore no policies defined for the service will be evaluated/enforced at runtime. Q: What is the the use of the the active status- enable/disable tab while creating a new ranger service? Ans: You have the option to decide per-service if you like to use ranger for authorization. Lets say you have configured authorization for hive in metastore you can continue to use this if you leave it set to disabled. If you set it to enabled you need to configure the authorization at the ranger policy level. HTH
... View more
05-16-2018
09:22 AM
I am trying to access pyspark where SPARK_MAJOR_VERSION is set to 2 by a certain user but I am getting the following error: at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
18/05/14 10:18:18 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
Traceback (most recent call last):
File "/usr/hdp/current/spark2/python/pyspark/sql/utils.py", line 63, in deco
return f(*a, **kw)
File "/usr/hdp/current/spark2/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o34.sessionState.b
: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState':
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$reflect(SparkSession.scala:983)
at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110)
at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:280)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:214)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$reflect(SparkSession.scala:980)
... 13 more
Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog':
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$reflect(SharedState.scala:176)
at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86)
at org.apache.spark.sql.SparkSession$anonfun$sharedState$1.apply(SparkSession.scala:101)
at org.apache.spark.sql.SparkSession$anonfun$sharedState$1.apply(SparkSession.scala:101)
at scala.Option.getOrElse(Option.scala:121)
at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101)
at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100)
at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157)
at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32)
... 18 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$reflect(SharedState.scala:173)
... 26 more Although the same user is able to access pyspark from a different node in the same environment, it is only the edge node that is facing this error. Interestingly, other users are able to access pyspark from edge node too. I believe it is related to some permission issue but i am not sure where else to check for permission. Please suggest where can be the possible places to cause such an error. Thanks in advance.
... View more
04-02-2018
01:12 PM
Hi, Referring to https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/archiving_and_purging_data.html it says that archiving and purging of logs is possible in Ambari. But is it possible for all components or only Atlas, Ranger, and Log Search? actually my aim is to run a script to archive and delete all logs for major hadoop components post a stipulated date so that my cluster doesn't get cluttered. But i came across this, will Ambari-infra help in achieving this purpose? Thanks for your help in advance.
... View more
Labels:
03-22-2018
06:38 AM
@Vani Deeppak Have a look at this article: https://community.hortonworks.com/articles/53531/importing-data-from-teradata-into-hive.html it has the link(s) for Sqoop documentation and also talks about how to use it. Hope this helps.
... View more
03-19-2018
08:40 AM
@Jay Kumar SenSharmaThank you so much for your swift response , I wanted to confirm one thing more. If, for example I reset my admin password for mysql/ranger will that affect any service because I have database of ranger, oozie and ambari in that or in case of ranger i have policies for services there. I think not but just wanted to confirm. Also. if there does not exist any common place / configuration to update all the individual components can i achieve this with a script? using curl commands to call to change password?
... View more
09-07-2017
08:18 AM
I fetched details of job in zeppelin notebook query using phoenix interpreter and matched the submit time with the same job id in the Resource Manager UI in Ambari. I found that there is a two hour difference in both. How do i change the timezone in Zeppelin query? please help
... View more