Hi, I am running a spark job to compress a file, most of the times its successful, but every once in a while this issue occurs and sometimes its stuck indefinitely. Since the job is still in running phase , we don't find any error in the yarn logs. It gets stuck right after the driver requests to kill the executor. In this particular job it got stuck in this phase for about 6 hours although it easily completed for data much smaller in size. Also, dynamic allocation is enabled in our cluster. 19/01/28 09:19:40 INFO YarnAllocator: Driver requested a total number of 1 executor(s). 19/01/28 09:19:40 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 2, 1. 19/01/28 09:19:41 INFO YarnAllocator: Driver requested a total number of 0 executor(s). 19/01/28 09:19:41 INFO ApplicationMaster$AMEndpoint: Driver requested to kill executor(s) 3. 19/01/28 15:14:39 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. 19/01/28 15:14:39 INFO ApplicationMaster$AMEndpoint: Driver terminated or disconnected! Shutting down. ... View more

I am trying to update multiple policies in a particular service simultaneously. Using the below command i am getting all the policies in a particular service curl -vk -H "Content-type:application/json" --netrc-file my-password-file -X GET http://localhost:6080/service/public/v2/api/service/POC_hadoop/policy -o rangerpolicies.json In the file obtained i.e. rangerpolicies.json , i am updating a few properties. While updating, using the put command, i am facing error which i have attached as attachment. error.txt The command used:- curl -vk "Content-type:application/json" --netrc-file my-password-file -X PUT -d @rangerpolicies.json http://localhost:6080/service/public/v2/api/service/POC_hadoop/policy/ A single policy-wise, i am able to update, but is this option not available at service level? Also, is there any workaround available, because to update all policies one by one in case of 100+ policies is not feasible. Ranger version :0.7 Thanks in advance. ... View more

What is the the use of the the active status- enable/disable tab while creating a new ranger service? If the service is disabled, will the policies in the service not work anymore? I tried it, but they still work even after disabling the service. Ranger version=0.7 ... View more

I am trying to access pyspark where SPARK_MAJOR_VERSION is set to 2 by a certain user but I am getting the following error: at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:748) 18/05/14 10:18:18 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042. Traceback (most recent call last): File "/usr/hdp/current/spark2/python/pyspark/sql/utils.py", line 63, in deco return f(*a, **kw) File "/usr/hdp/current/spark2/python/lib/py4j-0.10.4-src.zip/py4j/protocol.py", line 319, in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o34.sessionState.b : java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveSessionState': at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$reflect(SparkSession.scala:983) at org.apache.spark.sql.SparkSession.sessionState$lzycompute(SparkSession.scala:110) at org.apache.spark.sql.SparkSession.sessionState(SparkSession.scala:109) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.sql.SparkSession$.org$apache$spark$sql$SparkSession$reflect(SparkSession.scala:980) ... 13 more Caused by: java.lang.IllegalArgumentException: Error while instantiating 'org.apache.spark.sql.hive.HiveExternalCatalog': at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$reflect(SharedState.scala:176) at org.apache.spark.sql.internal.SharedState.<init>(SharedState.scala:86) at org.apache.spark.sql.SparkSession$anonfun$sharedState$1.apply(SparkSession.scala:101) at org.apache.spark.sql.SparkSession$anonfun$sharedState$1.apply(SparkSession.scala:101) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession.sharedState$lzycompute(SparkSession.scala:101) at org.apache.spark.sql.SparkSession.sharedState(SparkSession.scala:100) at org.apache.spark.sql.internal.SessionState.<init>(SessionState.scala:157) at org.apache.spark.sql.hive.HiveSessionState.<init>(HiveSessionState.scala:32) ... 18 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.spark.sql.internal.SharedState$.org$apache$spark$sql$internal$SharedState$reflect(SharedState.scala:173) ... 26 more Although the same user is able to access pyspark from a different node in the same environment, it is only the edge node that is facing this error. Interestingly, other users are able to access pyspark from edge node too. I believe it is related to some permission issue but i am not sure where else to check for permission. Please suggest where can be the possible places to cause such an error. Thanks in advance. ... View more

Hi, Referring to https://docs.hortonworks.com/HDPDocuments/Ambari-2.6.0.0/bk_ambari-operations/content/archiving_and_purging_data.html it says that archiving and purging of logs is possible in Ambari. But is it possible for all components or only Atlas, Ranger, and Log Search? actually my aim is to run a script to archive and delete all logs for major hadoop components post a stipulated date so that my cluster doesn't get cluttered. But i came across this, will Ambari-infra help in achieving this purpose? Thanks for your help in advance. ... View more

When I Sqoop data from Teradata to Hive, I am running the command sqoop import --connect jdbc:teradata://link/database=db --connection-manager org.apache.sqoop.teradata.TeradataConnManager --username user -P --table table --hive-import -m 1 and getting the following error- 18/03/21 14:13:50 INFO teradata.TeradataSqoopImportHelper: Setting source table to : table 18/03/21 14:13:51 INFO common.ConnectorPlugin: load plugins in jar:file:/usr/hdp/current/hadoop/lib/teradata-connector-1.5.2-hadoop2.jar!/teradata.connector.plugins.xml 18/03/21 14:13:52 INFO teradata.TeradataSqoopImportHelper: Teradata import job completed with exit code 1 Exception in thread "main" java.lang.NoSuchFieldError: METASTORE_BATCH_RETRIEVE_OBJECTS_MAX at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:221) at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:209) at com.teradata.connector.hive.utils.HiveUtils.isHiveOutputTablePartitioned(HiveUtils.java:1140) at com.teradata.connector.common.tool.ConfigurationMappingUtils.importConfigurationMapping(ConfigurationMappingUtils.java:321) at org.apache.sqoop.teradata.TeradataSqoopImportHelper.runJob(TeradataSqoopImportHelper.java:364) at org.apache.sqoop.teradata.TeradataConnManager.importTable(TeradataConnManager.java:505) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:507) Any help is highly appreciated and thanks in advance.. ... View more

I want to reset my admin password for all the major components of Ambari namely, the ambari webui, mysql, smartsense, ranger etc. After I reset the password of all these places (for the admin user), what places are to be updated after such a reset is done so that my cluster doesn't get affected.. might be those places from where the admin password is being referred? Or the system updates such all those places on its own and we don't need to make any manual changes? I need to check if this won't pose a problem before all the password reset..and none of my services will get affected. thanks in advance. ... View more

I fetched details of job in zeppelin notebook query using phoenix interpreter and matched the submit time with the same job id in the Resource Manager UI in Ambari. I found that there is a two hour difference in both. How do i change the timezone in Zeppelin query? please help ... View more