Hi All,   While working with spark 2.3, Hive and yarn (HDP 3.1), Every job works fine and competes gracefully. But Overall spark job takes the same time as it takes time with larger data even if we have very small data.    For example: scheduling job on yarn takes some time either we have large data or small data. So, simple spark queries on small data also takes time and finish in 45 secs to 1+ mins (I guess, which includes yarn's scheduling and resource management time) and databases takes only few seconds to run same query.   Can we reduce the time with spark if we are using HDP 3.1 ---- 6 machines cluster. OR do we have any another mode to run spark with small data available in Hive in less time at-least for testing only.   Thanks ... View more

HI All, I am working to integrate ranger and presto and following this URL but ended with error and integration is not successful. Steps: Created a ranger-${RANGER_VERSION}-presto-plugin.tar.gz file. Extracted tar file to ranger-presto-plugin Changed install.properies file. Run ./enable-presto-plugin.sh It was successful run but It did not create access-control.properties & rules.json file. And I created these files manually in /etc/presto. Now presto is not starting and failed with the below error. 2019-06-21T04:59:21.731Z ERROR main com.facebook.presto.server.PrestoServer Access control ranger is not registered java.lang.IllegalStateException: Access control ranger is not registered at com.google.common.base.Preconditions.checkState(Preconditions.java:585) at com.facebook.presto.security.AccessControlManager.setSystemAccessControl(AccessControlManager.java:136) at com.facebook.presto.security.AccessControlManager.loadSystemAccessControl(AccessControlManager.java:118) at com.facebook.presto.server.PrestoServer.run(PrestoServer.java:142) at com.facebook.presto.server.PrestoServer.main(PrestoServer.java:73) Do we need to use ranger-servicedef-presto.json anywhere because its not mentioned in ranger presto integration URL. OR any another link which I can follow which is having all information. Thanks Bharat Bhushan ... View more

Hi All, I have a already setup HDP 3.0 cluster having total 7 machines with 3 worker nodes. I also installed Ranger to do security and a user already created on hive. Now I want to access hive table using spark. I have tried both using spark-shell and java code But both is getting same error. I am using spark llap to provide username/password for hive. But spark got an Exception:"No service instances found in registry" Log scala> df.show [Stage 0:> (0 + 0) / 1]18/12/10 17:08:51 WARN TaskSetManager: Stage 0 contains a task of very large size (423 KB). The maximum recommended task size is 100 KB.[Stage 0:> (0 + 1) / 1]18/12/10 17:08:55 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, s1.us-east4-c.c.asdf-224606.internal, executor 2): java.lang.RuntimeException: java.io.IOException: No service instances found in registry at com.hortonworks.spark.sql.hive.llap.HiveWarehouseDataReaderFactory.createDataReader(HiveWarehouseDataReaderFactory.java:66) If my table is having 0 records than it works fine and show me a blank table but if records are greater than 0 than it gives me exception. Thanks in advanced. ... View more