Hi , can we audit KMS operations like  ACL operation - adding /removing user from KMS ACL key operation  ... View more

  Hi , I have created a cluster in aws but service which connect to mysql database like oozie /hive metastore /hue are not starting since they are not able to connect  with mysql . i am getting below error , not sure what is wrong with cred files  RROR: Oozie could not be started REASON: org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Could not load password for [oozie.service.JPAService.jdbc.password] Stacktrace: ----------------------------------------------------------------- org.apache.oozie.service.ServiceException: E0103: Could not load service classes, Could not load password for [oozie.service.JPAService.jdbc.password] at org.apache.oozie.service.Services.loadServices(Services.java:309) at org.apache.oozie.service.Services.init(Services.java:213) at org.apache.oozie.servlet.ServicesLoader.contextInitialized(ServicesLoader.java:46) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4276) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4779) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:803) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:780) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:583) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:944) at org.apache.catalina.startup.HostConfig.deployWARs(HostConfig.java:779) at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:505) at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1322) at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:325) at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1069) at org.apache.catalina.core.StandardHost.start(StandardHost.java:822) at org.apache.catalina.core.ContainerBase.start(ContainerBase.java:1061) at org.apache.catalina.core.StandardEngine.start(StandardEngine.java:463) at org.apache.catalina.core.StandardService.start(StandardService.java:525) at org.apache.catalina.core.StandardServer.start(StandardServer.java:761) at org.apache.catalina.startup.Catalina.start(Catalina.java:595) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:289) at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:414) Caused by: java.lang.IllegalArgumentException: Could not load password for [oozie.service.JPAService.jdbc.password] at org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:598) at org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:585) at org.apache.oozie.service.JPAService.init(JPAService.java:160) at org.apache.oozie.service.Services.setServiceInternal(Services.java:386) at org.apache.oozie.service.Services.setService(Services.java:372) at org.apache.oozie.service.Services.loadServices(Services.java:305) ... 26 more Caused by: java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.oozie.service.ConfigurationService.getPassword(ConfigurationService.java:591) ... 31 more Caused by: java.io.IOException: Configuration problem with provider path. at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2118) at org.apache.hadoop.conf.Configuration.getPassword(Configuration.java:2037) ... 36 more Caused by: java.io.IOException: Bad configuration of hadoop.security.credential.provider.path at localjceks://file/{{CMF_CONF_DIR}}/creds.localjceks at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:86) at org.apache.hadoop.conf.Configuration.getPasswordFromCredentialProviders(Configuration.java:2098) ... 37 more Caused by: java.net.URISyntaxException: Illegal character in path at index 18: localjceks://file/{{CMF_CONF_DIR}}/creds.localjceks at java.net.URI$Parser.fail(URI.java:2829) at java.net.URI$Parser.checkChars(URI.java:3002) at java.net.URI$Parser.parseHierarchical(URI.java:3086) at java.net.URI$Parser.parse(URI.java:3034) at java.net.URI.<init>(URI.java:595) at org.apache.hadoop.security.alias.CredentialProviderFactory.getProviders(CredentialProviderFactory.java:67) ... 38 more -----------------------------------------------------------------       ... View more

Hi , do we have a coomand to check kudu table size. like hive and hbase table we can check size on hdfs . do we have similar thing for kudu  ... View more

Hi , I have shutdown hbase cluster and moved all hdfs file to s3 . now we need to create or move this backup(hdfs files ) to a new hbase cluster . i have table defination so i can create the table and also copy the data into new cluster like  /hbase/data/namespace/tablename .   i can see table but no data . how to load the data into table .   do i need meta table information from old cluster ? or i can create a new meta for table in new cluster so it detect the data . ... View more

Hi , I want to take cluster backup to S3 and then wipe out the cluster ,may be later spin it again . for taking backup to s3 what is the best apporoach . 1- copy the entire hdfs content to S3 ( in single bucket or create mutiple bucket) 2- do i need to take hbase snapshot along with /hbase copy to s3 or either one will work. 3- REST encryption is enabled ,any special consideration to take while moving backup to S3 4- and how restore will work from  S3 , just copy to hdfs  ... View more

  How to drop hive solr table . i tried to delete collection as well but that didnt help.   hive> drop table compliance.solr_test; FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. MetaException(message:Failed to load storage handler: Error in loading storage handler.com.chimpler.hive.solr.SolrStorageHandler) hive> ... View more

Hi, does impala support automatic metadata update from hive (DDL/DML) ? is this available in any version  ... View more

Hi , I am trying to understand this situation .   Lets say i have 5 impalad sever and if anyone one them is down impala queries will get timeout error or will not execute . what is the reason behind this ?    ... View more

HI, I am trying to authentocate hadoop client on windows server with linunx kdc . can anyone provide a document or link for such integration   hadoop client - windows server 2012 kdc - linux  hadoop cluster - linux ... View more

Hi  code is not executing on terminal window . its failing with    Setting default log level to "ERROR". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/07/18 16:12:28 ERROR spark.SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: java.net.UnknownHostException: wewcw0hd3mn02.wedip.com at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:728) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:671) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:155) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2815) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2852) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2834) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:186) at org.apache.spark.deploy.yarn.Client$$anonfun$7.apply(Client.scala:123) at org.apache.spark.deploy.yarn.Client$$anonfun$7.apply(Client.scala:123) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.deploy.yarn.Client.<init>(Client.scala:123) at org.apache.spark.deploy.yarn.Client.<init>(Client.scala:69) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:55) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:171) at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at org.apache.toree.kernel.api.Kernel.createSparkContext(Kernel.scala:344) at org.apache.toree.kernel.api.Kernel.createSparkContext(Kernel.scala:363) at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeSparkContext(ComponentInitialization.scala:103) at org.apache.toree.Main$$anon$1.initializeSparkContext(Main.scala:35) at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86) at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35) at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:100) at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:40) at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24) at scala.Function0$class.apply$mcV$sp(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.App$class.main(App.scala:76) at org.apache.toree.Main$.main(Main.scala:24) at org.apache.toree.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.net.UnknownHostException: wewcw0hd3mn02.wedip.com ... 52 more 17/07/18 16:12:28 ERROR util.Utils: Uncaught exception in thread main java.lang.AssertionError: assertion failed: Attempted to stop this scheduler before starting it! at scala.Predef$.assert(Predef.scala:170) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:138) at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:496) at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1588) at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1826) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1283) at org.apache.spark.SparkContext.stop(SparkContext.scala:1825) at org.apache.spark.SparkContext.<init>(SparkContext.scala:587) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at org.apache.toree.kernel.api.Kernel.createSparkContext(Kernel.scala:344) at org.apache.toree.kernel.api.Kernel.createSparkContext(Kernel.scala:363) at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeSparkContext(ComponentInitialization.scala:103) at org.apache.toree.Main$$anon$1.initializeSparkContext(Main.scala:35) at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86) at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35) at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:100) at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:40) at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24) at scala.Function0$class.apply$mcV$sp(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.App$class.main(App.scala:76) at org.apache.toree.Main$.main(Main.scala:24) at org.apache.toree.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 17/07/18 16:12:28 ERROR util.Utils: Uncaught exception in thread main java.lang.NullPointerException at org.apache.spark.network.shuffle.ExternalShuffleClient.close(ExternalShuffleClient.java:152) at org.apache.spark.storage.BlockManager.stop(BlockManager.scala:1390) at org.apache.spark.SparkEnv.stop(SparkEnv.scala:89) at org.apache.spark.SparkContext$$anonfun$stop$11.apply$mcV$sp(SparkContext.scala:1842) at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1283) at org.apache.spark.SparkContext.stop(SparkContext.scala:1841) at org.apache.spark.SparkContext.<init>(SparkContext.scala:587) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at org.apache.toree.kernel.api.Kernel.createSparkContext(Kernel.scala:344) at org.apache.toree.kernel.api.Kernel.createSparkContext(Kernel.scala:363) at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeSparkContext(ComponentInitialization.scala:103) at org.apache.toree.Main$$anon$1.initializeSparkContext(Main.scala:35) at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86) at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35) at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:100) at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:40) at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24) at scala.Function0$class.apply$mcV$sp(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.App$class.main(App.scala:76) at org.apache.toree.Main$.main(Main.scala:24) at org.apache.toree.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException: wewcw0hd3mn02.wedip.com at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:406) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:728) at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:671) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:155) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2815) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:98) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2852) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2834) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:387) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:186) at org.apache.spark.deploy.yarn.Client$$anonfun$7.apply(Client.scala:123) at org.apache.spark.deploy.yarn.Client$$anonfun$7.apply(Client.scala:123) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.deploy.yarn.Client.<init>(Client.scala:123) at org.apache.spark.deploy.yarn.Client.<init>(Client.scala:69) at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:55) at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:171) at org.apache.spark.SparkContext.<init>(SparkContext.scala:509) at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2313) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:868) at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:860) at scala.Option.getOrElse(Option.scala:121) at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:860) at org.apache.toree.kernel.api.Kernel.createSparkContext(Kernel.scala:344) at org.apache.toree.kernel.api.Kernel.createSparkContext(Kernel.scala:363) at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeSparkContext(ComponentInitialization.scala:103) at org.apache.toree.Main$$anon$1.initializeSparkContext(Main.scala:35) at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86) at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35) at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:100) at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:40) at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24) at scala.Function0$class.apply$mcV$sp(Function0.scala:34) at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.App$$anonfun$main$1.apply(App.scala:76) at scala.collection.immutable.List.foreach(List.scala:381) at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35) at scala.App$class.main(App.scala:76) at org.apache.toree.Main$.main(Main.scala:24) at org.apache.toree.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.net.UnknownHostException: wewcw0hd3mn02.wedip.com ... 52 more   wewcw0hd3mn02.wedip.com server is pinging from cdsw   [root@WEWCW0HDDSW01 conf]# ping wewcw0hd3mn02.wedip.com PING wuwcw0hd3mn02.wedip.com (10.10.160.12) 56(84) bytes of data. 64 bytes from wewcw0hd3mn02.wedip.com (10.10.160.12): icmp_seq=1 ttl=64 time=1.19 ms 64 bytes from wewcw0hd3mn02.wedip.com (10.10.160.12): icmp_seq=2 ttl=64 time=0.234 ms 64 bytes from wewcw0hd3mn02.wedip.com (10.10.160.12): icmp_seq=3 ttl=64 time=0.174 ms ^C --- wewcw0hd3mn02.wedip.com ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2001ms rtt min/avg/max/mdev = 0.174/0.532/1.190/0.466 ms also spark2 shell is working fine from cdsw server [25761081@WEWCW0HDDSW01 ~]$ spark2-shell Setting default log level to "WARN". To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 17/07/18 12:17:58 WARN spark.SparkContext: Support for Java 7 is deprecated as of Spark 2.0.0 Spark context Web UI available at http://10.10.10.10:4040 Spark context available as 'sc' (master = yarn, app id = application_1500175454875_2461). Spark session available as 'spark'. Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 2.1.0.cloudera1 /_/ Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_67) Type in expressions to have them evaluated. Type :help for more information. ... View more