@Wendell Bu thanks for your article. At the end you mentioned that you will discuss the detailed configuration in other articles. Can you please share the other articles. ... View more

@rtrivedi Thank for your reply, I figure it out. There is a partition issue. I have dropped and add the partition, now it is working fine. Thanks for the help. ... View more

@Ajay Thanks for your reply. Actually I am installing opentsdb on one of the cluster. This cluster is secured cluster. The hbase configuration in that cluster are secure. Please find the property which is present in hbase-site.xml . I have seen a article in Goolge forum, in that they said that it is bug and they come up with a solution. https://groups.google.com/forum/#!searchin/opentsdb/java.lang.IndexOutOfBoundsException|sort:relevance/opentsdb/sDWQmsNPakM/JJmm9YaQBgAJ [sg865w@blpd214 ~]$ cd /opt/app/opentsdb/opentsdb-2.3.0/build [sg865w@blpd214 build]$ sudo -su zkeeper [zkeeper@blpd214 build]$ ./tsdb tsd --config=/opt/app/opentsdb/opentsdb-2.3.0/src/opentsdb.conf --zkbasedir=/hbase-secure 2017-02-15 15:15:04,189 INFO [main] TSDMain: Starting 2017-02-15 15:15:04,192 INFO [main] TSDMain: net.opentsdb.tools 2.3.0 built at revision cac608a (MINT) 2017-02-15 15:15:04,192 INFO [main] TSDMain: Built on 2016/12/29 13:57:15 +0000 by root@centos.localhost:/home/hobbes/opentsdb_OFFICIAL/build 2017-02-15 15:15:04,196 INFO [main] Config: Successfully loaded configuration file: /opt/app/opentsdb/opentsdb-2.3.0/src/opentsdb.conf 2017-02-15 15:15:04,250 INFO [main] Config: Successfully loaded configuration file: /opt/app/opentsdb/opentsdb-2.3.0/src/opentsdb.conf 2017-02-15 15:15:04,344 WARN [main] PluginLoader: Unable to locate any plugins of the type: net.opentsdb.query.filter.TagVFilter 2017-02-15 15:15:04,360 INFO [main] ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2017-02-15 15:15:04,360 INFO [main] ZooKeeper: Client environment:host.name=blpd214.bhdc.att.com 2017-02-15 15:15:04,360 INFO [main] ZooKeeper: Client environment:java.version=1.7.0_111 2017-02-15 15:15:04,360 INFO [main] ZooKeeper: Client environment:java.vendor=Oracle Corporation 2017-02-15 15:15:04,360 INFO [main] ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/jre 2017-02-15 15:15:04,360 INFO [main] ZooKeeper: Client environment:java.class.path=/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jexl/commons-logging-1.1.1.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/guava/guava-18.0.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/slf4j/log4j-over-slf4j-1.7.7.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/logback/logback-classic-1.0.13.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/logback/logback-core-1.0.13.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jackson/jackson-annotations-2.4.3.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jackson/jackson-core-2.4.3.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jackson/jackson-databind-2.4.3.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/javacc/javacc-6.1.2.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jexl/commons-jexl-2.1.1.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jgrapht/jgrapht-core-0.9.1.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/netty/netty-3.9.4.Final.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/slf4j/slf4j-api-1.7.7.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/suasync/async-1.4.0.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/apache/commons-math3-3.4.1.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/hbase/asynchbase-1.7.2.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/protobuf/protobuf-java-2.5.0.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/zookeeper/zookeeper-3.4.6.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/tsdb-2.3.0.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/../src 2017-02-15 15:15:04,360 INFO [main] ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2017-02-15 15:15:04,361 INFO [main] ZooKeeper: Client environment:java.io.tmpdir=/tmp 2017-02-15 15:15:04,361 INFO [main] ZooKeeper: Client environment:java.compiler=<NA> 2017-02-15 15:15:04,361 INFO [main] ZooKeeper: Client environment:os.name=Linux 2017-02-15 15:15:04,361 INFO [main] ZooKeeper: Client environment:os.arch=amd64 2017-02-15 15:15:04,361 INFO [main] ZooKeeper: Client environment:os.version=2.6.32-642.6.2.el6.x86_64 2017-02-15 15:15:04,361 INFO [main] ZooKeeper: Client environment:user.name=zkeeper 2017-02-15 15:15:04,361 INFO [main] ZooKeeper: Client environment:user.home=/home/zkeeper 2017-02-15 15:15:04,361 INFO [main] ZooKeeper: Client environment:user.dir=/opt/app/opentsdb/opentsdb-2.3.0/build 2017-02-15 15:15:04,361 INFO [main] ZooKeeper: Initiating client connection, connectString=blpd214.bhdc.att.com:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@53cd7dc7 2017-02-15 15:15:04,372 INFO [main] HBaseClient: Need to find the -ROOT- region 2017-02-15 15:15:04,394 INFO [main-SendThread(blpd214.bhdc.att.com:2181)] Login: successfully logged in. 2017-02-15 15:15:04,395 INFO [Thread-1] Login: TGT refresh thread started. 2017-02-15 15:15:04,399 INFO [main-SendThread(blpd214.bhdc.att.com:2181)] ZooKeeperSaslClient: Client will use GSSAPI as SASL mechanism. 2017-02-15 15:15:04,405 INFO [Thread-1] Login: TGT valid starting at: Wed Feb 15 13:24:23 CST 2017-02-15 15:15:04,405 INFO [Thread-1] Login: TGT expires: Thu Feb 16 13:24:23 CST 2017 2017-02-15 15:15:04,406 INFO [Thread-1] Login: TGT refresh sleeping until: Thu Feb 16 08:55:02 CST 2017 2017-02-15 2017-02-15 15:15:04,635 ERROR [AsyncHBase I/O Worker #1] RegionClient: Unexpected exception from downstream on [id: 0xa673b30d, /130.5.106.4:52229 => /130.5.106.11:16020] java.lang.IndexOutOfBoundsException: Not enough readable bytes - Need 132, maximum is 120 at org.jboss.netty.buffer.AbstractChannelBuffer.checkReadableBytes(AbstractChannelBuffer.java:668) ~[netty-3.9.4.Final.jar:na] at org.jboss.netty.buffer.AbstractChannelBuffer.readBytes(AbstractChannelBuffer.java:338) ~[netty- I can't submit the entore error. The remaining error is show in the above comment. Please help me on this Thanks & Regards Shyam Gurram ... View more

@Josh Elser Still I am getting the same error. :-(. In github also they said that it is not working. [zkeeper@blpd214 opentsdb-2.3.0]$ ./build/tsdb tsd --port=4242 --staticroot=build/staticroot --cachedir=/tmp/opentsdb --zkquorum=blpd213.bhdc.att.com:2181,blpd214.bhdc.att.com:2181,blpd215.bhdc.att.com:2181 --zkbasedir=/hbase-unsecure 2017-02-14 12:43:21,624 INFO [main] TSDMain: Starting. 2017-02-14 12:43:21,627 INFO [main] TSDMain: net.opentsdb.tools 2.3.0 built at revision cac608a (MINT) 2017-02-14 12:43:21,627 INFO [main] TSDMain: Built on 2016/12/29 13:57:15 +0000 by root@centos.localhost:/home/hobbes/opentsdb_OFFICIAL/build 2017-02-14 12:43:21,631 INFO [main] Config: No configuration found, will use defaults 2017-02-14 12:43:21,774 WARN [main] PluginLoader: Unable to locate any plugins of the type: net.opentsdb.query.filter.TagVFilter 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:host.name=blpd214.bhdc.att.com 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:java.version=1.7.0_111 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:java.vendor=Oracle Corporation 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:java.home=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.111.x86_64/jre 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:java.class.path=/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jexl/commons-logging-1.1.1.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/guava/guava-18.0.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/slf4j/log4j-over-slf4j-1.7.7.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/logback/logback-classic-1.0.13.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/logback/logback-core-1.0.13.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jackson/jackson-annotations-2.4.3.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jackson/jackson-core-2.4.3.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jackson/jackson-databind-2.4.3.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/javacc/javacc-6.1.2.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jexl/commons-jexl-2.1.1.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/jgrapht/jgrapht-core-0.9.1.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/netty/netty-3.9.4.Final.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/slf4j/slf4j-api-1.7.7.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/suasync/async-1.4.0.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/apache/commons-math3-3.4.1.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/hbase/asynchbase-1.7.2.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/protobuf/protobuf-java-2.5.0.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/third_party/zookeeper/zookeeper-3.4.6.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/tsdb-2.3.0.jar:/opt/app/opentsdb/opentsdb-2.3.0/build/../src 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:java.io.tmpdir=/tmp 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:java.compiler=<NA> 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:os.name=Linux 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:os.arch=amd64 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:os.version=2.6.32-642.6.2.el6.x86_64 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:user.name=zkeeper 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:user.home=/home/zkeeper 2017-02-14 12:43:21,790 INFO [main] ZooKeeper: Client environment:user.dir=/opt/app/opentsdb/opentsdb-2.3.0 2017-02-14 12:43:21,791 INFO [main] ZooKeeper: Initiating client connection, connectString=blpd213.bhdc.att.com:2181,blpd214.bhdc.att.com:2181,blpd215.bhdc.att.com:2181 sessionTimeout=5000 watcher=org.hbase.async.HBaseClient$ZKClient@5987b427 2017-02-14 12:43:21,825 INFO [main] HBaseClient: Need to find the -ROOT- region 2017-02-14 12:43:21,839 INFO [main-SendThread(blpd213.bhdc.att.com:2181)] ClientCnxn: Opening socket connection to server blpd213.bhdc.att.com/130.5.106.3:2181. Will not attempt to authenticate using SASL (unknown error) 2017-02-14 12:43:21,843 INFO [main-SendThread(blpd213.bhdc.att.com:2181)] ClientCnxn: Socket connection established to blpd213.bhdc.att.com/130.5.106.3:2181, initiating session 2017-02-14 12:43:21,849 INFO [main-SendThread(blpd213.bhdc.att.com:2181)] ClientCnxn: Session establishment complete on server blpd213.bhdc.att.com/130.5.106.3:2181, sessionid = 0x15a396fea2300aa, negotiated timeout = 5000 2017-02-14 12:43:21,856 ERROR [main-EventThread] HBaseClient: The znode for the -ROOT- region doesn't exist! 2017-02-14 12:43:22,878 ERROR [main-EventThread] HBaseClient: The znode for the -ROOT- region doesn't exist! 2017-02-14 12:43:23,897 ERROR [main-EventThread] HBaseClient: The znode for the -ROOT- region doesn't exist! ... View more

@lraheja and team, Still I am unable to see the job on Spark History Server. I followed the below steps 1) spark-shell 2) scala> val sqlContext = new org.apache.spark.sql.hive.HiveContext(sc) 3) scala> sqlContext.sql("CREATE TABLE IF NOT EXISTS employee(id INT, name STRING, age INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'") I have a employee.txt in my directory (/home/sg865w/employee.txt) 4) scala> sqlContext.sql("LOAD DATA LOCAL INPATH 'employee.txt' INTO TABLE employee") 5) scala> val result = sqlContext.sql("FROM employee SELECT id, name, age") 6) scala> result.show() 7) :paste val df = spark .read .option("url","jdbc:hive2://localhost:10000") .option("dbtable","people") .format("jdbc") .load 😎 Ctrl +D The out message I got it df = spark undefined. Another Error I got is: 1)I am trying to start the spark-thrift-server it is throwing an error message. Error Message: 17/01/13 00:49:45 INFO metastore: Connected to metastore. 17/01/13 00:49:45 INFO SessionState: Created local directory: /tmp/cc5a2235-5a42-4e82-acd8-0b9c9677c3f1_resources 17/01/13 00:49:45 INFO SessionState: Created HDFS directory: /tmp/hive/sg865w/cc5a2235-5a42-4e82-acd8-0b9c9677c3f1 17/01/13 00:49:45 INFO SessionState: Created local directory: /tmp/sg865w/cc5a2235-5a42-4e82-acd8-0b9c9677c3f1 17/01/13 00:49:45 INFO SessionState: Created HDFS directory: /tmp/hive/sg865w/cc5a2235-5a42-4e82-acd8-0b9c9677c3f1/_tmp_space.db SET spark.sql.hive.version=1.2.1 17/01/13 00:49:46 ERROR HiveThriftServer2: Error starting HiveThriftServer2 org.apache.hive.service.ServiceException: Unable to login to kerberos with given principal/keytab at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIService.init(SparkSQLCLIService.scala:58) at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$anonfun$initCompositeService$1.apply(SparkSQLCLIService.scala:79) at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$anonfun$initCompositeService$1.apply(SparkSQLCLIService.scala:79) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at scala.collection.AbstractIterable.foreach(Iterable.scala:54) at org.apache.spark.sql.hive.thriftserver.ReflectedCompositeService$class.initCompositeService(SparkSQLCLIService.scala:79) at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.initCompositeService(HiveThriftServer2.scala:263) at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.init(HiveThriftServer2.scala:283) at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$.main(HiveThriftServer2.scala:85) at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2.main(HiveThriftServer2.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:738) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.io.IOException: Login failure for hive/blpd217.bhdc.att.com@BRHMLAB01.LAB.ATT.COM from keytab /etc/security/keytabs/hive.service.keytab: javax.security.auth.login.LoginException: Unable to obtain password from user at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:987) at org.apache.hive.service.auth.HiveAuthFactory.loginFromKeytab(HiveAuthFactory.java:198) at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIService.init(SparkSQLCLIService.scala:53) ... 20 more Caused by: javax.security.auth.login.LoginException: Unable to obtain password from user at com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:856) at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:719) at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:584) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at javax.security.auth.login.LoginContext.invoke(LoginContext.java:784) at javax.security.auth.login.LoginContext.access$000(LoginContext.java:203) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:721) at javax.security.auth.login.LoginContext$5.run(LoginContext.java:719) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.login.LoginContext.invokeCreatorPriv(LoginContext.java:718) at javax.security.auth.login.LoginContext.login(LoginContext.java:590) at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:978) ... 22 more 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static/sql,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/execution,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/SQL,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} 17/01/13 00:49:46 INFO ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} 17/01/13 00:49:46 INFO SparkUI: Stopped Spark web UI at http://130.5.106.7:4042 17/01/13 00:49:46 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 17/01/13 00:49:46 INFO MemoryStore: MemoryStore cleared 17/01/13 00:49:46 INFO BlockManager: BlockManager stopped 17/01/13 00:49:46 INFO BlockManagerMaster: BlockManagerMaster stopped 17/01/13 00:49:46 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 17/01/13 00:49:46 INFO SparkContext: Successfully stopped SparkContext 17/01/13 00:49:46 ERROR Utils: Uncaught exception in thread pool-7-thread-1 java.lang.NullPointerException at org.apache.spark.sql.hive.thriftserver.HiveThriftServer2$anonfun$main$1.apply$mcV$sp(HiveThriftServer2.scala:80) at org.apache.spark.util.SparkShutdownHook.run(ShutdownHookManager.scala:267) at org.apache.spark.util.SparkShutdownHookManager$anonfun$runAll$1$anonfun$apply$mcV$sp$1.apply$mcV$sp(ShutdownHookManager.scala:239) at org.apache.spark.util.SparkShutdownHookManager$anonfun$runAll$1$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:239) at org.apache.spark.util.SparkShutdownHookManager$anonfun$runAll$1$anonfun$apply$mcV$sp$1.apply(ShutdownHookManager.scala:239) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1801) at org.apache.spark.util.SparkShutdownHookManager$anonfun$runAll$1.apply$mcV$sp(ShutdownHookManager.scala:239) at org.apache.spark.util.SparkShutdownHookManager$anonfun$runAll$1.apply(ShutdownHookManager.scala:239) at org.apache.spark.util.SparkShutdownHookManager$anonfun$runAll$1.apply(ShutdownHookManager.scala:239) at scala.util.Try$.apply(Try.scala:161) at org.apache.spark.util.SparkShutdownHookManager.runAll(ShutdownHookManager.scala:239) at org.apache.spark.util.SparkShutdownHookManager$anon$2.run(ShutdownHookManager.scala:218) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 17/01/13 00:49:46 INFO ShutdownHookManager: Shutdown hook called 17/01/13 00:49:46 INFO ShutdownHookManager: Deleting directory /opt/data/data01/paxata/spark-8fdd13a5-0db8-42e7-b172-9b910d82df55 17/01/13 00:49:46 INFO ShutdownHookManager: Deleting directory /tmp/spark-79a77df7-0ab3-4be6-bea4-472bd8597cab 17/01/13 00:49:46 INFO RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. 17/01/13 00:49:46 INFO RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 17/01/13 00:49:46 INFO RemoteActorRefProvider$RemotingTerminator: Remoting shut down. Awaiting for your reply. Please let me know in case of any concerns. ... View more

@Pradeep Bhadani Thanks for the reply, yes this is an auto- process. Every minute there are some log files which are landing on HDFS. We need to move all these small log text file into one text file. I am using -getmerge command, but will work for local file system to HDFS. But, my log files are landing on HDFS. Help me on this. ... View more