I am trying to setup 3 kafka brokers in multi node setup for creating imply.io dashboard for my projects. Initially I created 2 brokers in multi node broker setup. But I am getting the below error. Even though I had 2 brokers. [2019-04-25 20:24:29,577] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 2 larger than available brokers: 1 Below mentioned the kafka brokers setup. Broker-1: broker.id=0 listeners = PLAINTEXT://01.bm-imply-server.dev.lax1:9092 advertised.listeners=PLAINTEXT://01.bm-imply-server.dev.lax1:9092 log.dirs=/tmp/kafka-logs-9092 Broker-2: broker.id=1 listeners = PLAINTEXT://01.bm-imply-server.dev.lax1:9093 advertised.listeners=PLAINTEXT://01.bm-imply-server.dev.lax1:9093 log.dirs=/tmp/kafka-logs-9093 ... View more

I am trying to create a hive table in parquet format with snappy compression. Instead of sqlContext I am using HiveContext to directly save my dataframe results into a table using saveAsTable("<table name>"). I set the format using "hc.setConf('spark.sql.parquet.compression.codec','snappy')" But the hive table is always created as parquet with gz compression instead of parquet with snappy compression codec. Is there any solution for this? ... View more

How to find the no of records in the json file. It is a production data and it is having records more than billion records. Is there any util or command to find it quickly? ... View more

I am trying to start kafka server. But I am getting the below error. Any idea about fix? [2019-01-23 16:36:17,954] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer) org.apache.kafka.common.KafkaException: Socket server failed to bind to 0.0.0.0:9092: Address already in use. at kafka.network.Acceptor.openServerSocket(SocketServer.scala:442) at kafka.network.Acceptor.<init>(SocketServer.scala:332) at kafka.network.SocketServer$$anonfun$createAcceptorAndProcessors$1.apply(SocketServer.scala:149) at kafka.network.SocketServer$$anonfun$createAcceptorAndProcessors$1.apply(SocketServer.scala:145) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at kafka.network.SocketServer.createAcceptorAndProcessors(SocketServer.scala:145) at kafka.network.SocketServer.startup(SocketServer.scala:94) at kafka.server.KafkaServer.startup(KafkaServer.scala:250) at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38) at kafka.Kafka$.main(Kafka.scala:75) at kafka.Kafka.main(Kafka.scala) Caused by: java.net.BindException: Address already in use at sun.nio.ch.Net.bind0(Native Method) at sun.nio.ch.Net.bind(Net.java:433) at sun.nio.ch.Net.bind(Net.java:425) at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67) at kafka.network.Acceptor.openServerSocket(SocketServer.scala:438) ... View more

I am trying to process the data after sqoop import. But I am getting the below error while loading the data into RDD. at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) Below I mentioned my SQOOP import sqoop import \ -- connect "jdbc:mysql://01-mysql-test232855.envnxs.net:3306/retail_db" \ -- username autoenv_root -- password UkdWMmIzQnpVbTlqYTNvPQ = \ -- table lnld_products_23425 \ -- as -textfile \ -- target -dir / user / mpalanisamy / lnld_products_23425 \ -- fields -terminated - by '|'; Below mentioned the Spark commands for processing it. case class products_rec( product_id: Int,product_category_id: Int,product_name: String,product_desc: String,product_price: Double,product_image: String,... ... ) val products = sc.textFile("/user/mpalanisamy/problem2/lnld_products_23425") val productsRDD=products.map(rec=> { var r = rec.split("|") (r(0).toInt, r(1).toInt, r(2), r(3), r(4).toFloat, r(5)) }) After giving the below I am getting the above error. productsRDD.take(10).foreach(println) It looks like I am not doing anything wrong. But still I am getting the below error. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in stage 5.0 (TID 17, 01.hadoop-datanode.test232855.nym2): java.lang.NumberFormatException: For input string: "|" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:580) at java.lang.Integer.parseInt(Integer.java:615) at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229) at scala.collection.immutable.StringOps.toInt(StringOps.scala:31) at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$anonfun$1.apply(<console>:28) at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$anonfun$1.apply(<console>:26) at scala.collection.Iterator$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$anon$10.next(Iterator.scala:312) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) at scala.collection.AbstractIterator.to(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) at org.apache.spark.rdd.RDD$anonfun$take$1$anonfun$28.apply(RDD.scala:1302) at org.apache.spark.rdd.RDD$anonfun$take$1$anonfun$28.apply(RDD.scala:1302) at org.apache.spark.SparkContext$anonfun$runJob$5.apply(SparkContext.scala:1850) at org.apache.spark.SparkContext$anonfun$runJob$5.apply(SparkContext.scala:1850) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Driver stacktrace: at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$failJobAndIndependentStages(DAGScheduler.scala:1283) at org.apache.spark.scheduler.DAGScheduler$anonfun$abortStage$1.apply(DAGScheduler.scala:1271) at org.apache.spark.scheduler.DAGScheduler$anonfun$abortStage$1.apply(DAGScheduler.scala:1270) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270) at org.apache.spark.scheduler.DAGScheduler$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) at org.apache.spark.scheduler.DAGScheduler$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697) at scala.Option.foreach(Option.scala:236) at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458) at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447) at org.apache.spark.util.EventLoop$anon$1.run(EventLoop.scala:48) at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837) at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850) at org.apache.spark.rdd.RDD$anonfun$take$1.apply(RDD.scala:1302) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108) at org.apache.spark.rdd.RDD.withScope(RDD.scala:310) at org.apache.spark.rdd.RDD.take(RDD.scala:1276) at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:29) at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:34) at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:36) at $iwC$iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:38) at $iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:40) at $iwC$iwC$iwC$iwC$iwC.<init>(<console>:42) at $iwC$iwC$iwC$iwC.<init>(<console>:44) at $iwC$iwC$iwC.<init>(<console>:46) at $iwC$iwC.<init>(<console>:48) at $iwC.<init>(<console>:50) at <init>(<console>:52) at .<init>(<console>:56) at .<clinit>(<console>) at .<init>(<console>:7) at .<clinit>(<console>) at $print(<console>) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340) at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657) at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665) at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$loop(SparkILoop.scala:670) at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply$mcZ$sp(SparkILoop.scala:997) at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945) at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945) at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$process(SparkILoop.scala:945) at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) at org.apache.spark.repl.Main$.main(Main.scala:31) at org.apache.spark.repl.Main.main(Main.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:683) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:189) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:214) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) Caused by: java.lang.NumberFormatException: For input string: "|" at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) at java.lang.Integer.parseInt(Integer.java:580) at java.lang.Integer.parseInt(Integer.java:615) at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229) at scala.collection.immutable.StringOps.toInt(StringOps.scala:31) at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$anonfun$1.apply(<console>:28) at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$anonfun$1.apply(<console>:26) at scala.collection.Iterator$anon$11.next(Iterator.scala:328) at scala.collection.Iterator$anon$10.next(Iterator.scala:312) at scala.collection.Iterator$class.foreach(Iterator.scala:727) at scala.collection.AbstractIterator.foreach(Iterator.scala:1157) at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273) at scala.collection.AbstractIterator.to(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157) at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252) at scala.collection.AbstractIterator.toArray(Iterator.scala:1157) at org.apache.spark.rdd.RDD$anonfun$take$1$anonfun$28.apply(RDD.scala:1302) at org.apache.spark.rdd.RDD$anonfun$take$1$anonfun$28.apply(RDD.scala:1302) at org.apache.spark.SparkContext$anonfun$runJob$5.apply(SparkContext.scala:1850) at org.apache.spark.SparkContext$anonfun$runJob$5.apply(SparkContext.scala:1850) at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66) at org.apache.spark.scheduler.Task.run(Task.scala:88) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) ... View more

I am getting the below error while doing SQOOP EXPORT. java.io.IOException: Minimum split size pernode 536870912 cannot be larger than maximum split size 41at org.apache.sqoop.mapreduce.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:200) at org.apache.sqoop.mapreduce.ExportInputFormat.getSplits(ExportInputFormat.java:73) at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:597) at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1306) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1303) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1303) at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1324) at org.apache.sqoop.mapreduce.ExportJobBase.doSubmitJob(ExportJobBase.java:324) at org.apache.sqoop.mapreduce.ExportJobBase.runJob(ExportJobBase.java:301) at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:442) at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931) at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80) at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) I removed the most of the fields and renamed the table to some other name due to project compliance. Below I mentioned the Sqoop Export I used create table sup_api_bidder_test( id int, name string, vendor_id bigint) ... View more

I am getting the below error while SQOOP HIVE import. It is working fine for another table. But the same set of instructions are not working for this tables. Even the same code worked fine in the production box. Below I mentioned the error messages. My SQOOP command: sqoop import --connect jdbc:mysql://01-mysql-test232855.envnxs.net:3306/retail_db --username autoenv_root -P --query "select * from agg_dw_orders_lnin_customers where \$CONDITIONS" --hive-import--hive-table agg_dw_orders_lnin_customers --hive-database hivesqoop --target-dir /user/hive/warehouse/orders_warehouse.db/agg_dw_orders_lnin_customers --split-by order_id 18/11/14 23:32:08 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 Enter password: 18/11/14 23:32:25 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 18/11/14 23:32:25 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc. 18/11/14 23:32:25 WARN tool.BaseSqoopTool: It seems that you're doing hive import directly into default 18/11/14 23:32:25 WARN tool.BaseSqoopTool: hive warehouse directory which is not supported. Sqoop is 18/11/14 23:32:25 WARN tool.BaseSqoopTool: firstly importing data into separate directory and then 18/11/14 23:32:25 WARN tool.BaseSqoopTool: inserting data into hive. Please consider removing 18/11/14 23:32:25 WARN tool.BaseSqoopTool: --target-dir or --warehouse-dir into /user/hive/warehouse in 18/11/14 23:32:25 WARN tool.BaseSqoopTool: case that you will detect any issues. 18/11/14 23:32:26 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 18/11/14 23:32:26 INFO tool.CodeGenTool: Beginning code generation 18/11/14 23:32:26 INFO manager.SqlManager: Executing SQL statement: select * from orders where (1 = 0) 18/11/14 23:32:26 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@564718df is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries. java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@564718df is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries. at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931) at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2747) at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1911) at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2618) at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2568) at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1557) at com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3002) at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602) at com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445) at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:305) at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260) at org.apache.sqoop.manager.SqlManager.getColumnTypesForQuery(SqlManager.java:253) at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:336) at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1872) at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1671) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) 18/11/14 23:32:26 ERROR tool.ImportTool: Import failed: java.io.IOException: No columns to generate for ClassWriter at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1677) at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106) at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501) at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628) at org.apache.sqoop.Sqoop.run(Sqoop.java:147) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234) at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243) at org.apache.sqoop.Sqoop.main(Sqoop.java:252) ... View more

I am trying to create a hive table with ";" as delimiter for my project. Because the input file that I am getting from another application having data delimited by ";" . When I try to create the table by giving row format delimited fields terminated by ',' I am getting the below error. MismatchedTokenException(-1!=302) at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617) at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115) at org.apache.hadoop.hive.ql.parse.HiveParser.tableRowFormatFieldIdentifier(HiveParser.java:34966) at org.apache.hadoop.hive.ql.parse.HiveParser.rowFormatDelimited(HiveParser.java:33919) at org.apache.hadoop.hive.ql.parse.HiveParser.tableRowFormat(HiveParser.java:34195) at org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:4979) at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:2355) at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1579) at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1057) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199) at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307) at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1110) at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1158) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1047) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1037) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370) at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:756) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.apache.hadoop.util.RunJar.run(RunJar.java:221) at org.apache.hadoop.util.RunJar.main(RunJar.java:136) FAILED: ParseException line 6:62 mismatched input '<EOF>' expecting StringLiteral near 'by' in table row format's field separator ... View more

In my project I need to use Kafka ad Spark streaming for doing real time analytics. So I am trying to import Kafka and Spark streaming dependencies. In order to read the Kakfa topics in my spark streaming program, I am trying to import kafka.serializer.StringDecoder. But I am not able to import it.screen-shot-2018-11-05-at-51111-pm.png Even I wrote a dummy program to fix this issue. But still I couldn't find a solution. ... View more

How to create jar without dependencies in intellij. I mean only jar with our main class instead of jar with all the dependencies are added to the main class jar. This way I am looking for light weight jar ... View more

Hi Guys, In my project I need to validate the Kafka messages. I mean I want to validate the qualify of the data from kafka message. ex: I need to get the log messages to the source of my analytics application. I need to know whether the message I received as it is from kafka. Is there any tool (or) way to validate it? ... View more

Hi Guys, I am trying to use Spring STS for as IDE for my mapreduce programs. But when ever I try to launch STS.exe, I am getting main class of "org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar" not found error. But I can see the the missing jar file in the path. Is anybody faced the same problem in the past? Do you have any idea what went wrong. Even I restarted the system and getting the same error. I attached the screen shot of the issues. Regards, P.Mahendiran ... View more

Hi Guys, I am trying to typesafe config in my scala project. I am using Intellij to run this program. But I am getting "Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found" error. I also attached the screen shot of the error. Regards, P.Mahendiran ... View more

Hi Guys, I am trying to use the avro-tools to get the avro schema from avro file. But I am getting the message like saying "file doesn't exist". If I issue "Hadoop fs -ls" command I can see the file. Even I can see the avro file content by giving the path. But if I issue the "avro-tools getschema <avro file path> I am getting file doesn't exist error. Anybody got the same type of error earlier. I also attached the screen shot of the error. Regards, P.Mahendiran ... View more