Member since
12-29-2017
33
Posts
2
Kudos Received
1
Solution
My Accepted Solutions
Title | Views | Posted |
---|---|---|
398 | 11-14-2018 01:40 AM |
04-25-2019
08:58 PM
I am trying to setup 3 kafka brokers in multi node setup for creating imply.io dashboard for my projects. Initially I created 2 brokers in multi node broker setup. But I am getting the below error. Even though I had 2 brokers. [2019-04-25 20:24:29,577] ERROR org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 2 larger than available brokers: 1 Below mentioned the kafka brokers setup. Broker-1: broker.id=0 listeners = PLAINTEXT://01.bm-imply-server.dev.lax1:9092 advertised.listeners=PLAINTEXT://01.bm-imply-server.dev.lax1:9092 log.dirs=/tmp/kafka-logs-9092 Broker-2: broker.id=1 listeners = PLAINTEXT://01.bm-imply-server.dev.lax1:9093 advertised.listeners=PLAINTEXT://01.bm-imply-server.dev.lax1:9093 log.dirs=/tmp/kafka-logs-9093
... View more
Labels:
04-25-2019
08:27 PM
Thanks, Shu.. It is working now
... View more
04-18-2019
10:28 PM
I am trying to create a hive table in parquet format with snappy compression. Instead of sqlContext I am using HiveContext to directly save my dataframe results into a table using saveAsTable("<table name>"). I set the format using "hc.setConf('spark.sql.parquet.compression.codec','snappy')" But the hive table is always created as parquet with gz compression instead of parquet with snappy compression codec. Is there any solution for this?
... View more
Labels:
01-23-2019
07:34 PM
I just opened it in vim editor. At the bottom it is showing the no of records in the json. This count is not a blind record count. It counts the record based on the json structure.
... View more
01-23-2019
07:30 PM
How to find the no of records in the json file. It is a production data and it is having records more than billion records. Is there any util or command to find it quickly?
... View more
01-23-2019
04:42 PM
1 Kudo
I am trying to start kafka server. But I am getting the below error. Any idea about fix? [2019-01-23 16:36:17,954] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: Socket server failed to bind to 0.0.0.0:9092: Address already in use.
at kafka.network.Acceptor.openServerSocket(SocketServer.scala:442)
at kafka.network.Acceptor.<init>(SocketServer.scala:332)
at kafka.network.SocketServer$$anonfun$createAcceptorAndProcessors$1.apply(SocketServer.scala:149)
at kafka.network.SocketServer$$anonfun$createAcceptorAndProcessors$1.apply(SocketServer.scala:145)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at kafka.network.SocketServer.createAcceptorAndProcessors(SocketServer.scala:145)
at kafka.network.SocketServer.startup(SocketServer.scala:94)
at kafka.server.KafkaServer.startup(KafkaServer.scala:250)
at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:38)
at kafka.Kafka$.main(Kafka.scala:75)
at kafka.Kafka.main(Kafka.scala)
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:67)
at kafka.network.Acceptor.openServerSocket(SocketServer.scala:438)
... View more
Labels:
12-04-2018
08:13 PM
This issue got fixed by doing using single quote like below. val productsRDD=products.map(rec=>{ var r = rec.split('|') (r(0).toInt, r(1).toInt, r(2), r(3), r(4).toFloat, r(5)) })
... View more
11-27-2018
08:29 PM
I am trying to process the data after sqoop import. But I am getting the below error while loading the data into RDD. at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65) Below I mentioned my SQOOP import sqoop import \ -- connect "jdbc:mysql://01-mysql-test232855.envnxs.net:3306/retail_db" \ -- username autoenv_root -- password UkdWMmIzQnpVbTlqYTNvPQ = \ -- table lnld_products_23425 \ -- as -textfile \ -- target -dir / user / mpalanisamy / lnld_products_23425 \ -- fields -terminated - by '|'; Below mentioned the Spark commands for processing it. case class products_rec( product_id: Int,product_category_id: Int,product_name: String,product_desc: String,product_price: Double,product_image: String,... ... ) val products = sc.textFile("/user/mpalanisamy/problem2/lnld_products_23425") val productsRDD=products.map(rec=> { var r = rec.split("|") (r(0).toInt, r(1).toInt, r(2), r(3), r(4).toFloat, r(5)) }) After giving the below I am getting the above error. productsRDD.take(10).foreach(println) It looks like I am not doing anything wrong. But still I am getting the below error. org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 5.0 failed 4 times, most recent failure: Lost task 0.3 in stage 5.0 (TID 17, 01.hadoop-datanode.test232855.nym2): java.lang.NumberFormatException: For input string: "|"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$anonfun$1.apply(<console>:28)
at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$anonfun$1.apply(<console>:26)
at scala.collection.Iterator$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$anon$10.next(Iterator.scala:312)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$anonfun$take$1$anonfun$28.apply(RDD.scala:1302)
at org.apache.spark.rdd.RDD$anonfun$take$1$anonfun$28.apply(RDD.scala:1302)
at org.apache.spark.SparkContext$anonfun$runJob$5.apply(SparkContext.scala:1850)
at org.apache.spark.SparkContext$anonfun$runJob$5.apply(SparkContext.scala:1850)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$failJobAndIndependentStages(DAGScheduler.scala:1283)
at org.apache.spark.scheduler.DAGScheduler$anonfun$abortStage$1.apply(DAGScheduler.scala:1271)
at org.apache.spark.scheduler.DAGScheduler$anonfun$abortStage$1.apply(DAGScheduler.scala:1270)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1270)
at org.apache.spark.scheduler.DAGScheduler$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGScheduler$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:697)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:697)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1496)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1458)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1447)
at org.apache.spark.util.EventLoop$anon$1.run(EventLoop.scala:48)
at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:567)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1824)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1837)
at org.apache.spark.SparkContext.runJob(SparkContext.scala:1850)
at org.apache.spark.rdd.RDD$anonfun$take$1.apply(RDD.scala:1302)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:310)
at org.apache.spark.rdd.RDD.take(RDD.scala:1276)
at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:29)
at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:34)
at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:36)
at $iwC$iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:38)
at $iwC$iwC$iwC$iwC$iwC$iwC.<init>(<console>:40)
at $iwC$iwC$iwC$iwC$iwC.<init>(<console>:42)
at $iwC$iwC$iwC$iwC.<init>(<console>:44)
at $iwC$iwC$iwC.<init>(<console>:46)
at $iwC$iwC.<init>(<console>:48)
at $iwC.<init>(<console>:50)
at <init>(<console>:52)
at .<init>(<console>:56)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065)
at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1340)
at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)
at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819)
at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857)
at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902)
at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814)
at org.apache.spark.repl.SparkILoop.processLine$1(SparkILoop.scala:657)
at org.apache.spark.repl.SparkILoop.innerLoop$1(SparkILoop.scala:665)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$loop(SparkILoop.scala:670)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply$mcZ$sp(SparkILoop.scala:997)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop$anonfun$org$apache$spark$repl$SparkILoop$process$1.apply(SparkILoop.scala:945)
at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$process(SparkILoop.scala:945)
at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059)
at org.apache.spark.repl.Main$.main(Main.scala:31)
at org.apache.spark.repl.Main.main(Main.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$runMain(SparkSubmit.scala:683)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:189)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:214)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.NumberFormatException: For input string: "|"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.parseInt(Integer.java:615)
at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$anonfun$1.apply(<console>:28)
at $iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$iwC$anonfun$1.apply(<console>:26)
at scala.collection.Iterator$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$anon$10.next(Iterator.scala:312)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$anonfun$take$1$anonfun$28.apply(RDD.scala:1302)
at org.apache.spark.rdd.RDD$anonfun$take$1$anonfun$28.apply(RDD.scala:1302)
at org.apache.spark.SparkContext$anonfun$runJob$5.apply(SparkContext.scala:1850)
at org.apache.spark.SparkContext$anonfun$runJob$5.apply(SparkContext.scala:1850)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:66)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
... View more
Labels:
11-18-2018
10:58 PM
After trying multiple options finally fixed this issue to passing per node and rack min sizes. I passed the values are parms to Sqoop like below. sqoop export \
-Dmapreduce.input.fileinputformat.split.minsize.per.rack=749983 \
-Dmapreduce.input.fileinputformat.split.minsize.per.node=749983 \
--connect jdbc:mysql://01-mysql-test232855.envnxs.net:3306/retail_export \
--username autoenv_root \
--export-dir /user/hive/warehouse/retail_db.db/orders \
-table orders\
-P
... View more
11-18-2018
10:56 PM
After trying multiple options. I fixed the issue by giving the below options in SQOOP Export. I fixed the per node min split size, min rack size. Then the job ran successfully. sqoop export \
-Dmapreduce.input.fileinputformat.split.minsize.per.rack=749983 \
-Dmapreduce.input.fileinputformat.split.minsize.per.node=749983 \
--connect jdbc:mysql://01-mysql-test232855.envnxs.net:3306/retail_export \
--username autoenv_root \
--export-dir /user/hive/warehouse/retail_db.db/orders \
-table orders\
-P
... View more
11-15-2018
08:45 PM
Please find the screen shot the HIVE QL. screen-shot-2018-11-15-at-34106-pm.png
... View more
11-15-2018
08:44 PM
It is working perfectly for Hive1.1.0-cdh5.4.0. Below I mentioned hive QL I used for creating an dummy table. Also attached the screen shot for the reference.
hive (retail_warehouse)> create table vendor(
> vendor_id int,
> vendor_name string)
> row format delimited fields terminated by "\;"
> stored as textfile;
OK
Time taken: 0.857 seconds
... View more
11-15-2018
08:26 PM
I am using Hive 1.1.0-cdh5.4.0. My HIVE runs in cloudera 5.4.0.
mpalanisamy@01:~$ hive --version
Hive 1.1.0-cdh5.4.0
Subversion file:///data/jenkins/workspace/generic-package-ubuntu64-14-04/CDH5.4.0-Packaging-Hive-2015-04-21_12-09-14/hive-1.1.0+cdh5.4.0+103-1.cdh5.4.0.p0.56~trusty -r Unknown
Compiled by jenkins on Tue Apr 21 12:12:12 PDT 2015
From source with checksum 2bf708133bf40715eaa74b142056808a
... View more
11-15-2018
08:12 PM
I am getting the below error while doing SQOOP EXPORT. java.io.IOException: Minimum split size pernode 536870912 cannot be larger than maximum split size 41at org.apache.sqoop.mapreduce.CombineFileInputFormat.getSplits(CombineFileInputFormat.java:200)
at org.apache.sqoop.mapreduce.ExportInputFormat.getSplits(ExportInputFormat.java:73)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:597)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1306)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1303)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1303)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1324)
at org.apache.sqoop.mapreduce.ExportJobBase.doSubmitJob(ExportJobBase.java:324)
at org.apache.sqoop.mapreduce.ExportJobBase.runJob(ExportJobBase.java:301)
at org.apache.sqoop.mapreduce.ExportJobBase.runExport(ExportJobBase.java:442)
at org.apache.sqoop.manager.SqlManager.exportTable(SqlManager.java:931)
at org.apache.sqoop.tool.ExportTool.exportTable(ExportTool.java:80)
at org.apache.sqoop.tool.ExportTool.run(ExportTool.java:99)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252) I removed the most of the fields and renamed the table to some other name due to project compliance. Below I mentioned the Sqoop Export I used create table sup_api_bidder_test(
id int,
name string,
vendor_id bigint)
... View more
- Tags:
- sqoop-export
Labels:
11-15-2018
04:12 PM
I issue got fixed after adding --driver class. New SQOOP import looks like below.
sqoop import
--connect jdbc:mysql://01-mysql-test232855.envnxs.net:3306/retail_db
--username autoenv_root
-P
--query "select * from agg_dw_orders_lnin_customers where \$CONDITIONS"
--hive-import
--hive-table agg_dw_orders_lnin_customers
--hive-database hivesqoop
--target-dir /user/hive/warehouse/orders_warehouse.db/agg_dw_orders_lnin_customers
--split-by order_id
--drive com.mysql.jdbc.Driver
... View more
11-14-2018
11:38 PM
I am getting the below error while SQOOP HIVE import. It is working fine for another table. But the same set of instructions are not working for this tables. Even the same code worked fine in the production box. Below I mentioned the error messages. My SQOOP command: sqoop import
--connect jdbc:mysql://01-mysql-test232855.envnxs.net:3306/retail_db
--username autoenv_root
-P
--query "select * from agg_dw_orders_lnin_customers where \$CONDITIONS"
--hive-import--hive-table agg_dw_orders_lnin_customers
--hive-database hivesqoop
--target-dir /user/hive/warehouse/orders_warehouse.db/agg_dw_orders_lnin_customers
--split-by order_id 18/11/14 23:32:08 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
Enter password:
18/11/14 23:32:25 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override
18/11/14 23:32:25 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc.
18/11/14 23:32:25 WARN tool.BaseSqoopTool: It seems that you're doing hive import directly into default
18/11/14 23:32:25 WARN tool.BaseSqoopTool: hive warehouse directory which is not supported. Sqoop is
18/11/14 23:32:25 WARN tool.BaseSqoopTool: firstly importing data into separate directory and then
18/11/14 23:32:25 WARN tool.BaseSqoopTool: inserting data into hive. Please consider removing
18/11/14 23:32:25 WARN tool.BaseSqoopTool: --target-dir or --warehouse-dir into /user/hive/warehouse in
18/11/14 23:32:25 WARN tool.BaseSqoopTool: case that you will detect any issues.
18/11/14 23:32:26 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
18/11/14 23:32:26 INFO tool.CodeGenTool: Beginning code generation
18/11/14 23:32:26 INFO manager.SqlManager: Executing SQL statement: select * from orders where (1 = 0)
18/11/14 23:32:26 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@564718df is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@564718df is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2747)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1911)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2163)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2618)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2568)
at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1557)
at com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3002)
at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602)
at com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:305)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForQuery(SqlManager.java:253)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:336)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1872)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1671)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
18/11/14 23:32:26 ERROR tool.ImportTool: Import failed: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1677)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
... View more
- Tags:
- Data Processing
- Sqoop
Labels:
11-14-2018
01:40 AM
This issue is fixed by using the --drive like below in the SQOOP script
--driver com.mysql.jdbc.Driver
... View more
11-13-2018
10:04 PM
I am getting error while importing the mysql into HDFS using sqoop. But the job runs in production successfully. But in my test environment it is not working. Even mysql jar is available in the test environment. com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:344)
at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2332)
at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2369)
at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2153)
at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:792)
at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:381)
at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:305)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:247)
at org.apache.sqoop.manager.SqlManager.makeConnection(SqlManager.java:904)
at org.apache.sqoop.manager.GenericJdbcManager.getConnection(GenericJdbcManager.java:59)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:763)
at org.apache.sqoop.manager.SqlManager.execute(SqlManager.java:786)
at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:289)
at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:260)
at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:246)
at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:327)
at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1872)
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1671)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:244)
at com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:257)
at com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:294)
... 33 more
18/11/13 19:19:46 ERROR tool.ImportTool: Import failed: java.io.IOException: No columns to generate for ClassWriter
at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1677)
at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:106)
at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:501)
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:628)
at org.apache.sqoop.Sqoop.run(Sqoop.java:147)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:183)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:234)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:243)
at org.apache.sqoop.Sqoop.main(Sqoop.java:252)
screen-shot-2018-11-13-at-22404-pm.png
... View more
Labels:
11-07-2018
03:25 PM
I fixed this issue by using the below by adding escape sequence.
create table ... ... row format delimited fields terminated by
'\;'
... View more
11-06-2018
10:20 PM
I am trying to create a hive table with ";" as delimiter for my project. Because the input file that I am getting from another application having data delimited by ";" .
When I try to create the table by giving
row format delimited fields terminated by ',' I am getting the below error.
MismatchedTokenException(-1!=302)
at org.antlr.runtime.BaseRecognizer.recoverFromMismatchedToken(BaseRecognizer.java:617)
at org.antlr.runtime.BaseRecognizer.match(BaseRecognizer.java:115)
at org.apache.hadoop.hive.ql.parse.HiveParser.tableRowFormatFieldIdentifier(HiveParser.java:34966)
at org.apache.hadoop.hive.ql.parse.HiveParser.rowFormatDelimited(HiveParser.java:33919)
at org.apache.hadoop.hive.ql.parse.HiveParser.tableRowFormat(HiveParser.java:34195)
at org.apache.hadoop.hive.ql.parse.HiveParser.createTableStatement(HiveParser.java:4979)
at org.apache.hadoop.hive.ql.parse.HiveParser.ddlStatement(HiveParser.java:2355)
at org.apache.hadoop.hive.ql.parse.HiveParser.execStatement(HiveParser.java:1579)
at org.apache.hadoop.hive.ql.parse.HiveParser.statement(HiveParser.java:1057)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:199)
at org.apache.hadoop.hive.ql.parse.ParseDriver.parse(ParseDriver.java:166)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:393)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:307)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1110)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1158)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1047)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1037)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:207)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:159)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:370)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:756)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:615)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
FAILED: ParseException line 6:62 mismatched input '<EOF>' expecting StringLiteral near 'by' in table row format's field separator
... View more
- Tags:
- Data Processing
- Hive
Labels:
11-06-2018
03:55 PM
I could fix this issue by adding "import _root_.kafka.serializer.StringDecoder" in my imports. I attached the screen shot of the my code. screen-shot-2018-11-06-at-105540-am.png
... View more
11-06-2018
03:51 PM
1 Kudo
I could create the light weight jar following the below steps. (i) Project Structure -> artifacts -> click the "+" symbol (ii) Now go to JAR -> Empty -> <give name of the jar in "Name"> (iii) Now press "Apply" button (iv) Now you can see the jar file created with the given name. On top of that we can see a "+" symbol (v) Now select "Module output" and then select your project (vi) Create Manifest if it is not available. This way I created jar file with less than 125 KB. If I add all the dependencies it is coming around 300 MB.
... View more
11-05-2018
10:12 PM
In my project I need to use Kafka ad Spark streaming for doing real time analytics. So I am trying to import Kafka and Spark streaming dependencies. In order to read the Kakfa topics in my spark streaming program, I am trying to import kafka.serializer.StringDecoder. But I am not able to import it.screen-shot-2018-11-05-at-51111-pm.png Even I wrote a dummy program to fix this issue. But still I couldn't find a solution.
... View more
- Tags:
- Kafka
- spark-streaming
Labels:
11-05-2018
02:46 AM
How to create jar without dependencies in intellij. I mean only jar with our main class instead of jar with all the dependencies are added to the main class jar. This way I am looking for light weight jar
... View more
- Tags:
- intellij
11-02-2018
02:02 PM
Hi Guys, In my project I need to validate the Kafka messages. I mean I want to validate the qualify of the data from kafka message. ex: I need to get the log messages to the source of my analytics application. I need to know whether the message I received as it is from kafka. Is there any tool (or) way to validate it?
... View more
Labels:
12-29-2017
11:03 PM
Hi Guys, I am trying to use Spring STS for as IDE for my mapreduce programs. But when ever I try to launch STS.exe, I am getting main class of "org.eclipse.equinox.launcher_1.4.0.v20161219-1356.jar" not found error. But I can see the the missing jar file in the path. Is anybody faced the same problem in the past? Do you have any idea what went wrong. Even I restarted the system and getting the same error. I attached the screen shot of the issues. Regards, P.Mahendiran
... View more
12-19-2017
08:19 PM
Hi Guys, The issue got fixed now. I did the following. Root Cause: The system expects the "application.properties" file under main/resources folder. In my project resources folder was missing. Solution: I created a folder called "resources" under main and moved "application.properties" under it. Now the issue is fixed. I also attached the screen shot of the fix. Regards, P.Mahendiran
... View more
12-19-2017
08:09 PM
Thanks, Shu.. The problem is fixed after I move the file to local. Thanks for the response.
... View more
12-19-2017
08:05 PM
Hi Guys, I am trying to typesafe config in my scala project. I am using Intellij to run this program. But I am getting "Exception in thread "main" com.typesafe.config.ConfigException$Missing: No configuration setting found" error. I also attached the screen shot of the error. Regards, P.Mahendiran
... View more
Labels:
12-08-2017
02:13 AM
Hi Guys, I am trying to use the avro-tools to get the avro schema from avro file. But I am getting the message like saying "file doesn't exist". If I issue "Hadoop fs -ls" command I can see the file. Even I can see the avro file content by giving the path. But if I issue the "avro-tools getschema <avro file path> I am getting file doesn't exist error. Anybody got the same type of error earlier. I also attached the screen shot of the error. Regards, P.Mahendiran
... View more
- Tags:
- avroschema