Member since
07-20-2018
5
Posts
0
Kudos Received
0
Solutions
12-13-2018
12:51 PM
Hi @Mahesh Thanks for your reply... 1. Table is in orc format 2. Yes I tried increasing number of mappers. Too many mappers causing Oracle side locking, too many insert operation on the same block. I tried using -Dsqoop.export.records.per.statement and -Dsqoop.export.statements.per.transaction with --batch and --direct but seems it doesn't help that much, --fetch-size didn't work with export Thanks!
... View more
12-12-2018
11:35 PM
Hi Experts, I am sqooping out a Hive table with 20-30 columns and apprx. 100 millions records to Oracle. sqoop export -D mapred.job.queue.name=<yarn queue name> -Dsqoop.export.records.per.statement=100000 --connect jdbc:oracle:thin:<db server> --username xxxx --P --table employee --hcatalog-database default --hcatalog-table employee --num-mappers 10 --batch
Source Hive table is unpartitioned and has 200 part files, total 1.1 GB of data. And destination oracle table has identity defined on pk column which has cache of 100000.
It's taking approx 5-6 minutes to sqoop out data completely with default fetch size 1000. Any suggestion how can I improve more to get better performance. Is there anyway I can change fetch size? Thanks in advance.
... View more
Labels:
- Labels:
-
Apache Sqoop
11-27-2018
07:52 PM
Hi Experts, I want to offload data from my hive table to apache cassandra cluster and was evaluating possible options. Tables size can vary from small to large. Is any tool other than sqoop and spark can someone recommend which is being used mostly and have performance benefit as well. Thanks in advance!
... View more
Labels:
- Labels:
-
Apache Hive
07-26-2018
07:25 PM
Thank you Shawn for your prompt response. I found an alternate way. Did UTF-8 conversion using iconv before reading in external table with RegexSerDe. In my case Hive by default supports UTF-8 charactersets.
... View more
07-21-2018
09:09 AM
Hi, I have a requirement to load Fixed Width file in hive table where input file is not always UTF-8 encoded.
I found 2 different classes are available for this - 'org.apache.hadoop.hive.serde2.RegexSerDe' to read from fixed width file on defined offset values and 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe' for non utf8 encoding. But unable to use them together when creating external table.
Can someone of you please help me with a solution. Thanks in advance!!
... View more
Labels:
- Labels:
-
Apache Hive