Member since
03-28-2016
194
Posts
18
Kudos Received
0
Solutions
05-27-2016
01:56 PM
Please find the details abc\u0001paper\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30
abc\u0001paper1\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30
abc\u0001paper2\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30
abc\u0001paper3\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30
abc\u0001paper4\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30
abc\u0001paper5\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30
... View more
05-27-2016
01:25 PM
I tried ruining the command, i dont see any change onthe file format. and i still getting NULL values...
... View more
05-27-2016
12:29 PM
Is the file is in correct format. Do i have any option to have any REGULAR EXPRESSION to accept this file format. ?
... View more
05-27-2016
12:11 PM
This is the data file, separated by \u0001 abc\u0001su\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001test1\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001resh2\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001sush3\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001test5\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001tes6\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001test52\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001sh3\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001s4\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001s5\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001s6\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30
... View more
05-27-2016
11:29 AM
Hi Team, Iam using delimiter /u0001 on Hive. When i insert data on that table i see just data on the first column. Rest of the column are showing as a NULL. Please help find the details below TABLE ::: CREATE TABLE
`man_def_3`(
| | `cntry_cde`
char(3),
| | `cntry_nm`
varchar(45),
| | `iso_cntry_nbr`
char(3),
| | `intl_tran_ind`
char(1),
| | `ofac_prhbt_tran_ind` char(1),
| | `dflt_fin_nwk_cde`
char(5),
| | `eff_dt`
string,
| | `exp_dt`
string,
| | `trnsfrm_dt`
string)
| | ROW FORMAT
DELIMITED
| | FIELDS TERMINATED BY
'\u0001'
| | LINES TERMINATED BY
'\n'
| | STORED AS
INPUTFORMAT
| |
'org.apache.hadoop.mapred.TextInputFormat'
| |
OUTPUTFORMAT
| |
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
| |
LOCATION
| |
'hdfs://dshdp-dev-cluster/apps/hive/warehouse/neela.db/cntry_def_1' | | TBLPROPERTIES
(
| |
'COLUMN_STATS_ACCURATE'='true',
| |
'numFiles'='1',
| |
'totalSize'='2654',
| | 'transient_lastDdlTime'='1464346137') OUTPUT: select * from man_def_3 ; +------------------------+-----------------------+----------------------------+----------------------------+----------------------------------+-------------------------------+---------------------+---------------------+-------------------------+--+ | cntry_def_3.cntry_cde | cntry_def_3.cntry_nm |
cntry_def_3.iso_cntry_nbr | cntry_def_3.intl_tran_ind |
cntry_def_3.ofac_prhbt_tran_ind | cntry_def_3.dflt_fin_nwk_cde |
cntry_def_3.eff_dt | cntry_def_3.exp_dt |
cntry_def_3.trnsfrm_dt | +------------------------+-----------------------+----------------------------+----------------------------+----------------------------------+-------------------------------+---------------------+---------------------+-------------------------+--+ |
abc
|
NULL
| NULL
|
NULL
|
NULL
|
NULL
|
NULL
| NULL
|
NULL
| |
abc
|
NULL
| NULL
|
NULL
|
NULL
| NULL
|
NULL
|
NULL
| NULL
| |
abc
|
NULL
| NULL
|
NULL
|
NULL
| NULL |
NULL
|
NULL
| NULL
| |
abc
|
NULL
| NULL
|
NULL
|
NULL
| NULL
|
NULL
|
NULL
|
NULL
| |
abc
|
NULL
| NULL
|
NULL
|
NULL
| NULL
|
NULL
| NULL
|
NULL
| Part of Data : abc\u0001su\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001test1\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001resh2\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001sush3\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001test5\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001tes6\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001test52\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001sh3\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001s4\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001s5\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30 abc\u0001s6\u0001abc\u0001Y\u0001Y\u0001Test\u00012014-05-30\u00012015-05-30\u00012015-05-30
... View more
Labels:
- Labels:
-
Apache Hive
05-13-2016
09:46 AM
Thanks for your information, Alt east can you tell me the advantage of SKEW joins and where to use ? and Instead of using multiple joins what is the best way to run the qurey
... View more
05-13-2016
09:06 AM
1 Kudo
Hi Team Can you help to under Stand HIVE best practices on Horton works HDP 2.3, to support better HIVE Best Practices
1.What is the max joins that we can used in Hive for best
performance ? what is the limitation of using joins ? what happen if we use
multiple joins (will it affect performance or Job fail )?
2.While Querying what kind of fields should be used for
join keys?
3.How will you make use of Partitioning and bucketing
4.Criticality of type casting ? Converting the data types on fly
over the queries ?
5.Using multiple casting will it affect the HIVE job performance ?
6.How to avoid using multiple Internal Joins, any alternative that
we can use of avoiding multiple joins?
7.What is the best way of doing splitting
8.when to use left outer join and right outer join to avoid full
table scan.
9.What is best way to use select query instead of scanning
full table
10.Map join optimization ? when to use Map joins ?
11.SKEW join optimization ? when to use SKEW joins?
12.SMB join optimization? When to go SMP joins ?
13.During huge data process what needs to do to prevent from
job failures ? what is the best practices in that scenario ?
14.Advantage of De-Normalization and where should I use on HIVE
... View more
Labels:
- Labels:
-
Apache Hadoop
-
Apache Hive
05-13-2016
09:02 AM
Thanks for the information. as per your above reply 0.99f but 0.999 is the value. just want to make sure. 0.99 is also a correct value. ?
... View more
05-12-2016
07:10 AM
I want to add the below parameter in hdfs-site.xml."""" Property name: ha.zookeeper.acl Value = sasl:nn:rwcda """ since its belong to zookeer do i need to restart the zookeerper service ?
... View more
05-11-2016
01:07 PM
One last question, I don't have this parameter on my cluster. dfs.namenode.replication.min=1 . Is it mandatory to have it on the cluster..
... View more