Member since
12-21-2016
83
Posts
5
Kudos Received
2
Solutions
My Accepted Solutions
Title | Views | Posted |
---|---|---|
44114 | 02-08-2017 05:56 AM | |
7143 | 01-02-2017 11:05 PM |
02-16-2017
03:15 AM
I found a solution to export this kind of data to any RDBS in the form of UTF8 or any other character set by giving the specific character set after the database/host name.
... View more
03-20-2017
07:14 PM
Hi Reddy.. Choose a delimiter which will not used easily in a data. Choose unicode as delimiter it will solve your issue. 90% of the data will not contain unicode. (row format delimited Fields terminated by '/u0001') . In your case export the the data with '/u0001' as delimiter and then insert into a hive table which has delimiter as '|'
... View more
02-08-2017
12:14 PM
it is possible, I see people sqoop into manged table then alter table to be external, sqoop continues to work http://stackoverflow.com/questions/27991258/how-to-create-external-table-in-hive-using-sqoop-need-suggestions#29602510 Another option is to manually create table in hive as external and sqoop reference to that table, see section on incremental_table http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.0/bk_dataintegration/content/incrementally-updating-hive-table-with-sqoop-and-ext-table.html For sqoop to create an external table outright, you might want to open an enhancement jira
... View more
02-26-2017
06:18 PM
For Pig and Hive implementations, I'd suggest you create a UDF. If new territory for you, here are some quick blog posts on creating (simple) UDFs for Pig and Hive; https://martin.atlassian.net/wiki/x/C4BRAQ and https://martin.atlassian.net/wiki/x/GoBRAQ. Good luck
... View more
04-27-2019
03:55 PM
To drop partitions with a Range filter, use below syntax. (CDH 5.7, Hive 1.1.0). ALTER TABLE tableName DROP PARTITION (date >='20190410', date <='20190415'); Example: Before: +-----------------------+--+ | partition | +-----------------------+--+ | date=20190401 | | date=20190402 | | date=20190403 | | date=20190404 | | date=20190405 | | date=20190406 | | date=20190410 | | date=20190411 | +-----------------------+--+ ALTER TABLE tableName DROP PARTITION (date >='20190410', date <='20190415'); AFTER: +-----------------------+--+ | partition | +-----------------------+--+ | date=20190401 | | date=20190402 | | date=20190403 | | date=20190404 | | date=20190405 | | date=20190406 | +-----------------------+--+
... View more
02-08-2017
05:56 AM
2 Kudos
Here is work around, which i have implemented. For an external table, If you are trying to drop a partition and as-well would like to delete the data. This can be achieved as below. 1. Alter external table as internal table -- by changing the TBL properties as external =false 2. Drop the partitions -- when you drop the partitions, data pertained to the partitions will also be dropped as now this table is managed table 3. Alter back the table as external=True. By doing this, there more controlled on what we are deleting and drop the partitions rather than using hadoop rm command
... View more
01-21-2017
11:54 PM
1 Kudo
Hi @Praveen PentaReddy, 1. If you want to backup a data in archive location as a table, then create a table as original one create external table MYTABLE_archive like MYTABLE location '<hdfs_archive_location>'; 2. Copy the data from original table to archive one: 2.a. using HQL: INSERT INTO TABLE MYTABLE_archive partition(date) SELECT * from MYTABLE where date>= <start_date> and date<=<end_date>;
-- dynamic partitioning should be enabled; 2.b. using file transfer: Identify location of each partition to be copied. Copy directories to the location of archive table and then add partitions: hdfs dfs -cp <orig_partitions_loc> <archive_table_location>/
# alternatively you can use distcp in case of huge volumes, but check if you have permissions to execute it. Also make sure you understand the difference between those two command in regards to file attributes. then add new partitions in archve: MSCK REPAIR TABLE MYTABLE_archive;
... View more
01-04-2017
07:05 PM
Thanks.. It was typo mistake where i didn't capitalize 'T' in fileType
... View more
12-27-2016
07:34 AM
what is the difference between both of the configuration which i posted and the one which you gave me?
... View more
- « Previous
-
- 1
- 2
- Next »