About pmj

pmj · ‎01-09-2018

Hi, I have an ORC table which updates every 5 min in different date partitions, i want to run the CONCATENATE alter table command, but how do i run it on all partitions at once? with the below command i could do only on single partition: ALTER TABEL tablename PARTITION (dt=20180109) CONCATENATE; Thanks.

pmj · ‎12-22-2017

@bkosaraju Thanks a lot, the splitting part works.... but i am still getting only the first match ... how do i get all matches?

pmj · ‎12-21-2017

@bkosaraju The separator I used is Select tickets, split(all_comments, ‘$’) from table I also used another separator which says “endhere” but nothing works. Thanks.

pmj · ‎12-20-2017

@bkosaraju it works but i am getting only 1 value, first match . what to do to get all matches?

pmj · ‎12-19-2017

@bkosaraju Thanks for the reply. when i do this, it is giving me only first match and also the expression i used to split(ts_repid) is not showing in the output. ts_repid is timestamp<space>repid Also, expTBL will create a new table? how do i actually show the output of the above command? i tried show() and it only showed first row.. Thanks again.

pmj · ‎12-18-2017

Hi, I have a table in hive like this: Ticket# All_Comments a timestamp1 repid1 comments1 timestamp2 repid2 comments2 timestamp3 repid3 comments3 b timestamp1 repid1 comments1 timestamp2 repid2 comments2 timestamp3 repid3 comments3 timestamp4 repid4 comments4 Ticket# Timestamp Repid Comments a timestamp1 repid1 comments1 a timestamp2 repid2 comments2 a timestamp3 repid3 comments3 b timestamp1 repid1 comments1 b timestamp2 repid2 comments2 b timestamp3 repid3 comments3 b timestamp4 repid4 comments4 I am using pyspark to do this. Comments are a set of words any length. I have regex to catch the timestamp and rep id togetehr to split the string by regex ... so this will separate comments. Not sure how to proceed. Please help.

pmj · ‎10-24-2017

@Shu I am trying to use hive -e "select ,, * to txt but after logging initiliazed nothing is happening: Logging initialized using configuration in file:/etc/hive/2.5.3.0-37/0/hive-log4j.properties OK Is it because of the size of the table?

pmj · ‎10-24-2017

@Divakar Annapureddy I have data hourly with about 9+ million rows, firstly it is taking a lot of time to download it to csv from ambari view and secondly it is only exporting 47K+ rows into csv .. not sure why Is there any better way ? Also, what are some pros and cons of converting a text file to orc in terms of joins and anything else that is to be noted? Thanks.

pmj · ‎10-24-2017

I have a text data(.csv) file in hive external table. I am planning to use orc compression on the text data by creating a new orc table (the compression rate is more than 10 x times better...) and then i would like to delete the external tables csv file. In some cases if i want to convert the ORC table back into csv file, what is the easiest way to do it? i cannot do it directly when i do hive -e "select ... from orc table" >> /tmp/csvfile, it is just stuck.

pmj · ‎10-20-2017

@kgautam how about parquet?

Online	Offline
Last Visited	‎04-02-2020 02:53 PM

Member Since	‎10-24-2015 06:29 PM
Last Visited	‎04-02-2020 02:53 PM
Posts	207
Kudos received	17

Cloudera Community

Re: alter table add partition took almost an hour

Re: Decommisiong status for a dead node

Re: Service check failing for YARN & Mapreduce...

Re: Install stops at Datanode through Ambari

orc small files Concatenate in Hive

Re: split column by regex and create a table

Re: split column by regex and create a table

Re: split column by regex and create a table

Re: split column by regex and create a table

split column by regex and create a table

Re: convert orc table data into csv

Re: convert orc table data into csv

convert orc table data into csv

Re: best Compression technique?