Hi @Jan Kytara. Can you please update statistics on the table - run the command: analyze table corrupt_rows compute statistics ; Also would love to know if "select * from corrupt_rows limit nnn ;" returns properly formed rows with columns A..L, or if it has junk or boundaries. That could point to a delimiter issue. ... View more

Hi @Avinash P. Can you please confirm that table dg_2 does have a column called 'phone'? ... View more

Hi @Kaliyug Antagonist. The answers above from @slachterman and @mqureshi are excellent. Here is another way (at a higher-level) to look at this problem. Here are some tips to plan out a DR strategy for the smoldering datacenter problem that is mentioned above. 1. Use the term Disaster Recovery instead of Backup. This gets the administrators to move away from the RDBMS-like idea that they can simply run a script and recover the entire cluster. 2. Discuss RTO/RPO and let the business answers drive the architecture. RTO and RPO requirements need to be defined by the Business - these requirements drive all decisions around Disaster recovery A 1-hour/1-hour RTO/RPO is wildly different (cost and architecture) from a 2-week/1-day RTO/RPO. When they choose the RTO/RPO requirements they are also choosing the required cost & architecture. By having well-defined RTO/RPO requirements you will avoid having an over-engineered solution (which may be far too expensive) and will also avoid having an under-engineered solution (which may fail precisely when you need it most - during a Disaster event) 3.'Band’ your data assets into different categories for RTO/RPO purposes. Example: Band 1 = 1 hour RTO. Band 2 = 1 day RTO. Band 3 = 1 week RTO, Band 4 = 1 month RTO, Band 5 = Not required in the event of a disaster You would be surprised how much data can wait in the event of a SEVERE crash. For example, datasets that are used to provide a report that is distributed once per month - they should never require a 1-hour RTO. Hope that helps. ... View more

select TO_DATE(created_at), DATEDIFF(TO_DATE(current_date()), TO_DATE(sales_flat_order.created_at)) as delay, count(*) as NumberOfOrders FROM magentodb.sales_flat_order WHERE status IN ( 'packed' , 'cod_confirmed' ) GROUP BY TO_DATE(created_at), DATEDIFF(TO_DATE(current_date()), TO_DATE(sales_flat_order.created_at)) ... View more

hi @Tom McCuch and @vamsi valiveti. Just wanted to clarify - it is legal to have two bucketed tables where the number of buckets in one table is a multiple of the number of buckets in the other table, but for pragmatic performance reasons it is best to have the number of buckets be the same. IMHO If you are going to bucket your data, you are doing it because you need a more efficient join - and having a non-matching number of buckets removes that ability to do a sort-merge bucket join. See this post on bucket join versus sort-merge bucket join. it's very good. http://stackoverflow.com/questions/20199077/hive-efficient-join-of-two-tables ... View more

Hi Andrea. I suspect the issue is as Kuldeep mentioned... But please do send the DDL and query - I would be glad to try to recreate it. bpreachuk@hortonworks.com ... View more

Partitioning by Month is very acceptable, especially if the data comes in on a monthly basis. @Joseph Niemiec has written a great writeup on why you should use single Hive partitions like YYYYMMDD, YYYY-MM-DD, YYYYMM, YYYY-MM. Do this instead of nested partitions like YYYY/MM/DD or YYYY/MM. The reason is fairly simple - it makes for simpler querying with LIKE, IN and BETWEEN... and the the Hive optimizer can do partition pruning on those queries. ... View more

Hi Andrea. Could you post the UNION ALL query you are executing along with the DDL for the tables involved? (via 'Show create table <tblname>')? That would help the debugging... ... View more