- Subscribe to RSS Feed
- Mark Question as New
- Mark Question as Read
- Float this Question for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
how to load only new records from HIVE table
- Labels:
-
Apache Hive
Created ‎06-22-2023 09:32 PM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi every one
I have table in HIVE, that consists product_name, product_id, total sales (only three columns)
Table name : total_sales dataset
dataset:
yesterday sales:
product_ name | Product_ id | total sales |
1000 | laptop | 500 |
1001 | mobile | 5000 |
1002 | hard drive | 550 |
Note:
With yesterday datasets (3 records) i can calculate final stock list in spark.
spark :
product name | Product id | initial stock | total sales | Final stock |
1000 | laptop | 1000 | 500 | 500 |
1001 | mobile | 15000 | 5000 | 10000 |
1002 | hard drive | 1000 | 550 | 450 |
Today sales
product_ name | Product_ id | total sales |
1000 | laptop | 100 |
1001 | mobile | 500 |
1002 | hard drive | 200 |
note:
Three more records get added in total_sales dataset (HIVE).
total_sales table:
product name | Product id | total sales |
1000 | laptop | 500 |
1001 | mobile | 5000 |
1002 | hard drive | 550 |
1000 | laptop | 100 |
1001 | mobile | 500 |
1002 | hard drive | 200 |
Question:
- How to check any new records get added or not in total_sales dataset.
- If new dataset gets added, how to export only new records?
Can anyone help me on this please
Thanks
sen
Created ‎06-23-2023 07:20 AM
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Welcome to the community @sencae
While you wait for a more knowledgable person to respond, I did find this older post that hopefully gets you closer to where you need to be.
Cy Jervis, Manager, Community Program
Was your question answered? Make sure to mark the answer as the accepted solution.
If you find a reply useful, say thanks by clicking on the thumbs up button.
