Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Counting the number of rows after writing to a dataframe to a database with spark

Highlighted

Counting the number of rows after writing to a dataframe to a database with spark

New Contributor

I have seen similar question on stack overflow

but I am not really sure

1. How to use the code in actual working example. I have written some code but it is not working for the outputting the number of rows inputting rows works. The output metrics are always none.

16167-codeforlogginginputoutput.png

Code writing to db.

16168-savingtodb.png

2. the best or preferred way of doing this.

https://stackoverflow.com/questions/37496650/spark-how-to-get-the-number-of-written-rows

https://stackoverflow.com/questions/43934168/how-to-get-the-number-of-records-written-using-datafram...

Basically it seems like I can get the row count from the spark ui but how can I get it from within the spark code.

I know that before I write the database I can do a count on a dataframe but how do it after I write to get the count.

I have posted a lot of info but I just want to know how can I see programmatically the number of rows written by a dataframe to a database.


codeforlogginginputoutput.png
2 REPLIES 2

Re: Counting the number of rows after writing to a dataframe to a database with spark

New Contributor

Hi

I'm facing same issue. Can you please let me know , if you found the solution?

Thanks

Re: Counting the number of rows after writing to a dataframe to a database with spark

Super Guru