Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

Row count mismatch for kudu table

Row count mismatch for kudu table

New Contributor
Dear All,

Need your kind assistance on my below question.I have a kudu table(lets say xyz). When i am trying to get the count for the table from impala i am getting 300000 approx records however same table count from pyspark2 shows only 100000 approx. Why is there a mismatch?Appreciate your kind help and support on this.As this is my initial post, appologize if i am missing from any details here. Kindly let me know if any details needed from my end.
4 REPLIES 4

Re: Row count mismatch for kudu table

Guru
Hi,

Can you please show us how you create the table and did the count from both Impala and pyspark2?

Thanks
Eric

Re: Row count mismatch for kudu table

New Contributor

Hi Eric,

 

Thanks for your time and sorry for my delayed response. Here is the narrated view of the issue.

 

1)Getting the details from pyspark:

 

pyspark2 --jars /usrapps/tmgbatch/lib/kudu-spark2_2.11-1.1.0.jar

 

kuduDF = spark.read.format('org.apache.kudu.spark.kudu')\.option('kudu.master','defrag1.cg.com,defrag2.cg.com')\.option('kudu.table','impala::tbtch_tware.tware_sf')\.load()

 

 kuduDF.count() shows 1,329,145,670

 

From impala:

 

select count(*) from tbtch_tware.tware_sf;

I get 3,111,497,258.

 

2)The table was created from impala.

 

Let me know if i am able to provide the details required.

 

Thanks,

 

 

 

Re: Row count mismatch for kudu table

New Contributor
Hi Eric,

Would you be so kind to assist me on above plz?

Thanks,

Re: Row count mismatch for kudu table

Guru
Hi,

Not too sure, but the next step I would like to do is print out the records and see if you can spot anything obvious:

kuduDF.show(1000)
select * from tbtch_tware.tware_sf;

And compare them.

Let me know how you go.

Cheers
Eric
Don't have an account?
Coming from Hortonworks? Activate your account here