Received ImpalaRuntimeException: Error creating Kudu table

[#1]

Below is the create statement:

CREATE TABLE cv00_atty_kudu (
  bar_num STRING,
  ent_ts STRING,
  atty_name StRING,
  firm_name STRING,
  tax_id STRING,
  stat STRING,
  addr_tkn STRING,
  last_upd_ts STRING,
  last_upd_user STRING,
  last_upd_pgm STRING,
  PRIMARY KEY (ent_ts)
)
PARTITION BY HASH PARTITIONS 2
STORED AS KUDU;
INSERT INTO cv00_atty_kudu SELECT * FROM cv00_atty;

----------

This create failed with the following error msg:

ImpalaRuntimeException: Error creating Kudu table 'impala::default.cv00_atty_kudu' CAUSED BY:
NonRecoverableException: Got out-of-order key column: name: "ent_ts" type:
STRING is_key: true is_nullable: false cfile_block_size: 0

----

Any help in pointing the error is appreciated.

[#2]

Hi,

This is a current limitation in Kudu (see the second bullet point): http://kudu.apache.org/docs/known_issues.html#_schema_and_usage_limitations

Putting ent_ts first in the schema will solve this.

Arguably the error message could be better.

Regards,

J-D

[#3]

I switched the cols and made the bar_num the primary key as it is a unique non-null value. I was able to successfully create the Kudu table.  However, when I try to populate it using the statement:

INSERT INTO cv00_atty_kudu SELECT * FROM cv00_atty;

I received the following error:

Kudu error(s) reported, first error: Timed out: Failed to write batch of 13319 ops to tablet 076cd0db50034edc9448b1b9862dde84 after 175 attempt(s): Failed to write to server: (no server available): Write(tablet: 076cd0db50034edc9448b1b9862dde84, num_ops: 13319, num_attempts: 175) passed its deadline: Network error: Client connection negotiation failed: client connection to 10.91.62.124:7050: connect: No route to host (error 113)

Which did not make sense as according to the Cloudera Manager:  it shows the following status

• Master

  • 1 Good Health
• Tablet Server

  • 3 Good Health
• Hosts

  • 3 Good Health

  Your help is appreciated.

[#4]

The error suggests that your Impala processes can't communicate with the Kudu tservers. Impala gets the addresses of the tservers from the Kudu Master.

Look at the /tablet-servers page in the Kudu Master web UI; are the published tserver addresses/hostnames reasonable? Can you resolve them and connect to them from every machine in the cluster?

Separately, look at the process log for the Kudu Master. If Impala can't connect to the tservers, the Kudu Master won't be able to either, and you may see errors here relating to creation of tablets.

[#5]

We made the necessay configuration changes as suggested.  I was able to create and populate 2 tables with small size 260K rows.  However, when I tried a table with 5 million rows, I got the same error message I received before:

Status: Kudu error(s) reported, first error: Timed out: Failed to write batch of 13864 ops to tablet 22aba5fde2454409914dcf37d486cb1c after 171 attempt(s): Failed to write to server: (no server available): Write(tablet: 22aba5fde2454409914dcf37d486cb1c, num_ops: 13864, num_attempts: 171) passed its deadline: Network error: Client connection negotiation failed: client connection to 10.91.62.124:7050: connect: No route to host (error 113)

Is there some configuration that we need to change to allow the insertion of multi-million records. Below is the table create statement that was successful, but population failed:

CREATE TABLE IF NOT EXISTS cv00_case_cal_kudu(
    CNTY_NUM STRING,
    CASE_YEAR INT,
    CASE_COURT_TYPE STRING,
    CASE_SEQ_NUM INT,
    TKN STRING,
    LCTN  STRING,
    BEGN_DT  STRING,
    TRIAL_TM  STRING,
    APPL_TKN  STRING,
    LAST_UPD_TS  STRING,
    LAST_UPD_USER  STRING,
   PRIMARY KEY (CNTY_NUM,CASE_YEAR,CASE_COURT_TYPE,CASE_SEQ_NUM) )
PARTITION BY RANGE (CASE_YEAR)
(
    PARTITION VALUES < 1995,
    PARTITION 1995 <= VALUES < 2001,
    PARTITION 2001 <= VALUES < 2006,
    PARTITION 2006 <= VALUES < 2011,
    PARTITION 2011 <= VALUES
)
STORED AS KUDU;

Then the following statement failed:

insert into cv00_case_cal_kudu SELECT * from cv00_case_cal;

with error stated above.

Thanks.

[#6]

That suggests that there is still at least one machine running impala that can't communicate with a kudu server. Are you sure you've fixed the problem for all machines?

If there is at least one machine that still has the problem, that might be the reason why you're still seeing the error.

A small table that has less partitions and/or smaller data that only goes into a few of them only writes to a subset of all kudu tservers, while a big table with more partitions and/or data that goes into all of them might have to write to all of them.

If there is a single machine/resolution that is not "fixed" as per Adar suggestion and not used in the small table/data case might have to be used in the big table/data case.

[#7]

Just to close this thread.  After a bit more investigation, we found that despite the error message received the table was actually populated with the data.

Thanks for your help.