Reply
Highlighted
New Contributor
Posts: 4
Registered: ‎09-05-2016
Accepted Solution

Kudu API is opening file descriptors until tablet server crash

Hi,

 

I'm using the java kudu API to load data at kudu tables. I see every day that the numbers of file descriptors is getting higher and it doesn't decrease. Today one tablet server has gone down and I can see at logs that there are too many open files. I only do a simple upsert using kudu API and I don't know what I could do to close all these file descriptors.

 

Here the logs I get:

 

+ exec /opt/cloudera/parcels/KUDU-0.9.0-1.kudu0.9.0.p0.17/lib/kudu/sbin/kudu-tserver --tserver_master_addrs=ip-172-31-13-6.us-west-2.compute.internal --flagfile=/run/cloudera-scm-agent/process/124-kudu-KUDU_TSERVER/gflagfile
E0905 07:53:32.007859  2258 tablet.cc:220] T b47b1738b9424a7b874895a373b23a0e Failed to open rowset RowSet(2396): IO error: /data/kudu/tserver/data/ae/4a/6b/0191634652202395: Too many open files (error 24)
E0905 07:53:32.101156  2258 ts_tablet_manager.cc:656] T b47b1738b9424a7b874895a373b23a0e P 58db2d65b8d84a8596f570b31d29cb0c: Tablet failed to bootstrap: IO error: /data/kudu/tserver/data/ae/4a/6b/0191634652202395: Too many open files (error 24)
E0905 07:53:32.107659  2258 tablet.cc:220] T 0e4bf9a90cdb4898aa11b6ccafd45686 Failed to open rowset RowSet(1098): IO error: /data/kudu/tserver/data/6b/b9/f8/0118446477869049: Too many open files (error 24)
E0905 07:53:32.111331  2258 ts_tablet_manager.cc:656] T 0e4bf9a90cdb4898aa11b6ccafd45686 P 58db2d65b8d84a8596f570b31d29cb0c: Tablet failed to bootstrap: IO error: /data/kudu/tserver/data/6b/b9/f8/0118446477869049: Too many open files (error 24)
E0905 07:53:32.117516  2258 tablet.cc:220] T 7aee3fbe79c5419a914d5ed97d4a172d Failed to open rowset RowSet(1021): IO error: /data/kudu/tserver/data/85/12/34/0146313229433016: Too many open files (error 24)
E0905 07:53:32.121172  2258 ts_tablet_manager.cc:656] T 7aee3fbe79c5419a914d5ed97d4a172d P 58db2d65b8d84a8596f570b31d29cb0c: Tablet failed to bootstrap: IO error: /data/kudu/tserver/data/85/12/34/0146313229433016: Too many open files (error 24)
E0905 07:53:32.126981  2258 tablet.cc:220] T f4fd392022f14e7ba03a48fbcc3529d7 Failed to open rowset RowSet(8235): IO error: /data/kudu/tserver/data/7a/27/9f/0134310603747341: Too many open files (error 24)
E0905 07:53:32.130579  2258 ts_tablet_manager.cc:656] T f4fd392022f14e7ba03a48fbcc3529d7 P 58db2d65b8d84a8596f570b31d29cb0c: Tablet failed to bootstrap: IO error: /data/kudu/tserver/data/7a/27/9f/0134310603747341: Too many open files (error 24)
E0905 07:53:32.188652  2258 tablet.cc:220] T 5878e7a8d96e4a3d8aff44a662dbf7d5 Failed to open rowset RowSet(440): IO error: /data/kudu/tserver/data/86/6f/5c/0147812845209473: Too many open files (error 24)
E0905 07:53:32.192005  2258 ts_tablet_manager.cc:656] T 5878e7a8d96e4a3d8aff44a662dbf7d5 P 58db2d65b8d84a8596f570b31d29cb0c: Tablet failed to bootstrap: IO error: /data/kudu/tserver/data/86/6f/5c/0147812845209473: Too many open files (error 24)
E0905 07:53:32.674873  2258 delta_tracker.cc:64] Failed to open UNDO delta file 0094341439967042: IO error: /data/kudu/tserver/data/55/cd/94/0094341439967042: Too many open files (error 24)
E0905 07:53:32.675248  2258 tablet.cc:220] T 63a6a52fbee242a8a82c0bf60ac4463a Failed to open rowset RowSet(947): IO error: /data/kudu/tserver/data/55/cd/94/0094341439967042: Too many open files (error 24)
E0905 07:53:32.678386  2258 ts_tablet_manager.cc:656] T 63a6a52fbee242a8a82c0bf60ac4463a P 58db2d65b8d84a8596f570b31d29cb0c: Tablet failed to bootstrap: IO error: /data/kudu/tserver/data/55/cd/94/0094341439967042: Too many open files (error 24)

  Any suggestion? Thanks

Cloudera Employee
Posts: 43
Registered: ‎09-28-2015

Re: Kudu API is opening file descriptors until tablet server crash

Hi,

It sounds like you are running the Kudu tablet servers with the
'--block_manager=file' option in order to support running on an older file
system. This option is a workaround for small-scale development only,
because it has the issue that you identified: its use of file descriptors
grows quite quickly as you insert data.

I would recommend using the default ("log") block manager to solve your
issue.

-Todd
New Contributor
Posts: 4
Registered: ‎09-05-2016

Re: Kudu API is opening file descriptors until tablet server crash

Hi Todd,

 

Thanks for the answer. Seems that you are right but I have the problem that if I delete this flag I cannot start the tablet server due this error:

+ exec /opt/cloudera/parcels/KUDU-0.9.0-1.kudu0.9.0.p0.17/lib/kudu/sbin/kudu-tserver --tserver_master_addrs=ip-172-31-13-6.us-west-2.compute.internal --flagfile=/run/cloudera-scm-agent/process/131-kudu-KUDU_TSERVER/gflagfile
F0906 07:15:26.361237 11353 tablet_server_main.cc:55] Check failed: _s.ok() Bad status: IO error: Failed to load FS layout: Could not open /data/kudu/tserver/data/block_manager_instance: Wrong block manager type: file
*** Check failure stack trace: ***
    @           0x7cc2cd  google::LogMessage::Fail()
    @           0x7ce33c  google::LogMessage::SendToLog()
    @           0x7cbe29  google::LogMessage::Flush()
    @           0x7ceccf  google::LogMessageFatal::~LogMessageFatal()
    @           0x786840  (unknown)
    @     0x7f1156890f45  __libc_start_main
    @           0x7862e6  (unknown)
    @              (nil)  (unknown)

I'm using ubuntu 14.04 LTS and the filesystem is ext4 theoretically it supports hole punching. I have no idea why it crash

 

Filesystem Type 1K-blocks Used Available Use% Mounted on
udev devtmpfs 16469108 8 16469100 1% /dev
tmpfs tmpfs 3294812 380 3294432 1% /run
/dev/xvda1 ext4 1031978612 96991084 892946324 10% /
none tmpfs 4 0 4 0% /sys/fs/cgroup
none tmpfs 5120 0 5120 0% /run/lock
none tmpfs 16474056 0 16474056 0% /run/shm
none tmpfs 102400 0 102400 0% /run/user
cm_processes tmpfs 16474056 2252 16471804 1% /run/cloudera-scm-agent/process

 

Maybe I'm doing something wrong...

 

Thanks!

Cloudera Employee
Posts: 43
Registered: ‎09-28-2015

Re: Kudu API is opening file descriptors until tablet server crash

Hi,

One possible scenario is that these mount points were previously upgraded
from ext3, and haven't had all ext4 features enabled on them yet. In
particular, we rely on the 'extents' feature of ext4.

You can follow the guide here (see "Converting a non-root filesystem to
ext4"):
https://debian-administration.org/article/643/Migrating_a_live_system_from_ext3_to_ext4_filesystem

-Todd
Cloudera Employee
Posts: 26
Registered: ‎02-05-2016

Re: Kudu API is opening file descriptors until tablet server crash

Unfortunately, once you have on-disk data created with one block manager you can't switch to another, not without blowing away all of the tablet server's data and starting from scratch.

 

New Contributor
Posts: 4
Registered: ‎09-05-2016

Re: Kudu API is opening file descriptors until tablet server crash

Thank you very much for your answers!
Announcements