Support Questions
Find answers, ask questions, and share your expertise
Announcements
Alert: Welcome to the Unified Cloudera Community. Former HCC members be sure to read and learn how to activate your account here.

LLAP IO cache not working

LLAP IO cache not working

New Contributor

I'm trying to set up Hive LLAP with IO cache (it's working fine without). I'm using ORC data, my memory settings are fine, and LLAP/IO-specific Hive configurations are:

--hiveconf hive.execution.mode=llap

--hiveconf hive.llap.execution.mode=all

--hiveconf hive.llap.io.enabled=true

--hiveconf hive.llap.daemon.service.hosts=@llap0

--hiveconf hive.llap.io.memory.mode=cache

When trying to print the LLAP IO summary metrics at the end of a query, all the values are coming as 0, indicating that LlapIOCounters are not getting registered and OrcEncodedDataReader is not getting called (see attached screenshot).

Any ideas what I might be missing?

80441-screen-shot-2018-07-09-at-30150-pm.png

4 REPLIES 4

Re: LLAP IO cache not working

Contributor

It could be counters not being propagated correctly; usually when LLAP IO is not used, this counter section is not displayed at all. You might want to check LLAP logs to see if threads with names beginning with "IO-Elevator"; these are IO threads. Also check that LLAP daemon logs have IO initialized correctly (in the beginning when the daemon starts, there should be lines about cache size, eviction policy, etc. Then, HS2 and LLAP logs may have lines from "wrapForLlap" method in HiveInputFormat indicating errors when trying to use LLAP IO. Also, is this an ACID table? ACID tables are not able to use LLAP cache until Hive 3.1 in Apache, or HDP 3.X

Re: LLAP IO cache not working

New Contributor

Thanks for your reply. ACID is disabled. This is the log message I'm getting in HS2 logs:

io.HiveInputFormat: Not using llap for org.apache.hadoop.mapred.SequenceFileInputFormat@5c6e83c6: supported = false, vectorized = false

These are the commands I ran to ensure my sample data is created using ORC:

SET hive.default.fileformat=ORC; 

SET hive.default.fileformat.managed=ORC; 

create database tmp;

use tmp;

create table test (id int);

insert into test (id) values (1);

select count(*) from test;

What am I doing wrong here?

Re: LLAP IO cache not working

Contributor

Doesn't look like the default format is kicking in. Can you create the table as ORC explicitly (stored as ORC)?

Also depending on what version this is, vectorization may need to be enabled; it's good for performance anyway. Or hive.llap.io.row.wrapper.enabled needs to be set to true (if present in your version).

Re: LLAP IO cache not working

New Contributor

Did try creating the table as ORC explicitly but still facing the same issue. What input format should it be ideally picking up?

Don't have an account?
Coming from Hortonworks? Activate your account here