Support Questions

Find answers, ask questions, and share your expertise

How to Make Kylin work with HDP 2.3

avatar

Here is the issue with Kylin (http://kylin.incubator.apache.org/). For now it only supports HDP 2.2.x In HDP 2.3, Hbase has upgraded to 1.0 and APIS have changed. See https://issues.apache.org/jira/browse/KYLIN-892 and https://issues.apache.org/jira/browse/KYLIN-920

Kyle did not keep up to date with the changes and have not migrated to HBase v1.0 APIs. https://issues.apache.org/jira/browse/KYLIN-782

1 ACCEPTED SOLUTION

avatar

The installation and quick-start steps now work with HDP 2.3.2 out of the box. Make sure you download the binary for HBase 1.1.

View solution in original post

19 REPLIES 19

avatar

The github branch that actually works with HDP 2.3 is 1.x-HBase1.xhttps://github.com/apache/incubator-kylin/tree/1.x-HBase1.x

Use the following instructions to build it http://kylin.incubator.apache.org/development/howto_package.html

How to Build Binary Package

###Generate Binary Package

> git clone https://github.com/apache/incubator-kylin kylin

> cd kylin

> ./script/package.sh

In order to generate binary package, maven and npm are pre-requisites.

avatar
Master Mentor

@amcbarnett@hortonworks.com

Did you test that in your environment?

avatar
Master Mentor

@amcbarnett@hortonworks.com

[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project kylin-storage: Too many files with unapproved license: 1 See RAT report in: /root/kylin/storage/target/rat.txt -> [Help 1] [ERROR]

I followed the above steps and hitting this issue

avatar

Just delete the files mentioned in /root/kylin/storage/target/rat.txt

Usually they are log files or files user created/copied

avatar
Master Mentor

@abjawa@hortonworks.com

That helped. I had to delete this files.

rm -f /root/kylin/storage/src/main/java/org/apache/kylin/storage/hbase/coprocessor/endpoint/protobuf/II.proto


Unapproved licenses:


rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query04.sql.expected
rm -f   /root/kylin/query/src/test/resources/query/sql_verifyCount/query07.sql.expected
rm -f   /root/kylin/query/src/test/resources/query/sql_verifyCount/query08.sql.expected
 rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query06.sql.expected
  rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query05.sql.expected

avatar
Master Mentor

I have reached out to Kylin Dev team to comment on this.

I am working on setting up Kylin with HDP 2.3.2. Package.sh failed because of unapproved licensees messages in rat.txt
I deleted following and package.sh moved forward.
rm -f /root/kylin/storage/src/main/java/org/apache/kylin/storage/hbase/coprocessor/endpoint/protobuf/II.proto
Unapproved licenses:
rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query04.sql.expected
rm -f   /root/kylin/query/src/test/resources/query/sql_verifyCount/query07.sql.expected
rm -f   /root/kylin/query/src/test/resources/query/sql_verifyCount/query08.sql.expected
 rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query06.sql.expected
  rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query05.sql.expected
Issues
<strong>Permission
</strong>[root@nsfed01 bin]# ls -l
total 40
-rw-r--r--. 1 root hadoop 1604 Oct 21 03:46 check-env.sh
-rw-r--r--. 1 root hadoop 1300 Oct 21 03:46 find-hbase-dependency.sh
-rw-r--r--. 1 root hadoop 1785 Oct 21 03:46 find-hive-dependency.sh
-rw-r--r--. 1 root hadoop 1023 Oct 21 03:46 get-properties.sh
-rw-r--r--. 1 root hadoop 1638 Oct 21 03:46 health-check.sh
-rw-r--r--. 1 root hadoop 3881 Oct 21 03:46 kylin.sh
-rwxr-xr-x. 1 root hadoop 2321 Oct 21 03:46 metastore.sh
-rw-r--r--. 1 root hadoop 1395 Oct 21 03:46 performance-monitor.sh
-rw-r--r--. 1 root hadoop 1456 Oct 21 03:46 sample.sh
-rwxr-xr-x. 1 root hadoop 2487 Oct 21 03:46 setenv.sh
[root@nsfed01 bin]# 
[root@nsfed01 kylin]# bin/kylin.sh start
KYLIN_HOME is set to /root/kylin/
WARNING: Use "yarn jar" to launch YARN applications.
Logging initialized using configuration in file:/etc/hive/2.3.2.0-2950/0/hive-log4j.properties
Please visit <a href="http://%3Cyour_sandbox_ip%3E:7070/kylin">http://<your_sandbox_ip>:7070/kylin</a> to play with the cubes! (Useranme: ADMIN, Password: KYLIN)
You can check the log at bin/../tomcat/logs/kylin.log
[root@nsfed01 kylin]# bin/kylin.sh: line 62: bin/../tomcat/logs/kylin.log: No such file or directory
mkdir -p tomcat/logs
[root@nsfed01 tomcat]# 
[root@nsfed01 logs]# ls
kylin.log
[root@nsfed01 logs]# cat kylin.log 
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128M; support was removed in 8.0
Not a valid JAR: /root/kylin/tomcat/bin/bootstrap.jar
[root@nsfed01 logs]# 
[root@nsfed01 logs]# ls -l /root/kylin/tomcat/bin/bootstrap.jar
ls: cannot access /root/kylin/tomcat/bin/bootstrap.jar: No such file or directory
[root@nsfed01 logs]# 

avatar

@Neeraj I thought I got it to work. It does not. Did the Kylin Dev Team provide an answer?

avatar
Master Mentor

@Ancil McBarnett

No response from them. I tried again in the sandbox

You can check the log at bin/../tomcat/logs/kylin.log

[root@sandbox kylin]# tail -f tomcat/logs/kylin.log

Not a valid JAR: /root/kylin/tomcat/bin/bootstrap.jar

[root@sandbox kylin]# ls -l /root/kylin/tomcat/bin/bootstrap.jar

ls: cannot access /root/kylin/tomcat/bin/bootstrap.jar: No such file or directory

[root@sandbox kylin]#

avatar
Super Collaborator

A little late to this thread, but late last month, I used the binary build that Kylin says should work with the version of HBASE in HDP 2.3 (apache-kylin-1.2-HBase1.1-incubating-SNAPSHOT). It installed ok, but when creating a cube, I got the following error:

Error: java.lang.NullPointerException at org.apache.kylin.job.hadoop.cube.FactDistinctColumnsMapper.setup(FactDistinctColumnsMapper.java:73)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at 

org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)