How to Make Kylin work with HDP 2.3

[#1]

Here is the issue with Kylin (http://kylin.incubator.apache.org/). For now it only supports HDP 2.2.x In HDP 2.3, Hbase has upgraded to 1.0 and APIS have changed. See https://issues.apache.org/jira/browse/KYLIN-892 and https://issues.apache.org/jira/browse/KYLIN-920

Kyle did not keep up to date with the changes and have not migrated to HBase v1.0 APIs. https://issues.apache.org/jira/browse/KYLIN-782

[#2]

The github branch that actually works with HDP 2.3 is 1.x-HBase1.xhttps://github.com/apache/incubator-kylin/tree/1.x-HBase1.x

Use the following instructions to build it http://kylin.incubator.apache.org/development/howto_package.html

How to Build Binary Package

###Generate Binary Package

> git clone https://github.com/apache/incubator-kylin kylin

> cd kylin

> ./script/package.sh

In order to generate binary package, maven and npm are pre-requisites.

[#3]

@amcbarnett@hortonworks.com

Did you test that in your environment?

[#4]

@amcbarnett@hortonworks.com

[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (default) on project kylin-storage: Too many files with unapproved license: 1 See RAT report in: /root/kylin/storage/target/rat.txt -> [Help 1] [ERROR]

I followed the above steps and hitting this issue

[#5]

Just delete the files mentioned in /root/kylin/storage/target/rat.txt

Usually they are log files or files user created/copied

[#6]

@abjawa@hortonworks.com

That helped. I had to delete this files.

rm -f /root/kylin/storage/src/main/java/org/apache/kylin/storage/hbase/coprocessor/endpoint/protobuf/II.proto


Unapproved licenses:


rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query04.sql.expected
rm -f   /root/kylin/query/src/test/resources/query/sql_verifyCount/query07.sql.expected
rm -f   /root/kylin/query/src/test/resources/query/sql_verifyCount/query08.sql.expected
 rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query06.sql.expected
  rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query05.sql.expected

[#7]

I have reached out to Kylin Dev team to comment on this.

I am working on setting up Kylin with HDP 2.3.2. Package.sh failed because of unapproved licensees messages in rat.txt
I deleted following and package.sh moved forward.
rm -f /root/kylin/storage/src/main/java/org/apache/kylin/storage/hbase/coprocessor/endpoint/protobuf/II.proto
Unapproved licenses:
rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query04.sql.expected
rm -f   /root/kylin/query/src/test/resources/query/sql_verifyCount/query07.sql.expected
rm -f   /root/kylin/query/src/test/resources/query/sql_verifyCount/query08.sql.expected
 rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query06.sql.expected
  rm -f  /root/kylin/query/src/test/resources/query/sql_verifyCount/query05.sql.expected
Issues
<strong>Permission
</strong>[root@nsfed01 bin]# ls -l
total 40
-rw-r--r--. 1 root hadoop 1604 Oct 21 03:46 check-env.sh
-rw-r--r--. 1 root hadoop 1300 Oct 21 03:46 find-hbase-dependency.sh
-rw-r--r--. 1 root hadoop 1785 Oct 21 03:46 find-hive-dependency.sh
-rw-r--r--. 1 root hadoop 1023 Oct 21 03:46 get-properties.sh
-rw-r--r--. 1 root hadoop 1638 Oct 21 03:46 health-check.sh
-rw-r--r--. 1 root hadoop 3881 Oct 21 03:46 kylin.sh
-rwxr-xr-x. 1 root hadoop 2321 Oct 21 03:46 metastore.sh
-rw-r--r--. 1 root hadoop 1395 Oct 21 03:46 performance-monitor.sh
-rw-r--r--. 1 root hadoop 1456 Oct 21 03:46 sample.sh
-rwxr-xr-x. 1 root hadoop 2487 Oct 21 03:46 setenv.sh
[root@nsfed01 bin]# 
[root@nsfed01 kylin]# bin/kylin.sh start
KYLIN_HOME is set to /root/kylin/
WARNING: Use "yarn jar" to launch YARN applications.
Logging initialized using configuration in file:/etc/hive/2.3.2.0-2950/0/hive-log4j.properties
Please visit <a href="http://%3Cyour_sandbox_ip%3E:7070/kylin">http://<your_sandbox_ip>:7070/kylin</a> to play with the cubes! (Useranme: ADMIN, Password: KYLIN)
You can check the log at bin/../tomcat/logs/kylin.log
[root@nsfed01 kylin]# bin/kylin.sh: line 62: bin/../tomcat/logs/kylin.log: No such file or directory
mkdir -p tomcat/logs
[root@nsfed01 tomcat]# 
[root@nsfed01 logs]# ls
kylin.log
[root@nsfed01 logs]# cat kylin.log 
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=128M; support was removed in 8.0
Not a valid JAR: /root/kylin/tomcat/bin/bootstrap.jar
[root@nsfed01 logs]# 
[root@nsfed01 logs]# ls -l /root/kylin/tomcat/bin/bootstrap.jar
ls: cannot access /root/kylin/tomcat/bin/bootstrap.jar: No such file or directory
[root@nsfed01 logs]# 

[#8]

@Neeraj I thought I got it to work. It does not. Did the Kylin Dev Team provide an answer?

[#9]

@Ancil McBarnett

No response from them. I tried again in the sandbox

You can check the log at bin/../tomcat/logs/kylin.log

[root@sandbox kylin]# tail -f tomcat/logs/kylin.log

Not a valid JAR: /root/kylin/tomcat/bin/bootstrap.jar

[root@sandbox kylin]# ls -l /root/kylin/tomcat/bin/bootstrap.jar

ls: cannot access /root/kylin/tomcat/bin/bootstrap.jar: No such file or directory

[root@sandbox kylin]#

[#10]

A little late to this thread, but late last month, I used the binary build that Kylin says should work with the version of HBASE in HDP 2.3 (apache-kylin-1.2-HBase1.1-incubating-SNAPSHOT). It installed ok, but when creating a cube, I got the following error:

Error: java.lang.NullPointerException at org.apache.kylin.job.hadoop.cube.FactDistinctColumnsMapper.setup(FactDistinctColumnsMapper.java:73)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) at 

org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)