Apache Flink Tutorials, Examples and HDP Install Guides

[#1]

Hi All,

Does anyone have any steps for installing Flink on HDP? And any tutorials/examples of using Flink? Maybe with Hive ORC tables?

Thanks,

[#2 · Accepted solution]

I just tried this recently. Couple of options:

1. Compilation on HDP 2.3:

curl -o /etc/yum.repos.d/epel-apache-maven.repo https://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo
yum -y install apache-maven-3.2*
git clone https://github.com/apache/flink.git
cd flink
mvn clean install -DskipTests -Dhadoop.version=2.7.1.2.3.2.0-2950 -Pvendor-repos

2. Download prebuilt tarball:

wget http://www.gtlib.gatech.edu/pub/apache/flink/flink-0.9.1/flink-0.9.1-bin-hadoop27.tgz
tar xvzf  flink-0.9.1-bin-hadoop27.tgz
cd flink-0.9.1
export HADOOP_CONF_DIR=/etc/hadoop/conf
./bin/yarn-session.sh -n 1 -jm 768 -tm 1024

3. Experimental Ambari service:

https://github.com/abajwa-hw/ambari-flink-service

Once its up, run word count example

export HADOOP_CONF_DIR=/etc/hadoop/conf
./bin/flink run ./examples/flink-java-examples-0.9.1-WordCount.jar