Deep Learning on HDP

[#1]

Has anyone gotten TensorFlow or DeepLearning4J running on top of HDP 2.4?

I am interested in the setup.

[#2]

@Timothy Spann see webinar @Dhruv Kumar ran last year using older version of HDP

http://hortonworks.com/partners/learn/#spark813

[#3 · Accepted solution]

Cool.

http://www.slideshare.net/hortonworks/deep-learning-with-hortonworks-and-apache-spark-hortonworks-te...

Deep Learning with Microsoft

http://www.slideshare.net/mlprague/xuedong-huang-deep-learning-and-intelligent-applications

Deep Learning 4J

http://www.slideshare.net/agibsonccc/brief-introduction-to-distributed-deep-learning

[#4]

@Timothy Spann If it's still relevant, there is a project called TensorFrames that is pretty much a wrapper for Spark's DataFrames: https://github.com/tjhunter/tensorframes. Although I am not sure if it is being actively developed, last time I checked it looked functional.

[#5]

Hi Tim - I know this is an old question and we've chatted offline. I just wanted to have it on the record here that Deeplearning4j is just a Hadoop job that can be submitted as a JAR file. It's that simple.

https://github.com/deeplearning4j/

[#6]

Sweet. This will also work with HDP 2.5.

[#7]

Hello,

I have the same exact question but I couldn't link or associate listed variations of deep learning applications with the use of HDP.

How am I supposed to run any of them on my Hadoop cluster? In which practical, infrastructural way do DL applications and Hadoop applications get related?

Thanks in advance...

[#8]

in HDP 3.0, YARN 3 will run TensorFlow jobs in docker containers.

Tensorflow on Spark

Caffe on Spark

both run on Spark on YARN and will be distributed in your Hadoop cluster

For many applications I am running tensorflow python scripts (already trainined) on NiFi nodes, but could run on HDP nodes which all have python installed.

Other option is to run TensorFlow with TensorFlow Serving and call it from existing applications, NiFi and big data apps via gRPC

[#9]

https://dzone.com/articles/deep-learning-on-big-data-platforms

https://dzone.com/articles/tensorflow-on-the-edge-part-2-of-5

https://dzone.com/articles/deep-learning-for-data-engineers-part-1

https://dzone.com/articles/deep-learning-and-machine-learning-guide-part-iii

https://dzone.com/articles/deep-learning-and-machine-learning-guide-part-ii

https://dzone.com/articles/machine-learning-resources

https://dzone.com/articles/tensorflow-on-the-edge

TensorFlow, MXNet and Deep Learning4J all work well with HDP 2.6

https://community.hortonworks.com/articles/83872/data-lake-30-containerization-erasure-coding-gpu-p....

https://community.hortonworks.com/articles/104649/using-cloudbreak-recipes-to-deploy-anaconda-and-te...

https://community.hortonworks.com/articles/58265/analyzing-images-in-hdf-20-using-tensorflow.html

https://community.hortonworks.com/articles/54954/setting-up-gpu-enabled-tensorflow-to-work-with-zep....

https://community.hortonworks.com/articles/80339/iot-capturing-photos-and-analyzing-the-image-with.h...

https://community.hortonworks.com/articles/59349/hdf-20-flow-for-ingesting-real-time-tweets-from-st....

https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth...

https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt....