question Deep Learning on HDP in Support Questions

Deep Learning on HDP

TimothySpann — Sat, 11 Jun 2016 01:29:04 GMT

Has anyone gotten TensorFlow or DeepLearning4J running on top of HDP 2.4?

I am interested in the setup.

Re: Deep Learning on HDP

abajwa — Sat, 11 Jun 2016 01:33:28 GMT

@Timothy Spann see webinar @Dhruv Kumar ran last year using older version of HDP

http://hortonworks.com/partners/learn/#spark813

Re: Deep Learning on HDP

TimothySpann — Sat, 11 Jun 2016 20:16:33 GMT

Cool.

http://www.slideshare.net/hortonworks/deep-learning-with-hortonworks-and-apache-spark-hortonworks-technical-workshop

Deep Learning with Microsoft

http://www.slideshare.net/mlprague/xuedong-huang-deep-learning-and-intelligent-applications

Deep Learning 4J

http://www.slideshare.net/agibsonccc/brief-introduction-to-distributed-deep-learning

Re: Deep Learning on HDP

ymoiseev — Mon, 20 Jun 2016 13:33:03 GMT

@Timothy Spann If it's still relevant, there is a project called TensorFrames that is pretty much a wrapper for Spark's DataFrames: https://github.com/tjhunter/tensorframes. Although I am not sure if it is being actively developed, last time I checked it looked functional.

Re: Deep Learning on HDP

chris3 — Tue, 20 Dec 2016 05:01:16 GMT

Hi Tim - I know this is an old question and we've chatted offline. I just wanted to have it on the record here that Deeplearning4j is just a Hadoop job that can be submitted as a JAR file. It's that simple.

https://github.com/deeplearning4j/

Re: Deep Learning on HDP

TimothySpann — Tue, 20 Dec 2016 05:14:37 GMT

Sweet. This will also work with HDP 2.5.

Re: Deep Learning on HDP

sedatkestepe — Thu, 25 May 2017 23:24:44 GMT

Hello,

I have the same exact question but I couldn't link or associate listed variations of deep learning applications with the use of HDP.

How am I supposed to run any of them on my Hadoop cluster? In which practical, infrastructural way do DL applications and Hadoop applications get related?

Thanks in advance...

Re: Deep Learning on HDP

TimothySpann — Thu, 25 May 2017 23:52:50 GMT

in HDP 3.0, YARN 3 will run TensorFlow jobs in docker containers.

Tensorflow on Spark

Caffe on Spark

both run on Spark on YARN and will be distributed in your Hadoop cluster

For many applications I am running tensorflow python scripts (already trainined) on NiFi nodes, but could run on HDP nodes which all have python installed.

Other option is to run TensorFlow with TensorFlow Serving and call it from existing applications, NiFi and big data apps via gRPC

Re: Deep Learning on HDP

TimothySpann — Thu, 25 May 2017 23:59:53 GMT

https://dzone.com/articles/deep-learning-on-big-data-platforms

https://dzone.com/articles/tensorflow-on-the-edge-part-2-of-5

https://dzone.com/articles/deep-learning-for-data-engineers-part-1

https://dzone.com/articles/deep-learning-and-machine-learning-guide-part-iii

https://dzone.com/articles/deep-learning-and-machine-learning-guide-part-ii

https://dzone.com/articles/machine-learning-resources

https://dzone.com/articles/tensorflow-on-the-edge

TensorFlow, MXNet and Deep Learning4J all work well with HDP 2.6

https://community.hortonworks.com/articles/83872/data-lake-30-containerization-erasure-coding-gpu-p.html

https://community.hortonworks.com/articles/104649/using-cloudbreak-recipes-to-deploy-anaconda-and-te.html

https://community.hortonworks.com/articles/58265/analyzing-images-in-hdf-20-using-tensorflow.html

https://community.hortonworks.com/articles/54954/setting-up-gpu-enabled-tensorflow-to-work-with-zep.html

https://community.hortonworks.com/articles/80339/iot-capturing-photos-and-analyzing-the-image-with.html

https://community.hortonworks.com/articles/59349/hdf-20-flow-for-ingesting-real-time-tweets-from-st.html

https://community.hortonworks.com/articles/103863/using-an-asus-tinkerboard-with-tensorflow-and-pyth.html

https://community.hortonworks.com/articles/83100/deep-learning-iot-workflows-with-raspberry-pi-mqtt.html