About khaslbeck

khaslbeck · ‎05-12-2016

Great feature @bbende . This is much easier to visualize now.

khaslbeck · ‎05-12-2016

If the case where NiFi is reading from 30 database tables in a single flow what is the best way to visually identify which processor is connecting to each database and table?

khaslbeck · ‎05-11-2016

You can now visualize any Zeppelin notebook using Zeppelinhub viewer. https://www.zeppelinhub.com/viewer personal likes: 1. No need to sign up or register just paste a link 2. I've been posting my zeppelin notebooks to github but everyone that wants to visualize them or interact with them needs to download, move to environment, import into their instance of zeppelin. Not anymore just paste the link. 3. Less of a need to take screenshots and create a powerpoint just send the hyperlink examples: Stock Variance Notebook github - https://github.com/kirkhas/zeppelin-notebooks/blob/master/stock-variance/note.json vs Stock Variance Notebook zephub - https://www.zeppelinhub.com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tpcmtoYXMvemVwcGVsaW4tbm90ZWJvb2tzL21hc3Rlci9zdG9jay12YXJpYW5jZS9ub3RlLmpzb24 Credit Card Fraud Transactions git - https://github.com/vakshorton/CreditCardTransactionMonitor/blob/master/Zeppelin/notebook/2BGDWYZV9/note.json vs https://www.zeppelinhub.com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL3Zha3Nob3J0b24vQ3JlZGl0Q2FyZFRyYW5zYWN0aW9uTW9uaXRvci9tYXN0ZXIvWmVwcGVsaW4vbm90ZWJvb2svMkJHRFdZWlY5L25vdGUuanNvbg

khaslbeck · ‎05-11-2016

In addition to the HWX install guides online, this is a great best practices article for groups that want to consider some design options prior to install. http://hortonworks.com/blog/best-practices-in-hdfs-authorization-with-apache-ranger/

khaslbeck · ‎05-11-2016

Repo Description Zeppelin Notebook Stock Variance Example check it out on zeppelinhubview https://www.zeppelinhub.com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL2tpcmtoYXMvemVwcGVsaW4tbm90ZWJvb2tzL21hc3Rlci9zdG9jay12YXJpYW5jZS9ub3RlLmpzb24 Repo Info Github Repo URL https://github.com/kirkhas/zeppelin-notebooks Github account name kirkhas Repo name zeppelin-notebooks

khaslbeck · ‎04-30-2016

Vadim and I at hortonworks have a packaged demo of exactly what I described above on github. data streaming through NiFi into a spark model at this link https://github.com/vakshorton

khaslbeck · ‎04-30-2016

The more traditional approach in this situation is to use NiFi to read the incoming data and then add a NiFi processor to dump the data from the NiFi queue to either Storm or in your case SparkStreaming. Now you can build a Spark ML model test it and run it in Spark. You can push logic into NiFi but an ML model inside NiFi is overkill.

khaslbeck · ‎04-27-2016

In the latest version of Hortonworks sandbox 2.4 (you can download for free from hortonworks.com) zeppelin and spark run out of the box. Spark version is 1.6 and the toJSON method works, make sure you run it on a DataFrame not RDD val js = tradesRDD.toDF.toJSON js.take(2) output: Array[String] = Array({"trader":"Kirk","price":11.0,"qty":51,"vol":40000,"product":"goog","time":"2016-03-29 10:38:12.0"}, {"trader":"Kirk","price":0.0,"qty":66,"vol":40000,"product":"goog","time":"2016-03-29 10:56:12.0"})

khaslbeck · ‎04-27-2016

This is the common process many go through and many ways to skin the cat here. I prefer the below methodology. 1. Bring in the data with minimal transformation the "E" and "L". Depending on workload this could be sqoop for simple batch or NiFi for a more modern streaming approach with better control over flow, bi-direction and back pressure. 2. Decide on a transformation strategy and store a higher level or "enriched" data set typically in Hive or HBase. Now between Atlas and NiFi you should have some data lineage. Other formatting might take place here with native datatypes dates vs timestamps. Likely a partitioning strategy would take place here. Running a data cleansing strategy at this phase is also a good idea as well as computing feature vectors. 3. Use zeppelin + spark to analyze the data.

khaslbeck · ‎04-27-2016

After playing with the Spark 1.6 LinearRegression model I found it is very sensitive to the StepSize. What is the best practice around tuning this parameter? The mean squared error of the model I build varies greatly depending on this input. // Building the model val numIterations = 30 val stepSize = 0.0001 val linearModel = LinearRegressionWithSGD.train(trainingDataRDD, numIterations, stepSize) // Evaluate model on training examples and compute training error val valuesAndPreds = trainingDataRDD .map { point => val prediction = linearModel.predict(point.features) (point.label, prediction) } val MSE = valuesAndPreds.map{case(v, p) => math.pow((v - p), 2)}.mean() println("training Mean Squared Error = " + NumberFormat.getInstance().format(MSE) )

Online	Offline
Last Visited	‎08-02-2018 08:10 PM

Member Since	‎02-23-2016 02:08 AM
Last Visited	‎08-02-2018 08:10 PM
Posts	51
Kudos received	95

Cloudera Community

Cloudera Community

Re: Remove Zeppelin from Ambari then Add it back

Re: HAWQ to HIVE data type mapping

Re: Storage data in HDFS - What's next?

Re: HDP-2.3.4.0-3485 upgrade failed due to HDP-UTI...

Re: What is the best way to organize many NiFi Pro...

What is the best way to organize many NiFi Process...

Zeppelinhub Viewer

Best Practices In HDFS Authorization with Apache R...

Stock Variance using Zeppelin

Re: Machine Learning in Apache Nifi

Re: Machine Learning in Apache Nifi

Re: toJSON is not a member of org.apache.spark.sql...

Re: Storage data in HDFS - What's next?

How to find the best StepSize in a Spark ML Linear...