Livy is an open source REST interface for interacting with Apache Spark from anywhere. It supports executing snippets of code or programs in a Spark context that runs locally or in Apache Hadoop YARN.
Livy offers three modes to run Spark jobs:
Using programmatic API
Running interactive statements through REST API
Submitting batch applications with REST API
Livy provides the following features:
Interactive Scala, Python, and R shells
Batch submissions in Scala, Java, Python
Multiple users can share the same server (impersonation support)
Can be used for submitting jobs from anywhere with REST
Does not require any code change to your programs
Support Spark1/ Spark2, Scala 2.10/2.11 within one build.
Livy provides the following advantages:
Programmatically upload jar file and run job. Add additional applications that will connect to same cluster and upload jar with next job. If you use spark-submit, you must upload manually JAR file to cluster and run command. Everything must be prepared before run
Use Spark in interactive mode, hard to do with spark-submit or Thrift Server at scale.
Security. Reduce exposure of the cluster to the outside world.
Stability. Spark is a complex framework and there many factors which can affect its long term performance and stability. Decoupling Spark context and application allows to handle Spark issues gracefully, without full downtime of the application.
Gatling is a highly capable load testing tool. It is designed for ease of use, maintainability and high performance. Gatling server provides the following benefits.
Powerful scripting using Scala
Akka + Netty
Run multiple scenarios in one simulation
Scenarios = code + DSL
Graphical reports with clear & concise graphs
Is there a use case already .As of now my thoughts are to implement below,is this possible in clodera ?any suggestions or thoughts?