Spark Load testing
framework built on a number of distributed technologies, including Gatling, Livy,
Akka, and HDP. Using Akka Server powered by LIVY {Spark as a Service} provides
the following benefits.
REST
friendly and Docker Friendly Low latency
execution Sharing
cache across jobs Separation
of concern Multi
tenancy Direct
Spark SQL execution Configuration
at one place Auditing
and Logging Complete statement
history and metrics Livy Server Livy is an open source REST interface for interacting with
Apache Spark from anywhere. It supports executing snippets of code or programs
in a Spark context that runs locally or in Apache Hadoop YARN. Livy offers three modes to run Spark jobs:
Using programmatic
API Running interactive
statements through REST API Submitting batch
applications with REST API Livy provides the following features:
Interactive Scala,
Python, and R shells Batch submissions in
Scala, Java, Python Multiple users can
share the same server (impersonation support) Can be used for
submitting jobs from anywhere with REST Does not require any
code change to your programs Support Spark1/
Spark2, Scala 2.10/2.11 within one build. Livy provides the
following advantages:
Programmatically
upload jar file and run job. Add additional applications that will connect to
same cluster and upload jar with next job. If you use spark-submit, you must
upload manually JAR file to cluster and run command. Everything must be prepared
before run Use Spark in
interactive mode, hard to do with spark-submit or Thrift Server at scale. Security. Reduce
exposure of the cluster to the outside world. Stability. Spark is
a complex framework and there many factors which can affect its long term
performance and stability. Decoupling Spark context and application allows to
handle Spark issues gracefully, without full downtime of the application. Gatling Server Gatling is a highly capable load testing tool. It is designed
for ease of use, maintainability and high performance. Gatling server provides
the following benefits.
Powerful scripting using Scala Akka + Netty Run multiple scenarios in one
simulation Scenarios = code + DSL Graphical reports with clear
& concise graphs Gatling’s architecture is asynchronous as
long as the underlying protocol, such as HTTP, can be implemented in a non
blocking way. This kind of architecture lets us implement virtual users as
messages instead of dedicated threads, making them very resource cheap. Thus,
running thousands of concurrent virtual users is not an issue. val theScenarioBuilder =
scenario("Interactive Spark Command Scenario Using LIVY Rest Services $sessionId").exec(
/* myRequest1 is a name that describes the request. */
http("Interactive Spark Command Simulation")
.get("/insrun?sessionId=${sessionId}&statement=sparkSession.sql(%22%20select%20event.site_id%20from%20siteexposure_event%20as%20event%20where%20st_intersects(st_makeBBOX(${bbox})%2C%20geom)%20limit%205%20%22).show").check()
).pause(4 second)
So, this is great, we can load
test our spark interactive command with one user! Let’s increase the number of
users. To increase the number of
simulated users, all you have to do is to change the configuration of the
simulation as follows: setUp(
theScenarioBuilder.inject(atOnceUsers(10))
).protocols(theHttpProtocolBuilder) If you want to simulate 3000
users, you might not want them to start at the same time. Indeed, real users
are more likely to connect to your web application gradually. Gatling provides rampUsers to implement this behavior. The
value of the ramp indicates the duration over which the users will be linearly
started. In our scenario let’s have 10 regular users ramp them over 10 seconds
so we don’t hammer the Livy server: setUp(
theScenarioBuilder.inject(rampUsers(10) over (10 seconds)),
).protocols(theHttpProtocolBuilder)
... View more
