In a detailed demonstration, the vice president of engineering at Soasta explains the three major components of the company's new cloud application testing tool, CloudTest.
And for more on CloudTest, take a look at our interview with Soasta's CEO Tom Lounibos. In it, Tom discusses how CloudTest will help Verizon's website predict any iPhone-related traffic overloads.
Read the full text transcript from this video below. Please note the full transcript is for reference only and may include limited inaccuracies. To suggest a transcript correction, contact firstname.lastname@example.org.
Demo: Check out Soasta's cloud application testing tool
Jo Maitland: Hi, I'm Jo Maitland. I'm here with Tal Broda, who is the VP of
Engineering for Soasta. He's going to take us through the cloud test
product the company is announcing this week. It's a particularly exciting
application, as the market for cloud computing takes off. Lots of companies
want to make sure that their application scales to the user requirements
out there, so this is an interesting product. Anyway, Tal, take it away.
Tal Broda: OK, cool. Thank you. So basically what I'm going to walk you through
is the three major components that we deliver with this new offering. In
Cloud Test, what you can do is build the test, then you can execute the
test, then while you're executing the test you can basically do performance
intelligence--analyze the real-time results and monitor the testing
environment where they're using our monitors or the customer's monitors.
And bring it all together into one coherent view to really analyze in real-
time and find the root cause of your performance problem so you can go
ahead and fix it and continue to iterate and climb the performance
mountain, if you will, until you get to your user level.
So we'll start, as the product progresses, kind of in a linear manner,
we'll start with how you build the test. So I'm not going to take you
through how you record the test, but you basically record using a proxy-
based approach. You can record and use the product from any browser. What
There's no flash, no downloads, no installs, and the entire product is
delivered over the web, like a modern-day web application.
Jo Maitland: Cool.
Tal Broda: And so, after you record, basically you interact with the website
through a browser--and you can use any browser, IE, Chrome, Firefox, or
Safari--and when you do the recording, you basically get a kind of
immutable recording of what you did in the test. So you can see all of the
messages or present each of them in this little icon here. You can see some
messages are HTML, some messages are just images that you get from your
website. In this case, I was doing a recording of...
Jo Maitland: Messages meaning, the application is telling me something about what's
going on with the test?
Tal Broda: It's actually simpler than that. And when you interact with a webpage--
let's say, I went to google.com--and so you will see that I'm getting back
an HMTL page, and that would be a request response. And then I'm asking for
the image here, and then that's a request response, and so on and so forth.
So each message that we represent here is basically just a simple request
response that we did with the website. You can see that some of them are
CSS for the styles, and so on. So you can browse through the recording and
look at different messages.
You can see that this specific website was actually selling steaks, so it's
kind of cool--you can look at the pictures of the steaks that they offer on
the website, and so on and so forth. You can really understand what's going
on with the different messages, because you can see the request, you can
see the response, and you can see the header and body of both of them. So
as a tester, it's very important to me to understand exactly what I'm
recording, and if it's really what I'm wanting.
Jo Maitland: And the color coding here--so the green ones are always HTML.
Tal Broda: Exactly. So it's just to make it easy on the user to find it. Some
people like to look at it as an icon view, and some people like to see more
of a list manner, so we offer both, and both of them are very deep and you
can really go into details about the recording. But the recording is not
something you can mutate. It's there; it's just a fixed representation of
what you did and how you interacted with the website.
To actually create a test, you need to take this recording and convert it
into what we call a test clip. So you do that through this very simple
wizard that we have here, and you go ahead and you give the clip a name,
and you find all of the targets that you might want to use. You insert
think times. So as you were browsing through the website, maybe you stopped
on certain web pages and read them, or interacted with them, so those would
be the think times. So basically, we capture exactly the think times that
you did in the recording to create a real virtual user representation.
Because we want to test the website as if it's a real user, multiplied by
one thousand, ten thousand, a million, and so on.
Then we understand all of the targets that you might be testing against,
and you can see that even though this is a simple website, it actually has
a lot of different targets, and this is very typical for a modern-day web
application. A lot of web applications are using CVNs and offload all of
from their website. So each website will have some local targets, and some
targets that are on the outside. And people like to know about that,
because sometimes they want to test, and not actually exercise the CVNs,
they only want to test their infrastructure. Does that make sense?
Jo Maitland: Mm-hmm. Absolutely.
Tal Broda: And so, I'm creating the new targets that I want, and then we actually
have a way to automatically group all of those messages into logical web
pages. So each web page would have the initial HTML, kind of like when you
see on google.com, there's the HTML, and then all of the images and
And so, you can do that automatically, via just checking the check box for
creating pages, and you can even say if you want dynamic pages or static
pages, but I'm not going to go into that in this simple demo.
Then I go ahead and I create the clip, and you can see we get this little
status bar indicating, and then we can go directly into editing this clip
in more detail. So now we are in what we call the clip editor. And you can
see that, over here, again we took this icon-based approach, and now you
can see that each one of these icons is representing a page. We have some
resources that did not belong to the page--they were outside of the page--
and we can choose to move them into a page or just completely delete them
from the clip. And we can see the delays between each page, if we want to
This is the place where we can kind of start editing. So we can open a
page, and we can see the HTML document of the page, and we can see all of
the resources of the page. And you can see that this page actually has a
lot of resources. So as we scroll, we will gather all of the resources and
show them to you and you can manipulate all of this content and move things
around and edit them, and do everything that you need to do to get the
server to believe that each one of our virtual users is actually a real
entity in the system, including getting session IDs and sending back
session IDs and all of those things are done in the clip editor. You can
open... Yeah, go ahead.
Jo Maitland: Can you show us an alert, like where somebody would find there was a
problem with their system?
Tal Broda: Yeah, as I will run the test, you will see all of the real-time
feedback, but this is just the editing environment, where I can take a
message and move it out, and move it to another page, and do everything
that I want to do in editing the clip. So we're not going to edit this,
because I've got some compositions ready for you, but this would be the
editor to do that. And again, it's a very visual editor, and you can see
that you can interact with things via drag and drop, and the nice thing
about all of our test creation is that it's done in a completely visual
way, whereas if you go to a traditional testing tool you will see that
people require you to know how to write C code.
Jo Maitland: Such as?
Tal Broda: I don't want to name names, but most of the...
Jo Maitland: Oh, go on. Be brave.
Tal Broda: If you compared us to Lode Runner, if you compared us to a bunch of
different tools, they were created in the mid 90s, or maybe before that,
and the default thing to do then was to develop some kind of integrated
development environment, and you have to write your code in C, or maybe,
for more advanced libraries, on top of C. You have to be a developer to
write the code. In this environment, you're just looking at icons and
moving things around, and you don't need to write any code at all. You can,
if you want. We have an ability for you to add scripts into the clip and
So we have the ability for people to code, but most of the time they don't,
and they really edit it in a visual way. And it's nice because they can
also show the customer exactly the requirements that they wanted in a
visual way, so the customer doesn't need to be a developer, as well. Does
that make sense?
Jo Maitland: Mm-hmm.
Tal Broda: So that's the clip. Once you have a clip, you can take it and you can
create a composition from that clip, and I'll show you what that means. A
composition is basically a collection of clips that can be laid out on a
visual timeline. So essentially, the whole metaphor that we have is on
media editing, so you're almost thinking about like you're mixing music, or
creating a video. It may be familiar to the guy who's filming us here. So
basically, you lay out things on tracks, and you lay them in a visual way,
and I can duplicate this track just to show you. And I can take any message
that I have here, and I can basically just lay it out visually on a
timeline. And once I lay it down, basically we are going to execute all of
this visually and move on with the test, and we can see the results in real-
time. And this edit...
Jo Maitland: Sorry, so that clip that you just put there, would be things that
you've selected from the website that you want to...
Tal Broda: It would be this clip, for example.
Jo Maitland: Right.
Tal Broda: All of the complexity that we see here, about which pages we had,
which messages we had, the delays, the scripts, everything that we did for
a virtual user scenario--logging into a web site, and buying something, and
logging out--that would be a clip. And then we can take those and lay them
out visually on a timeline here to really create the test mix that we want
to have. And you can see that here it's very simple--I have 50% of my user
population coming from this track, and 50% coming from this track. And we
can define that Track 1 would be played from Europe, and Track 2 would be
played from the U. S.. Tracks are really the geographical location of where
you're going to be creating load, but there also can be the complex user
scenarios. So you're really not limited in how much complexity you can
build into this thing, and then for how many locations you can play them.
Does that make sense?
Jo Maitland: Yes, it does, yeah.
Tal Broda: OK, so, I don't want to save these changes. I'm just going to open
this composition again so I can play that for you. And before we go ahead
and play the composition, we have to understand where we're going to deploy
it from. What we've done so far is just test creation. Before we go ahead
and execute the test, we need to think about where we want to execute it
from. Now if I just wanted to execute it from this machine here, I can do
that, just click play, and that would be possible for anybody who's buying
this new appliance. They can do tests behind a firewall, and just execute
the test against infrastructure that's behind the firewall.
If they want to go ahead and execute the test from the cloud, or from many
cloud providers, many points of presence around the world, what they can do
is they can go ahead and deploy what we call a grid. And the grid is
basically collections of load servers and analytic servers that we would
deploy as a unit across the world in many cloud providers, and they would
get an assignment. Basically, each one of those load servers would get a
track to play in real time. And those tracks would be played, results would
be collected in real-time, be reported back to those results servers, and
then collected all together into one in-memory data store, from which we
will report on it in real-time in the dashboard.
Jo Maitland: So if I'm the customer, do I have to have relationships already with
AWS, with IBM testing, Dev Cloud?
Tal Broda: No, so you have two options. One option that we have today is that
basically we can run the test for you and we can use our EC2 account and
Microsoft Azure account, but in this new version that we will ship, there
will be an option for you to basically enter your credentials into Cloud
Test, and then you don't need to go to any other tool to deploy those
servers, check that they're healthy, make sure that Cloud Test is deployed
on them and that your test is deployed on them. We take care of all of that
complexity, and believe me it's pretty complex. Because each cloud provider
API is different, because sometimes servers are up, and sometimes they're
down, which is really the reality in cloud computing that nobody really
wants to talk about, but it is a reality.
Jo Maitland: It's true.
Tal Broda: So we solved that problem. In this new version, people would be able
to use their own accounts, and get billed directly from the cloud
providers, so really we are giving the people the ability to do everything
that I am doing right now.
Before I can run this, let's say that I want to actually run it from the
cloud, and I go into what we call the grids UI, and I can see that I have a
grid that is defined here, and I can go ahead and launch it, but I just
wanted to show you a little bit about how you actually build a grid. So the
first thing, you just give it a name and save it in a specific location, so
once you save this grid, you can actually go and reuse it later. So you
don't need to understand all of the definition. One way of building a grid
is just basically selecting a test composition, which is what I just
defined before, that complex matrix, and the grid would tell you where you
need servers. It will tell you, "Hey, you have all of these tracks, you
wanted to play them from this location, this location, and this location,
so you will need 100 servers in each location" and you're done.
I want to take you through the process of how you do it. So, as you can
see, we have different locations around the world. On my system it's pretty
simplistic. I just have EC2 Europe and EC2 U. S., but we support Microsoft
Azure, we support GoldGrid, IBM Cloud, and really you can define all of
those through the same UI, even though you are really interacting--or Cloud
Test, behind the scenes--is interacting with all these different APIs. You
can just go to this UI and say, "Look, here is where I want servers in five
.locations" and you don't need to know about the complexities of
negotiating getting those servers, making sure they're up and running,
correcting for errors that can happen--because errors happen--and making
sure that the environment is up and running so that you can actually start
executing this test.
Jo Maitland: What happens if I want service in a place where you guys don't have it?
Tal Broda: You can actually manually get servers there, deploy your load servers
there, and then you can go to the server's UI here and basically tell the
system, "Here are my servers, they are there, and I would like you to use
them for this test." And in the composition editor, you will see them
basically at this location drop-down here. So much like you see the load
servers in the U. S. and Europe, you will see all your local load servers,
or your manually provisioned, if you will, load servers here. So you can do
that as well. Once I defined all of these locations, I can say, "In the U.
S. I would like to have ten load servers, and I would want to have one
analytic server for each ten load servers." And really, the ratio that we
support is 50:1 or 100:1 because a lot of our analytics are actually done
in the load server where the data is changing in real-time, and we filed a
couple of patents on that, as well. And then you can get another ten
servers in Europe, for example, another analytic server that would be
working with them together.
Jo Maitland: Why so conservative? Could we not put a thousand in there, or...
Tal Broda: We could, it's just a demo, and we do tape, when we allocate them.
Jo Maitland: Fair enough.
Tal Broda: I'm just going to go kind of small, and...
Jo Maitland: We have a budget for this?
Tal Broda: I'm not going to run a big load test. It's not very costly for us, but
again, also, if we did more servers, then the chance of us getting errors
from cloud providers is higher, as well. But I will tell you that the grid
UI is completely multi-threaded, so the time it will take me to get five
servers, if the cloud provider's APIs are healthy, it's the same time it
would take me to get you 5000 servers, because we do all of those calls to
all of the cloud providers in parallel, so it's not sequential. Let's go
and actually deploy this grid, so we can see a little summary here of the
different locations that we have, and the different servers that we will
get in each location. We will get ten test servers in EC2 U. S. and another
ten in EC2 Europe, and one results server, or analytic server, in each one
And then, very simply, I can just go and deploy instances, and we can see
that we started making calls, like I said, in parallel, to all of these
cloud providers. We can see those blue lines that are progressing are
basically telling me that we got the servers, and the green check lines
that kind of chase them tell me that we went ahead and checked that Cloud
Test is deployed on them, and it is running, and it is ready to run the
test for you. So once all of the green finishes all the way to the end, and
we can see that we actually got some errors and we are correcting them for
you in real time, which is something you will need to do manually, and if
you need to do that manually on 5000 servers, you can imagine that that can
take you a very, very long time.
Jo Maitland: When you say that you guys are correcting it in real-time, what do you
mean? What are you doing?
Tal Broda: The product understands that we got an error, it captures that
exception, and throughout the years that we have worked with cloud
providers, we came up with best practices for manually deploying this
grids, because that's what we did two and a half years ago. We used to come
to this room, five guys, before each test, and manually get a grid up and
running, and it used to take hours. It was very, very painful, and so we
took all of this knowledge that we had and completely automated it in the
system. And so every error, we used to have a best practice solution for
it, and now the product knows how to do it. You can see that once we finish
deploying the grid, we get a big check mark that the grid is ready.
Jo Maitland: But Tal, what I mean is, specifically, what is being corrected there?
So could it be there's a load balancer problem, launch of a different load
balancer? Like, give me an example of...
Tal Broda: An example is that we launch the server, and the cloud provider told
us, "Yes, you can go ahead and use this server" and we tried to make a call
to deploy our product into the server, and then we got an exception that
really tells us that the server is not running. This happens.
Jo Maitland: As simple as that? There isn't other permutations of that, or-
Tal Broda: There are many, many, many permutations. For deploying servers we
catch about 18 different exceptions, and for checking servers, maybe 20
more. There's just a lot of different things that can happen in different
stages of the product, and you can see that the grid wizard takes care of
all of this automation and robustness in getting the environment up and
Jo Maitland: So just give me a couple, then, of the ones when you're trying to start
a server, that, it's not running.
Tal Broda: So it starts simple, it can be a Java sent an exception, just a
communication error with that server, and then after we've got the product
deployed into the server, we make a call into the product to see if the
product, if the load server that we deployed is up and running. And over
there, again, we can get an exception that says "That server went away."
And that's just the reality in cloud computing, and that's fine. The
problem is that if you're trying to do it manually it will take you a very
long time to get an environment that is coherent, knows about all of the
servers, all of them can communicate and run a test in concert for you. And
so, automating is a very, very big deal for us, and our team of performance
engineers, and, in the very near future, our customers.
Jo Maitland: Yes. So, in those instances, then, does the product say, "I could take
ABC actions" and go and do it? Does it say, "You need to take ABCD actions?
What happens when...
Tal Broda: When there's a problem, you mean?
Jo Maitland: Yes.
Tal Broda: So, when there's a problem, the grid manager runs on what we call the
main instance, and that can be on the cloud, and it can be on an appliance.
And that grid manager gets all of these exceptions and then it has the
automation and understanding on what needs to happen in each exception.
Sometimes, it just needs to try again, because we know that it's an error.
That is usually what we call a recoverable error. Sometimes, it needs to
just say, "OK, I tried five times; I'm going to kill this server and get
you a new one." And those are all of the things that we used to do
manually. We used to try five times and then kill the server. So now, it
does it for us. Does that make sense?
Jo Maitland: Yeah, yeah, OK.
Tal Broda: So now we have a grid up and running, which, as you can see, did not
take a very long time, because we do it all in parallel, and we are doing
it using machine automation, so it's not manual, nobody's sweating,
nobody's hysterical making sure that the test is going to be ready in time,
because usually these big tests have a very well-defined time window.
Sometimes, it's late at night, sometimes, it's before a certain launch, but
you have to get the test running when the customer is ready. And so this
really used to be a nail-biter, and right now it's pretty easy.
Once we have the grid up and running, we can go back to the composition
editor, and now we can run the test. We can simply click play, and the
composition editor will load the test into all of those load servers that
we have in the cloud. Those are going to tell us that the test is running,
and now it is playing. As the test is playing, the UI takes you immediately
into the performance intelligence area of the product. This is now the
third step of our demo, in which you can see in real-time all of the
metrics that are coming back, and we are collecting them from all of those
load servers around the world, and we are analyzing them so that you can
get up to the second accurate information on the performance of your
You can see a widget here that's showing us how the load on your website is
increasing in terms of how many virtual users we have, and I'm just running
a very simple test here, so it's going to ramp up to five virtual users.
You can see the average response time, so we calculate this average, in
memory, across thousands of load servers, and show you one metric, but you
can drill into it, and so on. And then you can also see send rate, receive
rate, and you can see different widgets that we have to actually allow you
to get deeper into the different pages that you have. You can see the
average response time for each different page, and you can see that those
metrics are getting updated in real- time.
The nice thing about our dashboards is that they're completely interactive.
And by that I mean that you can basically start to interact with them while
the test is running. So I can take average response time, as a widget, and
you can see that once I start dragging it around, I get this nice drag
handle, and I can actually drop it on other widgets. And when I drop it on
another widget, I get this nice menu that tells me, "Do you want to combine
these two metrics together to see how they correspond to each other by
simply creating two series in the chart?" and we can actually do a real-
time statistical correlation of these two metrics to show you if average
response time reacts to increased number of virtual users. Does that make
Jo Maitland: Mm-hmm. Yep.
Tal Broda: And so, I'm just going to choose 'combine' to show you how this looks,
and you can see that we basically take average response time and overlay
that on top of virtual users. And you get two nice series, while the test
is running, no need to stop it, and you can see what's happening. The other
thing you could do is you can basically open our deep collection of widget
types and start to change the widgets as you want. So I can basically bind
send rate into this widget tree, and I can start to play around and kind of
find metrics that are interesting to me, very, very fast. I can look at
clicks completed, I can look at bytes sent, I can look at bytes received.
In the same way that you can see that we interact with our own metrics, we
can pull metrics from other monitoring providers. We have our own
monitoring, and we can monitor via SSH, two different machines just
basically doing the shelling to them. We can monitor by deploying an agent
on all of those machines, but a lot of our customers already have monitors
installed in their data centers, and they don't want to install yet another
agent on the machines. And so what we've built is basically an extraction
there that allows us in real-time to reach into those monitoring providers
and get real-time metrics information from them. I don't have it
configured, but it's basically the same way that I can get metrics from our
load agents. I can get them, for example, CA Wily Introscope. So I can pull
these metrics into the dashboards; I can show them on the same timeline,
and I can basically correlate them or combine them with other metrics such
that you can see your CPU on the servers, the number of threads on
databases, and different things that might be interesting.
Jo Maitland: Is there anything that we're looking at here that could be alarming?
Tal Broda: I'm just running a very low load test.
Jo Maitland: So this looks pretty nice, huh?
Tal Broda: Yeah, this is a very simple thing with only five users, against a
website that I already know can handle a lot more, but those things
basically can change very rapidly. You can see that I'm getting no errors.
There's actually no data for errors. I'm just going to get rid of that
widget. But you can visualize things in pie charts; you can visualize
things in different ways. And once there is an error, you can drill down
into exactly which page, and with that page, which resource is taking the
most time to render, or which resource is giving me the most errors. Or,
for example, maybe I'm getting more errors from the load agents that I have
on the West Coast than the East Coast, so you can really drill through
things and see them dynamically like that. Does that make sense?
Jo Maitland: Yep. Great. What does the road map look like for you guys? Is it about
supporting more cloud provider APIs, is it about more analytics, is it
about more third party tools like Wily, CA Wily? Where's the focus?
Tal Broda: It's a lot of everything, so yes.
Jo Maitland: You only have a certain amount of people and time, and so where do you
want to focus?
Tal Broda: I think we are going to focus--and maybe Tom wants to take that
Tom Lounibos: I'm not miked.
Tal Broda: But basically what we're going to do is we're going to build more
points of presence around the world, such that we can allow our customers
to get more visibility into where they want to test from. We are going to
make our product more of a platform, so we can allow people to run their
tests on our platform and benefit from the deployment into our-- all of the
cloud providers that we have, and benefit from these analytics, which I
don't believe any other testing tool in the market has the breadth and the
interactivity of the analytics that we offer. And we have a lot of cool
things that are coming in the analytics, as well, that I haven't showed you
yet. In terms of building tests and automating, we just released a new
product that manages, you know, if this is an environment that can do all
of this, there's a product that manages all of the environments and showing
us what all of the environments that we have and customers have, might be
doing in real-time. And, yeah, I think there's always more interesting
things to do, both in building tests, both in automating the execution of
tests, and finally in performance intelligence.
Jo Maitland: Awesome. That's great. Thank you very much.
Tal Broda: You're very welcome. Thank you.