Load Testing Live Streaming Servers

There are two types of test I’ll describe below. First of all using Apple HLS streams, which is HTTP Live Streaming via port 80, supported by iOS and Safari, and also by Android (apps and browser). Then we have Adobe’s RTMP over port 1935, mostly used by Flash players on desktop, this covers browsers like Internet Explorer and Chrome on desktop. These tests apply to Wowza server but I think it’ll also cover Adobe Media Server.

All links to files and software mentioned are duplicated at the end of this post.

It’s worth noting that you can stick to HLS entirely by using an HLS plugin for Flash video players such as this one, and that is what we’re doing in order to make good use of Amazon’s CloudFront CDN.

For the purpose of testing you may also wish to simulate some live camera streams from static video files, see further down this post for info on how to do that on your computer, server or EC2.

Testing RTMP Live Streaming with Flazr

In this test we want to load test a Wowza origin server itself to see the direct effect of a lot of users on CPU load and RAM usage. This test is performed with Flazr, via RTMP on port 1935.

Assuming you’ve set up your Wowza or Adobe Media server already, for example by using a pre-built Wowza Amazon EC2 AMI. We’re using an m3.xlarge instance for this test as it has high network availabilty and a tonne of RAM – and we’re streaming 4 unique 720p ~4Mbit streams to it, transcoded to multiple SD and HD outputs (CPU use from this alone is up to 80%).

Installing flazr

First up, for the instance size and test configuration I’m using I modified flazr’s client.sh to increase the Java heap size to 8GB, otherwise you run out of RAM. Next up FTP and upload (or wget) flazr to a directory on your server/EC2 instance. Then SSH in and:

apt-get install default-java
cd path/to/flazr
chmod +x client.sh
./client.sh -host yourserver.com -version 00000000 -load 1000 -length 60000 -port 1935 -app yourAppName yourStreamName

The order of parameters does seem to matter in later versions of flazr, but either way this test runs for 60 seconds, with a load of 1000 viewers. Given all the transcoding our CPU was already feeling the pain, but there was no sign of trouble. We managed 4500 before anything started to stutter in our test player from another m3.xlarge instance.

wowza
Wowza CPU Usage

Of course this only matters if you are not using a CDN, but it’s good to know this EC2 instance can handle a lot of HD viewers.

Testing HLS Live Streaming (or a CDN such as Amazon CloudFront) with hlsprobe

Onto HLS streaming, the standard for mobile apps and sites. We have used Wowza CloudFront Formations to set up HLS caching for content delivery, so that we can handle a very large number of viewers without impacting on the CPU load or network throughput of the origin server, and to giver us greater redundancy. Given CloudFront works with HLS streams we are not using RTMP for this test, so we cannot use Flazr again. To test HLS consumption –that being the continuous download of .m3u8 files and their linked .ts video chunks– we can use a tool called hlsprobe, which written in python.

If you’re on a Mac and don’t have python I recommend you install it via brew to get up and running quickly. If you don’t have brew, get it here.

#on a mac
brew install python
#on ubuntu/amazon
sudo apt-get python

hlsprobe also relies on there being an SMTP server running, not that you need a fully functional one but:

#on mac
sudo postfix start
#on ec2, this auto-starts
sudo apt-get install postfix

Then to install hlsprobe’s dependencies and hlsprobe itself:

pip install m3u8
pip install PyYAML
git clone https://github.com/grafov/hlsprobe
cd hlsprobe

A sample config is linked at the end of the post.

Running hls probe is as simple as this (note the -v verbose mode, you can turn that off once you have it working).

python hlsprobe -v -c config.yaml

Now if you fire up the Wowza Engine Manager admin interface, you should still see the connection count and network traffic, but the traffic. If you’re testing your CDN such as with CloudFront, you should note that your CPU usage does not increase substantially as you add thousands of clients.

Simulating cameras to Wowza via nodeJS

It’s good to be able to simulate live streams at any time, either from your computer or in my case, from some EC2 instances. To do this I’ve written a simple nodejs script which loops a video, optionally transcoding as you go. I recommend against that due to high CPU use and therefore frame-loss; in my sample script I am passing through video and audio directly, the video is already using the correct codecs, frame size and bitrate via Handbrake.

The script runs ffmpeg, so you’ll need to install that first:

#on a mac
brew install ffmpeg
#on ubuntu/Amazon you'll have to to install/compile ffmpeg the usual way

Edit the js script to point to your server, port, and video file, the run the script with:

node fakestream.js

If the video completes, it’ll restart the stream but there will be a second of downtime, some video players automatically retry, but make sure your video is long enough for the test to be safe.

These are just a couple of ways of load testing a live streaming server, there are 3rd parties out there but we’ve not had great success so far, and this way you have a lot more control over the test environments.

Links

fakestream.js – NodeJS script to simulate live streams
config.yaml – Sample config for hlsprobe
hlsprobe – Tool for testing HLS streams
Flazr – Tool for testing RTMP streams
OSMF-HLS – OSMF HLS Plugin to support HLS in Flash video players

Compiling FFMPEG Statically on OS X (and Windows)

I was up extremely late last night figuring this out, so I made some rough notes as I went along. Hopefully they’ll save someone a small nightmare.Disclaimer: I haven’t written C code since I was in my teens, and apart from some Objective-C from the…

I was up extremely late last night figuring this out, so I made some rough notes as I went along. Hopefully they’ll save someone a small nightmare.

Disclaimer: I haven’t written C code since I was in my teens, and apart from some Objective-C from the comfort of XCode, I haven’t really done a whole lot of GCC compiling on any system, so this may not be the best way to do any of this.
Continue reading “Compiling FFMPEG Statically on OS X (and Windows)”

HD Video Silverlight Firefox Extension, Requires Flash?

Quote:

“Move Media Player and Microsoft’s Silverlight To Create The Most Flexible, High Quality Streaming Video Platform”

Here’s a strange one. Move Networks is offering a HD video player “powered by Silverlight” that comes in the form of a…

Quote:

“Move Media Player and Microsoft’s Silverlight To Create The Most Flexible, High Quality Streaming Video Platform”

Here’s a strange one. Move Networks is offering a HD video player “powered by Silverlight” that comes in the form of a Firefox extension. Once installed you are redirected to a page that shows a HD video, but a look at the source shows the video player header and controls/progress bar are all made in Flash, it just overlays a block, streaming in a QMX file (doesn’t open in Windows Media Player when renamed asx).

It seems a lot of effort to go to, cross communicating between Flash, JavaScript, Silverlight and Firefox, when you could have made the whole player using just one technology. I don’t get the point of the Firefox extension in the first place, the video controls require Flash Player anyway, and if they had used Silverlight for them you’d need that plugin so why bother with this extension at all?

Try it here.

Subscription Model Should Dominate All Media Consumption

That means iTunes’ current model will be due for some drastic changes, no more downloading things “to keep”. Someone asked me the other day whether I really thought all our media consumption (music, video, insert-other-media-here) would be entirely subsc…

That means iTunes’ current model will be due for some drastic changes, no more downloading things “to keep”. Someone asked me the other day whether I really thought all our media consumption (music, video, insert-other-media-here) would be entirely subscription based in the near future. I gave a definite YES in response.

This is something I’ve had on my mind a few years, and personally I’m of the opinion the only viable solution to the increasingly complex problem of content management and ownership -whilst maintaining the rights of artists and authors- is to consume all of our content as the result of a subscription to a media conglomerate or third party broker.

The stepping stones to this are coming, one at a time. Nokia today announced the “Comes With Music” service, which allows customers purchasing a new device, unlimited free downloads of a million songs from (initially) Universal’s music library. On top of that the downloads will remain accessible and playable after that year is up.

That’s a good first step, and it’s not as if Nokia is the only one providing subscription based music services. But what I’m talking about here is entirely subscription based models, where you don’t ever have a file to “keep”, you just obtain the media when you wish to listen/watch/interact with it. This means that to the company facilitating the transfer, you cease to be a user that has paid for X, Y and Z. Instead you gain a “role”, your role specifying what sort of access to the content library you have, and in what circumstances you can consume that media.

Added to this I very much think that we will completely give up ownership of our content. Not only will we not be able to say “yeah, I own a copy of XYZ”, to me it seems sensible that we will also not be storing our content anywhere near our physical locality.

When speaking about this topic there are always throes of despair from people that are used to owning physical copies of original content, and that’s understandable, many people have grown up collecting music or films on physical media. I imagine you’d get a very different opinion from the current generation of children with regards to the importance of owning content. The BitTorrent/YouTube generation has very little interest in collecting media when its all there, all the time anyway.

The subscription/no-ownership model is a big shift in how you see media. It also requires a great deal of trust in the stability of the companies offering the content, as well as the speed and reliability of your internet connection with regards to whichever device you are using to consume the content. But all in all I think it’s a sensible proposition, and nothing new, but thankfully companies like Nokia are not afraid to dip their toe in the water and examine new ways to deal with media to make the necessary baby-steps toward this goal.

But will the subscription model be successful? I think so, and in other markets it already is. It certainly checks all of the right boxes… To me one of the keys to the success of a new technology or any mechanism that uses technology is that it must make things easier than before (or at the very least bring with it new abilities that far outweigh any new inconveniences.) Related to this the subscription model deals with the problem of piracy in the only way possible, by making it *easier* to consume your content by paying for a subscription, than going through alternative channels.