Geeks With Blogs

News This is the *old* blog. The new one is at blog.sixeyed.com
Elton Stoneman
This is the *old* blog. The new one is at blog.sixeyed.com

If you want to do some long-running tests that you can control remotely, a simple Web API controller fits very nicely.

We're currently evaluating an analytics platform using Kafka for messaging. Kafka is an Apache project, very fast and scalable and the preferred choice for many big data people.

Kafka has many supported clients, but it's from the Java world and there isn't much .NET integration going on in the wild.

James Roland has got a .NET Kafka client on github. The code is nicely written, with a simple interface, but in James' own words it's a pet project. I wanted to do some fairly heavy testing to be confident that we could use the WebAPI -> Kafka stack in production, ideally:

  • a soak test running several days
  • pushing a consistent (moderate) load into Kafka
  • with occasional, unpredictable, peaks pushing much more load in

We set up a a set of Kafka brokers, and a single WebAPI instance in the same VNET in Azure. The API had with a controller to stop, start and monitor the soak test.

I can add load to the soak test by POSTing to soak/start which fires up a new thread pushing a message to Kafka every 0.1 seconds:

  [Route("soak/start")]
  [HttpPost]
  public IHttpActionResult Start()
  {
    Debug.WriteLine("SoakController.Start called");
    _SoakTasks.Add(Task.Factory.StartNew(() => SendMessages(_CancellationTokenSource.Token), _CancellationTokenSource.Token));
    return Ok();
  }

Do that repeatedly, then check how many threads are running and how many messages sent with a GET to soak/stats:

  [Route("soak/stats")]
  [HttpGet]
  public IHttpActionResult Stats()
  {
    return Ok(new
    {
      MessagesSent = _MessageCount,
      ProducerCount = _ProducerCount
    });
  }

And those threads will keep running until they're all stopped by setting the cancellation token with a POST to soak/stopall:

  [Route("soak/stopall")]
  [HttpPost]
  public IHttpActionResult StopAll()
  {
    Debug.WriteLine("SoakController.StopAll called. Task Count: {0}, MessageCount: {1}", _SoakTasks.Count, _MessageCount);

    _CancellationTokenSource.Cancel();
    _CancellationTokenSource = new CancellationTokenSource();
    _SoakTasks = new List();
    _MessageCount = 0;
    _ProducerCount = 0;

    return Ok();
  }

For the intermittent bursts of load I'm using loader.io to fire up a few thousand requests per second to a separate WebAPI controller, which sends request log messages to the same Kafka instance. For what I want, manually running the loader tests at intervals is fine (but loader has an API so you can automate load tests too - more of that in my up-coming Pluralsight course).

Full SoakController gist here: SoakController - a simple WebAPI controller for remote-controlling a Kafka soak test.

And so far, the .NET Kafka client looks good, successfully sent a million or so messages today:

image

Posted on Wednesday, July 23, 2014 5:29 PM WebAPI , Kafka | Back to top


Comments on this post: Simple Soak Testing with Web API

No comments posted yet.
Your comment:
 (will show your gravatar)


Copyright © Elton Stoneman | Powered by: GeeksWithBlogs.net