Testing actors in Scala

Probably the most frequent question that people asked on Scala eXchange 2011 was how to test actors. Since I've long planned to write up a blog post on this topic, this was an indication that it's high time to get it done.

It seems that the main problem people have is that actors are asynchronous and this introduces non-determinism in their tests. How do you know when to check if the actor has received and processed the message? Do you just wait a certain number of seconds before checking? This would make tests unnecessarily slow.

Another problem I think folks have is that they don't know how to verify that an actor has sent a message to another actor. Developers are familiar with mocking objects to verify that a method has been called, but how do you mock an actor to verify it has received a certain message?

Finally, it is difficult for most to handle the fact that it's not easy (or at least not idiomatic) to check the actor's internal state. This is especially valid with Akka actors which are created using a factory method and you can't define methods for mucking with the actor's internals.

Let me first address the last one. How do you check that internal state of an actor has changed? You don't! I find that actors are better at following the Object-Oriented principle of encapsulation even than objects are. Relying on a certain internal state couples your tests unnecessarily to the implementation and makes them brittle. As Viktor Klang has pointed out, if the internal state of an actor cannot be observed outside the actor, does it really matter what it is?

So how do you know when an actor has processed a message which was sent asynchronously? An easy way to eliminate non-determinism is to define a method where the functionality is located and call that upon receiving a message:

object MyActor {

  def computation(arg1: String, arg2: Int) = {

    ...
    result
  }
}

class MyActor extends Actor {

  loop {
    react {
      case Message(arg1, arg2) =>

        anotherActor ! computation(arg1, arg2)
    }
  }

}

Then you can test the method the way you are familiar with. Putting the method in the companion object means that you can't test the internal state- but this also means this approach should work with Akka actors as well. We also avoid the problem of checking if the next actor in the chain has received the result.

Sometimes you cannot test a helper method and sometimes testing the method is not enough. In these cases you want to verify explicitly that an actor has sent a message to another actor as a result of receiving a certain trigger message. Here's another approach we use at Apache ESME, which works very well:

case object Wait

class ConductorActor extends Actor {
  def act {

    react {
      case Wait => reply {
        receive {

          case MessageReceived(msg, reason) => msg
        }

      }
    }
  }
}

What's going on here? We define a helper actor (the recipient) which is the one supposed to receive the message from the actor we want to test (the sender). Usually the sender we want to test doesn't send a message to a hardcoded recipient- it is a good idea to either inject it as a construction parameter at instantiation time or register it via a message representing a subscription request.

This actor uses a fairly rarely used nested syntax, which is only available with the actors in the standard Scala library. The recipient handler would reply synchronously (which is what we want) with the message received only after we give it the signal that we've already sent a message to the sender. This implementation also relies on the fact that unhandled messages are kept in the inbox if there's no handler for them. While this can lead to memory leaks if these messages don't get handled, it is a nice way to process out-of-order messages, which is something we take advantage of here. This is similar to selective receive in Erlang and is a fairly painless way to handle race conditions- it doesn't matter which message has been received first here.

These features are not present in Akka, but you could emulate nested handlers using become and keep unhandled messages using. An even better idea would be to use the built-in TestKit or the akka-expect project, which use the same technique in a not so ad-hoc manner (but AFAIK don't work for non-Akka actors).

So now the only thing we need to do is send the trigger message to the tested sender and then ask the recipient if the resulting message has been sent by the sender:

// wait till the message appears in the timeline
// or fail after 5 seconds
val msgReceived = conductor !? (5000L, Wait)

if (msgReceived.isEmpty) fail("no message received")

If the recipient gets the message within a certain timeframe, the test is successful, otherwise we time out and fail the test. The nice thing about this approach is that in the happy path case, the test can continue immediately without slowing down the test suite. The test is slowed down by the designated timeout only when the test is going to fail, but this should be an exceptional event.

A minor inconvenience is if the sender doesn't expect the recipient to be a Scala library actor, but e.g. a Lift actor, but this can be easily overcome by using a bridge actor, which only acts as an intermediary and just forwards the request to the designated recipient actor:

class BridgeActor(receiver: Actor) extends LiftActor {

  protected def messageHandler = {
    case nm @ MessageReceived(_, _) => receiver ! nm

  }
}

val liftActor = new BridgeActor(conductor)

Distributor ! Listen(theUser.id.is, liftActor)
Distributor ! Listen(followerUser.id.is, liftActor)

Here we're injecting one actor as a construction parameter and registering another via the Listen message.

Further research

If you're using Akka, your best bet is the recommended TestKit- here's an article on how to use it:

http://roestenburg.agilesquad.com/2011/02/unit-testing-akka-actors-with-testkit_12.html

Another solution would be to use the akka-expect framework:

https://github.com/joda/akka-expect

A more universal library is Awaitility, which uses a similar solution, but with more general applicability to Java threads and Scala actors:

http://code.google.com/p/awaitility/

ScalaTest and Specs also have the conductor actor, which implement a similar idea of using a CountDownLatch to make actors deterministic:

http://www.scalatest.org/scaladoc/doc-1.0/org/scalatest/concurrent/Conductor.html

Testing with Lift's TestFramework

Getting started

HTTP testing is sufficiently tedious that some folks don't do it. Even if we do it, in Java it doesn't look as pretty as it could be.

The Scala Lift web framework, apart from the other advantages it has (which are a topic for many a blog post) offers some syntax-sugar wrappers so that testing of our REST APIs can be concise and to the point.

Combined with Jetty, this leads to some seriously short and readable code.

First, we need to:

• use the trait "with TestKit" in our test


• override the baseUrl property


• start our Jetty server

Then if we want to output our own failure message later, we need to provide an implicit class of type ReportFailure, where we only need to implement the fail method, which predictably takes a String. For example, regardless of whether we use ScalaTest or Specs, we can fall back to the fail method which is inherited by our test class:



This is all we need so far.

Using it

Now we're ready to GET some action.



There's already a lot going on here. First of all, the get and post methods return a TestResponse (you don't see it, because it's inferred). And the nice thing about TestResponse (and Scala) is that since it implements the foreach method, it can be used in for expressions together with the "<-" symbol.

Whatever is extracted by the for expression can be used to issue get and post requests, but in the context of the previous request. This means that cookies are preserved, so you can use the same HTTP session. In this contrived example we use an httpClient as an argument to the get method (which is supposed to use some HTTP credentials and log us in). In the following call, we don't need to use the authenticating http client, because we can have our cookies (and eat them, too).

We can also use http parameters as a sequence of tuples, which can be delimited using the "->" operator.

For any TestResponse, we can use the xml property, which is the XML as a scala.xml.Elem wrapped in the (ubiquitous for Lift) Box.

Finally, there's concise syntax to assert certain properties about the response. For example the "!@" operator checks if the response code is 200, otherwise it fails with the error message specified after the operator.

If we expect a specific return code other than 200, we can also specify it explicitly:

Customizing

If you want to define a HTTP basic authentication client to be used by default by your requests, you can override the method theHttpClient. Lift's TestFramework provides the buildBasicAuthClient method, which can be reused to quickly create an HTTP client with a set user and password.



If you're like me, you might lose hours trying to find out why the request doesn't work when the server doesn't explicitly request authentication. Then setting the preemptive flag on the client would definitely save some time.

I know some of you Scala geeks are already bored because everything so far is just so simple. There wasn't even any mention of implicits! But rest assured, I will find some use for implicits.

Pimping

The usage snippet in the for expression above is not quite right. If the response contains no valid xml, then the specs matcher will not be executed and the test will not fail, although it should. Here we can use the fact that TestResponse also implements the filter method, which allows us to have if expressions (also known as filters). The post snippet could then be rewritten like this:



We are again using the Specs matcher to test if the response contains xml and fail otherwise. Notice that this expression does not return true- it need not be boolean. This is due to the fact that TestResponse's filter method intentionally returns Unit, not Boolean.

However, that's too much boilerplate, which we have to do for every time we want to check the contents of an xml response:

1. Check if the response is valid and has a response code of 200


2. Check if the xml is not empty


3. Extract the xml


4. Only then check the contents of the XML

If only we could make TestResponse do what Specs does with XML... But since this is Scala, we can create a wrapper and then transparently substitute it using our implicit conversion:



An important thing to remember about implicit conversions is that if we want them to feel seamless, the wrapper class' methods should return the original class type (the one before the conversion). Otherwise we could get unexpected object classes similarly to what occurred with the Scala 2.7 RichString, for instance with "aaa" reverse == "aaa"

Since we have chosen to use the same operator as Specs uses, we had to disambiguate with the XmlBaseMatchers class name, which is the specs trait containing all the XML matching goodness.

Now we can use our fancy TestResponse operators. Notice that since we return the same class type that we already had, we can chain:



This looks more compact than the original version so you can focus on the API differences and not be distracted by boilerplate preconditions.