Speaking my (programming) language?

Partially unexpected effects of chaining partial functions in Scala

2011-07-31T07:54:00.001+03:00

People who learn Scala usually agree that pattern matching is a great feature which helps make your code more expressive. Some time later they also discover that partial functions used in the match statement can also be used separately. And since partial functions are full-blown functions, you can combine them in a couple useful ways:

f1 orElse f2: Combines two partial function in a new one; if f1 is not defined in its argument, tries f2.
f1 andThen f2: Applies the result of f1 to f2. This means that the output of f1 must be a type compatible with the input of f2

orElse

As others have discovered, you can combine a list (TraversableOnce really) of partial functions into one with reduce. What's not so obvious though is that the way you combine them can lead to unexpected perfomance consequences.

In order to easily create a lot of partial functions to test, we will create a higher-order function to generate them (if you're used to Java, you can call it a factory). The produced partial function will print a short message when its isDefinedAt method is called (not when it's applied):


def genPF(defined: Int): PF = { case i if {println(defined); defined == i} => i }
val f123 = 1 to 3 map genPF reduceLeft(_ orElse _)

Let's try it:


> f123(1)
1
1
1
> f123(2)
1
2
1
2
1
2
> f123(3)
1
2
3

Wait, what? The isDefinedAt method is called up to 6 times. It gets even worse with a bigger number of composed functions.


val f1to5 = 1 to 5 map genPF reduceLeft(_ orElse _)


> f1to5(2)
1
2
1
2
1
2
1
2
> f1to5(3)
1
2
3
1
2
3
1
2
3
> f1to5(4)
1
2
3
4
1
2
3
4

Let's take a closer look at the definition of isDefinedAt and apply of the function created with orElse:


def orElse[A1 <: A, B1 >: B](that: PartialFunction[A1, B1]) : PartialFunction[A1, B1] =
  new PartialFunction[A1, B1] {
  def isDefinedAt(x: A1): Boolean =
    PartialFunction.this.isDefinedAt(x) || that.isDefinedAt(x)
  def apply(x: A1): B1 =
    if (PartialFunction.this.isDefinedAt(x)) PartialFunction.this.apply(x)
    else that.apply(x)
}

When you apply a composed partial function, we first check if it's defined in either f1 or f2, and then we check f1 again, so that we know which one to call. This means that in the worst case, isDefinedAt for f1 is called twice.

Given this, we can explain what happens here. The isDefinedAt delegates to the composed functions' methods, and when it's called twice... you know what happens when we do this again and again. We can fairly easily find out that isDefinedAt is called k * (n - k + 1) times, where n: number of composed functions, k: the first function that matches.

Luckily, there is an easy solution to combine partial functions in a more efficient way. We can use reduceRight, where isDefinedAt for each composed function is invoked at most twice. Verifying this and finding out why is left as an exercise for the curious reader (as you undoubtedly are, since you're reading this).

andThen

You would think that f1 andThen f2 should be defined only in the cases when the results of f1 are defined in f2


val doubler: PartialFunction[Int,Int] = { case i if Set(1,2) contains i => i * 2 }
val f = doubler andThen doubler

Of course, that's not how it works. In order to find out if the output of f1 is a valid input for f2, we would need to execute the function, and it's better not to do this in case it has side effects. This means that we cannot rely on calling isDefined for the combined function to avoid MatchErrors:


> f isDefinedAt 2
res3: Boolean = true
> f(2)
scala.MatchError: 4
...

Conclusion: when you're looking for performance, it always help to understand how the abstractions you're using decompose into simpler building blocks.

Testing actors in Scala

2011-06-17T15:04:00.001+03:00

Probably the most frequent question that people asked on Scala eXchange 2011 was how to test actors. Since I've long planned to write up a blog post on this topic, this was an indication that it's high time to get it done.

It seems that the main problem people have is that actors are asynchronous and this introduces non-determinism in their tests. How do you know when to check if the actor has received and processed the message? Do you just wait a certain number of seconds before checking? This would make tests unnecessarily slow.

Another problem I think folks have is that they don't know how to verify that an actor has sent a message to another actor. Developers are familiar with mocking objects to verify that a method has been called, but how do you mock an actor to verify it has received a certain message?

Finally, it is difficult for most to handle the fact that it's not easy (or at least not idiomatic) to check the actor's internal state. This is especially valid with Akka actors which are created using a factory method and you can't define methods for mucking with the actor's internals.

Let me first address the last one. How do you check that internal state of an actor has changed? You don't! I find that actors are better at following the Object-Oriented principle of encapsulation even than objects are. Relying on a certain internal state couples your tests unnecessarily to the implementation and makes them brittle. As Viktor Klang has pointed out, if the internal state of an actor cannot be observed outside the actor, does it really matter what it is?

So how do you know when an actor has processed a message which was sent asynchronously? An easy way to eliminate non-determinism is to define a method where the functionality is located and call that upon receiving a message:


object MyActor {

  def computation(arg1: String, arg2: Int) = {

    ...
    result
  }
}

class MyActor extends Actor {

  loop {
    react {
      case Message(arg1, arg2) =>

        anotherActor ! computation(arg1, arg2)
    }
  }

}

Then you can test the method the way you are familiar with. Putting the method in the companion object means that you can't test the internal state- but this also means this approach should work with Akka actors as well. We also avoid the problem of checking if the next actor in the chain has received the result.

Sometimes you cannot test a helper method and sometimes testing the method is not enough. In these cases you want to verify explicitly that an actor has sent a message to another actor as a result of receiving a certain trigger message. Here's another approach we use at Apache ESME, which works very well:


case object Wait

class ConductorActor extends Actor {
  def act {

    react {
      case Wait => reply {
        receive {

          case MessageReceived(msg, reason) => msg
        }

      }
    }
  }
}

What's going on here? We define a helper actor (the recipient) which is the one supposed to receive the message from the actor we want to test (the sender). Usually the sender we want to test doesn't send a message to a hardcoded recipient- it is a good idea to either inject it as a construction parameter at instantiation time or register it via a message representing a subscription request.

This actor uses a fairly rarely used nested syntax, which is only available with the actors in the standard Scala library. The recipient handler would reply synchronously (which is what we want) with the message received only after we give it the signal that we've already sent a message to the sender. This implementation also relies on the fact that unhandled messages are kept in the inbox if there's no handler for them. While this can lead to memory leaks if these messages don't get handled, it is a nice way to process out-of-order messages, which is something we take advantage of here. This is similar to selective receive in Erlang and is a fairly painless way to handle race conditions- it doesn't matter which message has been received first here.

These features are not present in Akka, but you could emulate nested handlers using become and keep unhandled messages using. An even better idea would be to use the built-in TestKit or the akka-expect project, which use the same technique in a not so ad-hoc manner (but AFAIK don't work for non-Akka actors).

So now the only thing we need to do is send the trigger message to the tested sender and then ask the recipient if the resulting message has been sent by the sender:


// wait till the message appears in the timeline
// or fail after 5 seconds
val msgReceived = conductor !? (5000L, Wait)

if (msgReceived.isEmpty) fail("no message received")

If the recipient gets the message within a certain timeframe, the test is successful, otherwise we time out and fail the test. The nice thing about this approach is that in the happy path case, the test can continue immediately without slowing down the test suite. The test is slowed down by the designated timeout only when the test is going to fail, but this should be an exceptional event.

A minor inconvenience is if the sender doesn't expect the recipient to be a Scala library actor, but e.g. a Lift actor, but this can be easily overcome by using a bridge actor, which only acts as an intermediary and just forwards the request to the designated recipient actor:


class BridgeActor(receiver: Actor) extends LiftActor {

  protected def messageHandler = {
    case nm @ MessageReceived(_, _) => receiver ! nm

  }
}

val liftActor = new BridgeActor(conductor)

Distributor ! Listen(theUser.id.is, liftActor)
Distributor ! Listen(followerUser.id.is, liftActor)

Here we're injecting one actor as a construction parameter and registering another via the Listen message.

Further research

If you're using Akka, your best bet is the recommended TestKit- here's an article on how to use it:

http://roestenburg.agilesquad.com/2011/02/unit-testing-akka-actors-with-testkit_12.html

Another solution would be to use the akka-expect framework:

https://github.com/joda/akka-expect

A more universal library is Awaitility, which uses a similar solution, but with more general applicability to Java threads and Scala actors:

http://code.google.com/p/awaitility/

ScalaTest and Specs also have the conductor actor, which implement a similar idea of using a CountDownLatch to make actors deterministic:

http://www.scalatest.org/scaladoc/doc-1.0/org/scalatest/concurrent/Conductor.html

Why Scala's Option won't save you from lack of experience

2010-10-21T00:00:00.002+03:00

Some time ago Cedric Beust was the cause for some excitement in the Scala community by declaring that he doesn't see the advantages of using Scala's Option type, which is also similar to Haskell's Maybe type.

There were a lot of insightful comments which outlined the benefits of using Option. James Iry has written a well-reasoned post called Why Scala's "Option" and Haskell's "Maybe" types will save you from null.

I wanted to approach things differently. I wanted to show people some patterns which usually come with experience and let them decide which is better. That's why this turned into a very long post which looks more like a tutorial. Of course, I'm not adding anything new to the discussion, but just summarizing some of the common wisdom accumulated by the community. I'm sure this is not the last typical blog post showing the wonders of Option, but I hope it can clear up some (Some?) misunderstandings. Or Maybe not.

But more importantly, I wanted to explain why having a solution as a language feature is a premature optimization. It's neither as flexible, nor as powerful as having it as just a type in the library.

NullPointerException

Cedric is definitely not alone- programmers who decide to give Scala a try (and moreso Haskell or the ML family) face conceptual differences from popular imperative languages. Just reading that Option is a type wrapper does not mean it's easy to wrap one's head around it.

One advantage of using Scala is that if you are convinced that NullPointerExceptions solve your problem better, you are free to use it. Option is just one option. And of course, you can come back later any time if you make up your mind that Option has Some advantages (e.g. composability). Of course, some might view having too many choices as a disadvantage.

But both Scala and Clojure must live with the design decisions on the JVM which were taken before them, and with interoperating with a wealth of existing libraries. So allowing null is first of all a practical decision.

Getting started

First of all, let's define a simplistic employee type, a list of employees, and a map where the "builder" occupation points to the list of employees.


case class Person(name: String, email: String)
val employees = List(Person("bob", "bob@builder.com"))
val occupations = Map("builder" -> employees)

Compile-time safety and backward compatibility

One of the examples given by Cedric is pattern matching on an Option type. His main point of contention is that this looks very much like testing for null. Except one minor point: where is the example similar to the case when you don't want to test for null?

Exactly. No such valid example exists if you want to use the value inside Option, at least not unless you're explicit about it. As Paul Snively mentions, the compiler will stop you. Cedric has noticed, "the worst part about this example is that it forces me to deal with the null case right here". But this is not the worst part, it's maybe the best part.

Do you remember a feature which was added to Java 5, which was intended to save you from another type of exception, ClassCastException? Of course, that would be generics! The problem is, it gives you type-safety, but only as long as you use classes compiled with the Java 5 compiler and you don't use the escape hatch of raw types. As soon as you start using legacy code, you leave the safety of compiler checked code. You can ignore the warnings at your own risk, because then there's no guarantee you won't get a ClassCastException.

Does this remind you of something? Compile-time safety as long as you don't use legacy code or the escape hatch? These restrictions sound exactly like the ones Option has.

And of course, there is an escape hatch. You can use Option.get or instead of pattern matching, you can even use your old friend the if statement (Scala veterans, please close your eyes now):


val evangelists = occupations.get("evangelist")
// ugly, ugly, ugly
if (evangelists == None)
  println("No such occupation here!")
else
  println("Found occupation " + evangelists.get)

But, as you'll see later, this doesn't mean that you have to deal with the "no value" case right here. Pattern matching is not the only option.

Simplification through a generalization

Instead of solving the most obvious problem, it always pays out to see if it isn't a manifestation of a bigger class of problems. Having a related class of problems solved by a common pattern simplifies things. There are fewer rules to remember. Not only that, but the specific applications of the general solution begin to interact in ways you couldn't have anticipated before. Eventually problems will appear which would be solved by a general solution, problems which you didn't know about when you implemented the solution.

As it turns out, the problem of syntax similar to the safe dereference operator can be solved in Scala. I would say that having no explicit syntax for this, it's a fairly elegant solution, but this is subjective opinion.

Handling both value and lack of value and stop processing

This is handled by our well-known pattern match. It seems easy to use and obvious in what it does.

The advantage of pattern matching is that it uses the type system in such a way that forgetting to handle one of the cases explicitly will result in a compile time warning.


occupations.get("builder") match {
        case Some(_) => println("builder occupation exists")
        // oops, forgot to check for None or the catch-all _
}
// warning: match is not exhaustive!
// missing combination           None

The disadvantage is that pattern matching doesn't compose very elegantly. If the result of the pattern match is just an intermediate step, you'll need to add another one, and another, and pattern matching does take some screen real estate.

Pattern matching is a bit like exception handling with try/catch blocks- you usually do it when you're interested in both the normal behaviour and the exceptional behaviour and that's fairly verbose. On a related note, did you know that you can use pattern matching in Scala's exception handlers?

So let's see what we can do to get more composable data processing.

Transform value

When we're interested in creating a series of steps for processing a value, we can use map. It will transform the value if it's there, but will leave it inside the Option. And map won't change the Option if it's empty (None.map will result in None).


val employee = employees find ( _.name == "bob" )
// Some(Person(bob,bob@builder.com))
employee map ( _.email )
// Some(bob@builder.com)

If you need to "flatten" the result you can use flatMap. This means that instead of an Option nested inside an Option, you get just one Option. It only results in Some (a "full" Option type) if it's called on Some and also results in Some:


val builders = occupations.get("builder")
// Some(List(Person(bob,bob@builder.com)))
val bobTheBuilder = builders flatMap { _ find ( _.name == "bob" ) }
// Some(Person(bob,bob@builder.com))

If you were using just map, you would get Some(Some(Person(bob,bob@builder.com))), which is probably a bit too nested for your taste.

Some of you are probably familiar with other languages which have map (like Ruby or Python) and are scratching their heads: "Wait, wasn't map defined only for lists/Enumerables?". Please be patient.

Only get the value if it satisfies a test

If you find only some of the possible values useful, you can weed out what you have by using filter. It will only result in Some for values which satisfy a certain condition (called a predicate).


bobTheBuilder filter { _.email endsWith "builder.com"}
// Some(Person(bob,bob@builder.com))

I'm sure at this point the folks who have used Google Collections have also joined the folks with past Ruby or Python experience screaming: "Hey, but filter is only used for Collections!"

Transform lack of value

That's fine, but eventually you want to get the value out. If there's no value, just assume some default value. We have to use pattern matching again, right?

But there is a shorter solution. getOrElse extracts the value or puts a default value of the same type if there's nothing in the Option container:


val larryWho = employees find ( _.name == "larry" )
// None
val emptyEmail = larryWho map ( _.email )

// None
emptyEmail.getOrElse("nobody@nowhere.com")
// nobody@nowhere.com

Groovy has this in the form of the Elvis operator. The trouble is, you can't get rid of the elvis operator, it's just adding cruft to the language, even though it's a useful one. It's also somewhat restrictive that it's all this operator can do.

Chain, chain, chain

The reason map, flatMap, filter and getOrElse are so useful is that they can be chained together, intermixed and the results can be passed around to other methods.

Let's shift to high gear and put it all together:


occupations.get("builder").
flatMap { _ find ( _.name == "bob" ) }.
map (_.email).
filter { _ endsWith "builder.com"}.
getOrElse("nobody@nowhere.com")
// bob@builder.com

If we're not yet interested in which step processing has failed, this is a clear way to express the process flow. It's also similar to the Fantom example Cedric desribed.

There is an ever shorter syntax for this using for expressions (or for comprehensions).


{for (builders <- occupations.get("builder");
      bobTheBuilder <- builders find (_.name == "bob");
      email = bobTheBuilder.email if email endsWith "builder.com"
      ) yield email
} getOrElse "nobody@nowhere.com"
// bob@builder.com

But wait, weren't for expressions a way to loop over stuff? Well, yes, this too. More generally, for expressions work with collections. And the beauty of it all is that we can use collections together with Option and do nested invocations. Let's modify the example a bit and suppose that there might be more than one person named Bob and we want them all.


for (builders <- occupations.get("builder") toList;
     bobTheBuilder <- builders if bobTheBuilder.name == "bob";
     email = bobTheBuilder.email if email endsWith "builder.com"
     ) yield email
// List(bob@builder.com)

Because for all practical purposes, Option behaves like a specialized collection. By viewing it as one, you reuse the experience of all the programmers using Groovy, Ruby, Python, Google Collections and whatnot, and flatten the learning curve.

Now imagine that the only way to work with a collection is to pattern match it. Would you use it? Yeah, me neither.

Safe invoke and composability

Now let's see how filter works in Fantom:


fansh> list := [1, 2, null]
fansh> list.findAll |v| { v.isEven }
sys::NullErr: java.lang.NullPointerException

Oh crap, then I need to to use the safe invoke operator:


fansh> list.findAll |v| { v?.isEven }
ERROR(20): Cannot return 'sys::Bool?' as 'sys::Bool'

But it all results in a compile-time error. It's the same story with the reduce higher-order function:


fansh> list.reduce(0) |r, v| { v + r }
sys::NullErr: java.lang.NullPointerException

So I can't practically use the safe invoke operator in nullable collections with filter/reduce, which the Fantom documentation has conveniently omitted from the documentation page. So we're back to checking for null the old way. This means that unlike flatMap, the safe invoke works fine when you chain, but not when you compose.

Iterator

Let's now see some other advantages of Option behaving like a collection. For instance, it lets you use Iterable's API in some elegant ways:


val noVal: Option[Int] = None
val someVal = Some(4)
List(1,2,3) ++ someVal ++ noVal
// List(1, 2, 3, 4)

Guess what happens here? Only the numbers contained in Some are added to the list. I think there's no operator for this in Fantom and Groovy, and it would be overkill to include one, too.

One size doesn't fit all

Wait, if Option is like a collection, does this mean that there are many types of Option? Does this mean I can create my own Option?

Yes, and yes. Just as there isn't just one type of List, or one type of Map, there can also be several types of Option. For instance, Lift defines its own, which it currently calls Box (I think it's a great metaphor). One of the things Box has in addition to Option is a type to collect Failures. It's no longer just a "dunno what happened, something failed along the way". It's a list of error messages which can pinpoint exactly what went wrong. This is invaluable for a web framework, because when a user expects a complex form to be validated, a simple "some of our input is wrong" just won't cut it.


for {
  id <- S.param("id") ?~ "id param missing" ~> 401
  u <- User.find(id) ?~ "User not found"
} yield u.toXml

And guess what, you can also define your own operators, which also work in for comprehensions.


for {
  login <- get("/account/verify_credentials.xml", httpClient, Nil)
    !@ "Failed to log in"
  message <- login.post("/statuses/update.xml", "status" -> "test_msg1")

    !@ "Couldn't post message"
  xml <- message.xml
} yield xml

Except that they're not operators. When conventions and patterns evolve, Lift folks can always change the "operator". Or another framework can do it.

Another good example of using an enhanced Option is Josh Suereth's (jsuereth) Scala ARM library. It's collecting a list of errors, and Java's safe resource blocks proposed for Java 7 looks primitive in comparison.

Option is not only Scala's to have

Actually, there's nothing specific about Option that ties it to Scala. The only thing which is Scala specific is the syntax sugar of for comprehensions. You can use Option in Java if you want, although some of the examples above wouldn't be as concise and so it might be a bit of a pain. But this doesn't stop people from trying to recreate Maybe in Java.

Many folks have even tried to cheat and use Java's enhanced for expression as syntax sugar. So if Java can afford some syntax sugar over Iterator, and according to Joshua Bloch the for expression is a clear win, why shouldn't Scala do it? The difference is only that Scala's is applicable to a wider set of problems.

Why language syntax won't save you from the future

One advantage which some people don't realize Scala has is that it's a relatively minimal language with a relatively rich library. Apart from Option, there are other examples where having a library instead of a language feature has brought huge benefits to Scala. One such example is actors.

Let's compare Scala's actors to Erlang's. Undoubtedly Erlang is the daddy of practical actor implementations. Actors in the Erlang virtual machine have some superb characteristics which other runtime implementations will have a hard time catching up with. They're scalable and lightweight. They work across hosts and in the same virtual machine. They can be hot swapped and you can create millions of them in a single virtual machine.

But there's only one type of actor. This means it must deal with all possible cases, and as it usually happens, it deals better with some and worse with others. I have no doubt that having actors as part of the language, choosing only one type of actor is a very sensible decision, but it can still be restrictive sometimes.

Scala deals with this differently. Scala's actors are not part of the language, and the actor message send syntax (which is borrowed from Erlang) is just a method invocation in a library. This means that Scala's free to evolve different actor implementation, and you're free to choose the one which suits your case better. Some are more full-featured, some are lightweight and performant; some are remote, some are local; some use a thread pools, some use a single scheduler; some use managed hierarchies, some don't. Regarding actors Scala the language is smaller than Erlang, but the Scala libraries are richer.

Which actor library will win? I don't know. And probably neither do you. There might not be one best answer. That's why hardcoding stuff in the language is not a good way to prepare for the future. Only experience will, either the collective experience of the community or the extensive experience of a genius Benevolent Dictator For Life.

Top 5 underused GNU screen features

2010-09-11T23:00:00.004+03:00

Most people use GNU screen mostly for its abilities to detach from the terminal and create multiple shell sessions. This makes it ideal in combination with remote terminal connections like ssh. There are some features which, although not very popular, are still useful on a number of occasions.

As a bonus, let's start with a short explanation of the popular features. If you're familiar with detaching and multiplexing, you might want to skip to the sections about more rarely used features.

Detaching

The first thing to know about screen is that all shortcuts start with Ctrl + A by default.

The most common screen workflow is:

Connect to a remote system via ssh

Start/attach screen:
```
screen -D -RR
```
This command will detach a running screen session and create a new one, if necessary.

Work

Detach screen by pressing Ctrl + A, then D.

Disconnect.

Take a break, return to 1 :)

You can find the currently running screen sessions using the command:


screen -ls

Alternative: nohup is also used when you want to detach a program when you disconnect, but it will never attach to the terminal, so you cannot use it if it's interactive.

Multiplexing

When you connect via ssh, one shell is rarely enough. Connecting a second time is often inconvenient and slow. Enter screen- you can create a new session using Ctrl + A, then C. You switch to the next one using Ctrl + A, then space (or n). You switch to the previous one using Ctrl + A, then backspace (or p). Using Ctrl + A and then a digit you can directly jump to the session numbered 0-9.

Alternative: using ssh's option ControlMaster will reuse an existing connection and open another shell immediately, but this is only relevant for ssh.

Now you know the basics, let's see what other goodies screen offers.

1. Sharing sessions

Typing in a terminal is usually a lonely experience. Sometimes you wish that you could show someone else what you're doing or even do it together. It might be because you want to teach someone some UNIX tricks, or you must solve an issue together or you're up to the challenge of remote pair programming. Your wish is granted! In screen, you can make it so that your keyboards and monitors are in control of a single session.

The screen host should do the following:

Ctrl + A then type:
```
:multiuser on
```

Ctrl + A then type:
```
:acladd guest
```
where guest is the name of the user you want to let in your session

Then the screen guest can join using the following command (which will work if there's a single multi-user session open):


screen -x username/

Alternative: VNC can be used only for graphical remote connection sharing.

2. Copy and paste

It's not always easy to copy and paste text in a terminal. For one, if you're on a getty console (the black screen with the login prompt and no graphics), you can't even use your mouse (usually). If you have a crappy X terminal, you might have the problem that wrapped lines are cut by newlines or it's hard to select the output if it spans more than a single screen... Whatever it is, with screen you can cut like a pro without even using the mouse.

Ctrl + A, then pressing "[" will enter copy mode. When in copy mode, you navigate around using the vi key shortcuts (you know the vi shortcuts, otherwise you wouldn't be reading about working in terminal sessions, right?).

Press space once to mark the beginning and a second time to mark the end of the snippet you want to cut.

Ctrl + A, then "]" will paste the copied text.

Did I mention screen can also copy rectangular blocks of text? Reading about it in the manual is left as an exercise for the curious reader.

Alternative: X-Server's select/middle-click can be used- only if you're running in an X session, though.

3. Log output

We humans are not good at remembering stuff, that's what computers are for. To record exactly what you have typed and what was the output, start logging using Ctrl + A, then H, and the same sequence to stop logging. The logging session is written to screenlog.0 if you're recording in the first window, screenlog.1 in the second one, etc.

Alternative: The UNIX command script will start a new shell and record all keystrokes and output in the file typescript.

4. Monitor for activity

Let's say you've started a long-running command and you're waiting for it to finish while you're busy typing in another window. Switching the window periodically just to check what's going on quickly becomes annoying, so you type Ctrl + A, then M and you're set- screen is watching the console for you. If anything changes in this window, you will see a notification in the status line at the bottom.

Alternative: Many linux terminals can monitor for activity, e.g. konsole- but they're graphical and need to run in an X session.

5. Lock screen

It's often useful to lock the screen session without detaching, especially when using a multi-user session. Ctrl + A, then x, and you're done. Type your password and your session is available again.

Alternative: Popular desktop environments use xscreensaver, but again- this only works in X sessions.

Where to go from here?

If you like GNU screen, you might also try tmux. According to the site, "tmux is intended to be a modern, BSD-licensed alternative to programs such as GNU screen". If you're interested, check out this detailed tmux blog post.

If you want to take screen to the next level, you might give tiling window managers a try. They do for the graphical environment what screen does for the terminal. The idea is that you don't resize your windows, but the window manager does that automatically by partitioning the desktop area in adjacent windows. Most of these also try to use keyboard shortcuts extensively and obviate the need of a mouse. I'm currently impressed by xmonad, although I've heard nice things about the awesome window manager as well (yes, that's its name).

Testing with Lift's TestFramework

2010-08-10T23:01:00.010+03:00

Getting started

HTTP testing is sufficiently tedious that some folks don't do it. Even if we do it, in Java it doesn't look as pretty as it could be.

The Scala Lift web framework, apart from the other advantages it has (which are a topic for many a blog post) offers some syntax-sugar wrappers so that testing of our REST APIs can be concise and to the point.

Combined with Jetty, this leads to some seriously short and readable code.

First, we need to:

use the trait "with TestKit" in our test

override the baseUrl property

start our Jetty server

Then if we want to output our own failure message later, we need to provide an implicit class of type ReportFailure, where we only need to implement the fail method, which predictably takes a String. For example, regardless of whether we use ScalaTest or Specs, we can fall back to the fail method which is inherited by our test class:

This is all we need so far.

Using it

Now we're ready to GET some action.

There's already a lot going on here. First of all, the get and post methods return a TestResponse (you don't see it, because it's inferred). And the nice thing about TestResponse (and Scala) is that since it implements the foreach method, it can be used in for expressions together with the "<-" symbol.

Whatever is extracted by the for expression can be used to issue get and post requests, but in the context of the previous request. This means that cookies are preserved, so you can use the same HTTP session. In this contrived example we use an httpClient as an argument to the get method (which is supposed to use some HTTP credentials and log us in). In the following call, we don't need to use the authenticating http client, because we can have our cookies (and eat them, too).

We can also use http parameters as a sequence of tuples, which can be delimited using the "->" operator.

For any TestResponse, we can use the xml property, which is the XML as a scala.xml.Elem wrapped in the (ubiquitous for Lift) Box.

Finally, there's concise syntax to assert certain properties about the response. For example the "!@" operator checks if the response code is 200, otherwise it fails with the error message specified after the operator.

If we expect a specific return code other than 200, we can also specify it explicitly:

Customizing

If you want to define a HTTP basic authentication client to be used by default by your requests, you can override the method theHttpClient. Lift's TestFramework provides the buildBasicAuthClient method, which can be reused to quickly create an HTTP client with a set user and password.

If you're like me, you might lose hours trying to find out why the request doesn't work when the server doesn't explicitly request authentication. Then setting the preemptive flag on the client would definitely save some time.

I know some of you Scala geeks are already bored because everything so far is just so simple. There wasn't even any mention of implicits! But rest assured, I will find some use for implicits.

Pimping

The usage snippet in the for expression above is not quite right. If the response contains no valid xml, then the specs matcher will not be executed and the test will not fail, although it should. Here we can use the fact that TestResponse also implements the filter method, which allows us to have if expressions (also known as filters). The post snippet could then be rewritten like this:

We are again using the Specs matcher to test if the response contains xml and fail otherwise. Notice that this expression does not return true- it need not be boolean. This is due to the fact that TestResponse's filter method intentionally returns Unit, not Boolean.

However, that's too much boilerplate, which we have to do for every time we want to check the contents of an xml response:

Check if the response is valid and has a response code of 200

Check if the xml is not empty

Extract the xml

Only then check the contents of the XML

If only we could make TestResponse do what Specs does with XML... But since this is Scala, we can create a wrapper and then transparently substitute it using our implicit conversion:

An important thing to remember about implicit conversions is that if we want them to feel seamless, the wrapper class' methods should return the original class type (the one before the conversion). Otherwise we could get unexpected object classes similarly to what occurred with the Scala 2.7 RichString, for instance with "aaa" reverse == "aaa"

Since we have chosen to use the same operator as Specs uses, we had to disambiguate with the XmlBaseMatchers class name, which is the specs trait containing all the XML matching goodness.

Now we can use our fancy TestResponse operators. Notice that since we return the same class type that we already had, we can chain:

This looks more compact than the original version so you can focus on the API differences and not be distracted by boilerplate preconditions.

Pomodoro- what's in it for me?

2010-03-12T17:14:00.009+02:00

Lately I've been trying to use the Pomodoro technique. I have been using time boxing for a longer time and find Pomodoro to be a simple and efficient framework which revolves around time boxing.

Pomodoro being so simple, I was rather surprised to find a critique of Pomodoro recently. The critic ignores the fact that the Pomodoro author doesn't mandate that Pomodoro should be used in every occasion; that Pomodoro may use a different time unit than 25 minutes; or that you can void any Pomodoro if you see fit. But let alone the fact that the author misinterprets how strict the Pomodoro should be. By focusing on productivity, it's easy to ignore an equally important aspect- Pomodoro also helps relieve stress and prevents burnout.

Let me try to explain how Pomodoro achieves this by describing the basic building blocks of the Pomodoro process, which applies to time-boxing in general.

What is time boxing? Simply put, it's deciding what to do in the next fixed short period of time- and sticking to it. This consists of the following key stages and transitions:

1) at the start of the Pomodoro- comitting to one thing only for the next time period

2) during the Pomodoro- trying to focus on this one thing without interruptions

3) at the end of the Pomodoro- wrapping up and detaching from the work, followed by a short break

Now let's enumerate some of the reasons for stress, anxiety and burnout. Among these are: multitasking, procrastination, interruptions and obsessing over the completion of a task.

Comitting to one thing is crucial both for a productive and stress-free state of mind. Multitasking is proven to lead to anxiety and inefficiency time and time again. Why? First of all, because switching contexts is very inefficient and exhausting at that. Getting into the right set of mind to do a task takes time and effort and switching to another task destroys all that mental preparation.

Multitasking also leads to the impression that we're not acomplishing anything. If you have followed Joel's article on the subject, the explanation is simple: if you do 2 tasks which take 10 minutes to finish sequentially, you will have a finished task in 10 minutes. If you switch the tasks every minute, you will only get the first task done after 19 minutes! Multiply this by the number of tasks you're trying to switch every day and you get the picture.

The conclusion is that multitasking takes more effort and gets things accomplished slower. If this is not frustrating, I don't know what is.

Furthermore, giving a time limit for a simple task helps you get started. It encourages decomposing the problem into subtasks and transforms a huge amorphous blob of work look like something more palatable. More importantly, getting started is the best way to defeat procrastination, and procrastination is also a major factor for anxiety. Now which is more stressful: a heap of work, where you don't know where to start, or neat organized small tasks, each of which is easier to estimate? Try for yourself.

Interruptions are another very frequent source of frustration. An extremely important but often forgotten type of interruptions is internal ones. Keeping track of one's desire to chat with someone, check your email or just browse the web for something interesting which has popped in your mind is hard to resist, and can break your flow. It is, however, much easier to control these urges if you know that a break is coming soon, rather than when you feel that it's all work all day long, and a small diversion will just take a couple of seconds. Except that it doesn't.

Interruptions from other people are harder to avoid, especially in a job, which is related to responding to different events, like answering a support hotline. However, even in the case of phone calls, you cannot be in two unrelated conversations at once. Besides, the criticism of Pomodoro listed two examples which are notably free from interruptions: a surgeon in an operation or a lawyer defending a case in court. You don't need to control task switching in these scenarios because the situation does not allow for any other tasks to be performed. Can you imagine a lawyer in court or an operating surgeon surf the web or check their mail? Thought so.

Finally, it is very easy to forget that the slow but steady runner wins the race. We often lose track of how much time and effort we have been spending on a task. The short length of the Pomodoro (or any time boxing technique) helps you take a step away and ask: where am I going with this? Is it taking longer than anticipated? Am I actually doing anything or going in circles? Taking a break and detaching yourself from the task at hand helps you see the bigger picture and backtrack if you've reached a dead end. Besides, taking a break will often let your subconscious find a solution, which is otherwise not obvious. Obsessing on completing the task is counterproductive and will leave you unable to take on the next task and burned out in the long run.

There are other benefits of time-boxing, but all in all, it is a very useful technique. Even people who claim they are able to concentrate without taking breaks, are often doing it subconsciously by letting their minds meander from time to time or switching for a couple of minutes to less stressful aspects of the task. But why rely on being a born time-boxer? Once you agree that time-boxing is helpful, it pays off to make it a habit. It will make you more concentrated and calm.

And remember- if it ever feels like you're more stressed by using time-boxing, don't push yourself too hard and stop it. Take a break; enjoy your own flow.

Is Scala more complicated than what Java tries to become?

2010-01-17T15:58:00.013+02:00

Is Scala more complicated than Java? My last post did not tell the whole truth. I've only listed Scala features, which have a Java analog. There is a glaring omission of advanced Scala features like implicit conversions, operator overloading, call-by-name parameters and pattern matching. These Scala features are more complicated than what Java has. There, I said it. But then Scala is more complicated in the way a calculator is more complicated than an abacus- sure you can do some of the same stuff with an abacus, but trying to calculate the square root of a number is much more cumbersome.

However, this complexity pays off, because it lets us simplify many day-to-day features. This post will try a different angle by comparing where Java wants to be and where Scala is right now. I hope after reading it you will at least question your assumptions whether this trade-off is worth it.

Upon its creation, Java was a fairly simple language. A major reason it took over C++ is because it was specifically designed to steer away from multiple inheritance, automatic memory management and pointer arithmetic. But it's not a simple language anymore, and it's getting more and more complicated.

Why? Java wasn't designed to be too extensible. Scala, on the other hand, was designed to be scalable, in the sense of flexible syntax. The very creators of Java knew very well that a "main goal in designing a language should be to plan for growth" (Guy Steele's famous words from Growing a Language)

We need to expand our definition of language complexity. The language needs to be able to abstract away accidental complexity, or using it will be difficult. Examples of accidental complexity: jumping to a position in your program with goto, and then remembering to go back (before procedural programming); or allocating memory, and then remembering to deallocate it (before garbage collectors). Another example: using a counter to access collections, and remembering to initialize and increment it correctly, not to mention checking when we're done.

Creating extensions of the language in order to hide these complexities doesn't happen often. When it does, it offers huge rewards. On the other hand, if a language is rigid, even though it looks simple, this forces you to invent your own arcane workarounds. When the language leaves you to deal with complexity on your own, the resulting code will necessarily be complicated.

Let's see what special new language features Java tries to add to the language, which Scala can do because of its flexibility and extensibility.

Pattern matching

Pattern matching is often compared with Java's switch/case statement. I have listed pattern matching as something which doesn't have an analog in Java, because comparing it to "switch" really doesn't do it justice. Pattern matching can be used for arbitrary types, it can be used to assign variables and check preconditions; the compiler will check if the match is exhaustive and if the types make sense. Meanwhile Java has only recently accepted Strings in switch statements, which is only scratching the surface of Scala's pattern matching.

Furthermore, Scala is using pattern matching all through the language- from variable assignment to exception handling. To compare, the proposal for handling multiple exceptions in Java is postponed yet again.

Case classes

In order to get rid of Java's verbose getters, setters, hashCode and equals, one solution is to muck with the javac compiler, like the folks from Project Lombok have done. Is going to the guts of javac complicated? I'm sure it is.

In Scala, you can do it if you just define your classes as case classes.

Implicit conversions

In short, implicit conversions help transparently convert one type to another if the original type doesn't support the operations requested.

There are many examples where this is useful.

What in Java is hardcoded in the language as conversions and promotions, in Scala is defined using implicit conversions. This is another example where Java can get quite complicated. In most cases where you need to decide how to convert a method argument, for instance, you must have in mind narrowing and widening conversions, promotions, autoboxing, varargs and overriding (whew!). In Scala, the advantage of having implicit conversions is that you can inspect the code, where no ambiguity can result. You can analyze the conversions taking place in the interpreter by supplying the "-Xprint:typer" parameter. You can even disable these implicits, if you don't like them, by shadowing the import.

Another example of what implicits can do is adding methods and functionality to existing classes. Dynamic languages already do that easily using open classes and "missing method" handlers. In Java one way to do this using bytecode manipulation trickery via libraries like cglib, bcel, asm or javassist.

Bytecode manipulation in Java is required for popular libraries like Hibernate, Spring and AspectJ. Few "enterprise" Java developers can imagine development without Hibernate and Spring. Although there are many more things you can do with AspectJ, it can be used to emulate implicits with type member declarations. However, even though using AspectJ is a more high-level way to solve the problem, it adds even more complexity, as it defines additional keywords and constructs.

If you're new to Scala, you don't lose much if you don't know how implicit conversions work, just like you don't need to know about the magic that happens behind the scenes when Hibernate persists objects or when Spring creates its proxies. Just as with bytecode generation, you're not advised to use this feature often, as it is difficult to use. Still, you'll be glad it exists, because someone will create a library which will make your life and the life of many developers so much easier.

Operator overloading

The line between operators and methods in Scala is blurred- you can use the symbols +, -, /, *, etc. as method names. In fact, that's exactly how arithmetic operators work in Scala- they are method invocations (relax, everything is optimized by the compiler).

Some people object that operator overloading adds unnecessary complexity, because they can be abused. Still, you can also abuse method naming in much the same way. For instance, some hapless folk can define methods with visually similar symbols, like method1, methodl and methodI. They can use inconsistent capitalization, like addJar or addJAR. One could use meaningless identifiers like ahgsl. Why should operator best practices be different than method naming best practices?

What is complicated is treating numeric types like ints and BigInteger differently. Not only that, but operations with BigInteger are very verbose and barely readable even with simple expressions. To compare, this is how a recursive definition of factorial looks like in Scala with BigInteger:


def factorial (x: BigInt): BigInt =
  if (x == 0) 1 else x * factorial(x - 1)

This is how it would look if Scala didn't support operator overloading:


def factorial (x: BigInteger): BigInteger =
  if (x == BigInteger.ZERO)
    BigInteger.ONE
  else
    x.multiply(factorial(x.subtract(BigInteger.ONE)))

Call by name

One of the proposals for Java 7 language extension was automatic resource management. This is one more rule to the language, which you need to remember. Without this feature, code is also unnecessarily complicated, because it forces you to remember to always close resources after using them- if you slip up, subtle bugs with leaking files or connections can result.

In Scala, it's easy to add language constructs like this. Using function blocks, which are evaluated only when they are invoked, one can emulate almost any language construct, including while, if, etc..

Existential types

Existential types are roughly an alternative to Java wildcards, only more powerful.

Martin Odersky: If Java had reified types and no raw types or wildcards, I don't think we would have that much use for existential types and I doubt they would be in Scala.

If Martin Odersky says that existential types wouldn't be in the language if it wasn't for Java compatibility, why would you even need to know about them? Mostly if you need to interoperate with Java generics.

Conclusion

Scala tries to define fewer language rules, which are however more universal. Many of these advanced features are not often used, but they pay off by allowing to create constructs, which in Java would require specific hardcoded additions to the language. In Scala, they can be defined simply as libraries.

Why does it matter that it's in the libraries, and not hardcoded in the language? You can more easily evolve and adapt these features, you can add your own extensions, and you can even disable some of the library parts or replace them.

The conclusion is that if a language is not designed to be extended, it will eventually develop features, which are not well-integrated and this language will collapse under the weight of its own complexity.

Finally, learning something so that you avoid a lot of routine error-prone operations reduces effort by increasing the level of abstraction, at the cost of additional complexity. When you were in school, it was complicated to learn multiplication, but if you got over it, it would save you from quite a bit of repetition than if you just used addition.

P.S. I realize it's not possible to resolve the issue once and for all which language is more complicated- Java or Scala- in a post or two. First of all, have in mind that simple is not the same as easy to use. There are also many topics which are open for discussion. I haven't touched on Scala traits; I haven't mentioned functions as first-class constructs compared to the Java 7 closure proposal; and there's a lot that can be said about how Scala obviates many Java design patterns. Extending the Scala syntax via compiler plugins is another interesting advanced topic.

I suppose someone could even write a blog post about these topics some day.

Is Scala more complicated than Java?

2010-01-04T20:46:00.001+02:00

One Scala-related thread on Artima drew a lot of attention: "Is Scala really more complicated than Java?". This post really struck a nerve. Whoever claims that Scala is much more complicated than Java has clearly not seen a Java Programmer Certification in a while and is probably not using many new features since Java 5 came out.

What I'll try to prove in this post is not that Scala is not a complicated language. There are certainly many languages which are simpler. The core features which are used reasonably often are indeed a simplification over Java. Scala also has features which are more complicated than what Java has. However, the complicated Scala features are more specialized at extending the language while the complexity of Java is usually imposed on everyone including the beginner.

This post will also not try to describe the language features in exhaustive detail- that's what the language specification tries to achieve, and the blog post is already long enough. I will assume that you know about the core language rules or can easily look them up.

What is complexity? Many conflate it with unreadability, some say it's the opposite to ease of use. Let's start with the following definition of complexity: it's the many special exceptions (pun not intended) to the rules, making the whole system difficult to understand and remember.

Based on that definition, let's run a comparison of language features of Java and Scala.

Keywords

Java has more reserved words than Scala. Even when we remove the words of primitive types (boolean, int, float, etc.) Scala still has less keywords!

Scala misses some of Java's control structures

continue

break

if/then/else

for loop

for

Scala keywords not in Java

override

@Override

override

Access modifiers

Java has four access modifiers: default (package), private, public and protected. Scala has three, but it can combine them with a scope for protection. This flexibility allows you to define access control, for which Java has no analogs. They are also readable because they look self explanatory. For instance, if you have the class org.apache.esme.actor.UserActor, these are the Scala equivalents for Java's access modifiers:

private[actor]: Same as package access in Java
private[UserActor]: Same as private in Java

Scala's default access is public (yay, one less keyword!).

On the other hand, by defining the scope, Scala allows visibility types, which Java doesn't have:

private[this]: the member is accessible only from this instance. Not even other instances from the same class can access it
private[esme]: access for a package and subpackages. How many times did you have to use clumsy workarounds in Java because subpackages couldn't access the parent package?
protected: only subclasses can access the field. This is more consistent than Java's protected keyword. In Java, you have to remember that both subclasses and classes in the same package can access the field. How's that for remembering arbitrary rules?
private: This will not allow you to access inner objects. This is also more consistent than Java's private access, which is perhaps indicative of the fact that inner objects in Java were bolted on after the first version of the language was created

Namespaces

Java has four namespaces (fields, methods, packages, types), Scala has two (one for types, one for everything else). This has important implications when overriding. In Scala, you can start with a parameterless def (method definition) and then override with a val (immutable variable). This helps enforce the uniform access principle: whether you access a field or a method, this doesn't restrict you later on, because the syntax is the same everywhere you access it.

One thing which you cannot do is define a variable and a method with the same method. The "Programming in Scala" book explicitly mentions that allowing this would be a code smell because of possible code duplication and the ambiguity which could arise.

Types and the type system

Scala's type system has been criticized for being too complex, but let's have a look at Java's type system.

Primitive types

There are many exceptions in Java's type system, which make the language hard to use and evolve. Java has primitive types, while in Scala everything is an object, making this valid code:


(1 to 3) foreach println

Java's primitive types make a lot of things harder- before Java 5, manual wrapping in collections, e.g. int in Integer, boolean in Boolean, etc. was a pain. After auto-boxing and unboxing came out, the syntax is cleaner, but the overhead remains, as well as some very subtle bugs. For example, auto-unboxing a null wrapper type will cause a NullPointerException.

There is a lot of code duplication because of primitive types- you always have to specify special cases and can't be truly generic.

Generics

Scala has generics done right. For instance, you can define covariance at the declaration site, whereas Java requires you to do this at the call site. This, combined with Scala's type inference allows one to use generified libraries without having to know or define the complete type signature.

Java has yet another "special" type: arrays. As a result the rules for the underlying array type and the generified ArrayList are quite different and inconsistent. The type of arrays is checked at runtime, while the genericity of ArrayList is checked at compile-time. As a result, inappropriate assignment to an array element results in a ArrayStoreException only at runtime.

Constructors

Java initialization order in object construction is a pain to get right. The puzzlers on the certification exam use the most bizarre mix of static and instance initializer blocks and constructors calling or inheriting other constructors.

In Scala, any code which isn't part of a method or member variable declaration is executed as the primary constructor. You can define auxiliary constructors which call either the primary one or another auxiliary one defined in the same class before it. Can you come up with anything simpler?

Uniform syntax

Scala is sometimes accused of using too many symbols. Whatever you've seen, it's mostly not part of the language, but of the libraries. You can override them and even disable them. What does Java have to say about special symbols?

Arrays

Arrays in Java are accessed via square brackets. In Scala, parentheses are used, because accessing an array index is a method call (called apply). Don't worry though, the compiler optimizes this away.

Collection literals

Java has a lot of special syntax for array instantiation and will soon have ones for instantiating lists, sets and maps. Scala, again, creates these special collections using factory methods in the companion objects: Array(1,2,3), List(1,2,3), Map((1,2)). Lists can also be created using the cons "operator", but here's the trick: it's not actually an operator. It's a method, appending to the list. You can also create a map using the arrow tuple syntax: Map(1->2). And again, this is not "special" syntax, which is part of the language- it's a method, constructing a tuple!

Now someone might smirk and think: "Ha, gotcha! Do you mean to say that simply because you've pushed the complexity out of the language and into the libraries you don't need to deal with it?". True, but let's have a look at Java and its ever growing standard libraries. It has AWT and Swing. It has old I/O and new I/O (gosh, which one do I use?). It has RMI (do you remember RMI?) and OMG, it even has CORBA. These libraries will never die. Methods in Thread have been deprecated for ages. There's also no sign that the ill-conceived Date/Calendar classes will ever be removed, but you still must know JodaTime if you hope to get any job with dates done.

More importantly, extending the language easily helps abstract away the details and evolve the language without creating tons of special cases. As per our definition, special cases add up to increase complexity. We'll explore the topic of extending the language in the followup post.

Bridge-crossing with Clojure

2009-12-28T00:24:00.005+02:00

Solving puzzles has become popular among colleagues at the end of this year. One of these was the popular Bridge and torch problem. To summarize: four men have to cross a bridge in the dark with a torch, which can only light the way for two persons. These guys can cross the bridge in 1, 2, 5 and 10 minutes, respectively. When two people cross the bridge, it takes the time needed for the slower one to pass.

I'm not going to explain the optimal solution, but I wanted to show how this is a graph-traversal problem, which is amenable to solving using Dijkstra's algorithm. I also noticed that some clever guy has solved the puzzle using Prolog. So an idea was formed: generate the tree of solutions using a language, which I'm not really comfortable with (so I can practice and learn it better). I settled on Clojure, because I wanted to learn it this year (and the year is almost over).

The solution has a lot to do with sets. The state of each of the banks (which I'll call left and right) is a set of people, which are identified by their crossing speed. If we generate the list of possible combinations between the persons for each next move and recurse into them, then we'll have the tree of all solutions. In order to avoid indefinitely long (and stupid) solutions, we assume that going forward always takes two men, and going back takes only one.

The necessary libraries are set, combinatorics and str-utils. That's where I start:


(use '[clojure.contrib.combinatorics])
(use '[clojure.set])
(use '[clojure.contrib.str-utils])
(use '[clojure.contrib.pprint])

Here are the initial data structures: the two river banks are sets, the right one is empty:


(def left #{1, 2, 5, 10})
(def right #{})

And here's a first version of the solution:


(defn forward [left right steps minutes]
        (map #(back
                (difference left %)
                (union right (set %))
                (cons % steps)
                (+ minutes (reduce max %)) )
            (combinations left 2)) )

(defn back [left right steps minutes]
    (if (empty? left)
        (list steps minutes)
        (map #(forward
                (union left (set %))
                (difference right %)
                (cons % steps)
                (+ minutes (reduce max %)) )
            (combinations right 1)) ) )

(pprint (forward left right nil 0))

This prints some semblance of a tree, constructed by nested lists (hey, that's what Lisp is good for, remember?). What I didn't like about this is that there is too much repetition. There are many parts of these functions, which look the same. In order to evolve the solution, I wanted to extract the common parts in a single function:


(defn solve [from to steps minutes group]
    (if (and (empty? to) (not group))
        (list steps minutes)
        (mapcat #(solve
                (union to (set %))
                (difference from %)
                (cons % steps)
                (+ minutes (reduce max %))
                (not group) )
            (combinations from (if group 2 1))) ) )

(pprint (solve left right nil 0 true))

The function behaves differently depending on whether the group of people crossing consists of two or just one person. Also, the tree is now flattened, so the optimal solution can be easily found like this:


(print (reduce #(if (< (second %1) (second %2)) %1 %2) (apply hash-map (solve left right nil 0 true))))

Next step, I want to be able to collect a lot more data about each state, so I need to stop messing around with lists of lists and use something more organized like a hashmap. There's a reason Clojure includes the function second, but not third or, heaven forbid, fourth. Naming each part of your ad-hoc data structure plays a major role in readability.


(defn solve [from to steps minutes group]
    (if (and (empty? to) (not group))
        {:steps steps, :from from, :time minutes}
        {:steps steps, :from from, :time minutes,
            :solutions (map #(solve
                    (union to (set %))
                    (difference from %)
                    (cons % steps)
                    (+ minutes (reduce max %))
                    (not group) )
                (combinations from (if group 2 1)))} ) )

So now the interesting part can begin- rendering the tree in a nice graphical form using GraphViz. Watch this:


(defn printgraph [solution_tree]
    (let [steps (solution_tree :steps)
        nodename (if (nil? steps) "root" steps)
        from (solution_tree :from)
        label (if (= (count from) (count left)) (solution_tree :time) from)]
        (str "\"" nodename "\" [label=\"" label "\"];\n"
            (if (contains? :solutions solution_tree)
                ""
                (str-join ""
                    (map #(str "\"" nodename "\" -> "
                            "\"" (:steps %) "\""
                            " [label=\"" (intersection from (:from %)) "\"];\n"
                            (printgraph %))
                        (solution_tree :solutions)) ) ) ) ) )

And now you can output the recursive stuff wrapped with the header and footer like this:


(println "digraph G {")
(print (printgraph (solve left right nil 0 true)))
(println "}")

This graph shows each node as the state of alternating river banks, and the edges show the people crossing the bridge. The leaves display the number of minutes which it took for this particular combination of crossings.

I also wanted to generate a mind map for FreeMind, but since it doesn't support edge labels, I used the persons crossing for the nodes:


(defn mindmap [solution_tree]
    (str-join "\n"
        (map #(str
                "<node text=\"" (intersection (:from solution_tree) (:from %)) "\">"
                (mindmap %)
                "</node>")
            (solution_tree :solutions)) ) )

Now generating the complete file according to the FreeMind format:


(println "<map><node text=\"root\">\n")
(print (mindmap (solve left right nil 0 true)))
(println "</node></map>\n")

Finally, finding the optimal solution is just a wee bit more involved, because it requires traversing the tree recursively, but that's no problem for us anymore:


(defn optimal [tree]
    (if (contains? tree :solutions)
        (reduce #(if (< (second %1) (second %2)) %1 %2) (map optimal (tree :solutions)))
        (list (tree :steps) (tree :time)) ) )

(print (optimal (solve left right nil 0 true)))

In conclusion, I knew that one can write very compact code in Clojure (and any Lisp, in general), and this was certainly true here. I didn't use the Clojure-specific concurrency primitives or the lazy sequences, so my example would probably look similar in another Lisp. I did try to make it as functional as possible and Clojure does encourage that.

Some of the difficulties I had could be solved by static type checking, but I guess I'm spoiled by Scala. I've ran into small problems like renaming a variable and forgetting to rename (or misspelling) some instance of it in my code. I also tend to mix up which of the arguments to contains? comes first, and which second. This is probably because you can exchange a symbol as a key in a hashmap, and the hashmap without changing the meaning, e.g. (:from solution_tree) and (solution_tree :from). Of course, this is a problem with dynamic typing, not of Clojure in particular (and could probably be interpreted as a problem of using the language inappropriately).

Whatever the small hurdles, the problem was solved. Having fun? Check. Generating nice graphs and mind maps to impress colleagues? Check. Learning a bit of Lispy Clojure in the process? Check. Mission accomplished.

String interpolation in Scala

2009-12-03T22:58:00.003+02:00

When I tried Scala for the first time, one of the first missing language features I noticed was string interpolation (you guessed it, I was using Ruby at the time). Of course, this is a small convenience rather than a major language feature and usually String.format and java.util.Formatter are pretty good substitutes. Still, string interpolation comes now and then in Scala discussions and one has to wonder if Scala's powerful language extension facilities can emulate a feature like this.

It turns out you can get reasonably close:

trait InterpolationContext {
  class InterpolatedString(val s: String) {

    def i = interpolate(s)
  }

  implicit def str2interp(s: String) = new InterpolatedString(s)

  def interpolate(s: String) = {
    val sb = new StringBuilder
    var prev = 0
    for (m <- "\\$\\{.+?\\}".r.findAllIn(s).matchData) {
      sb.append(s.substring(prev, m.start))
      val matchString = m.matched
      var identifier = matchString.substring(2, matchString.length - 1)
      try {
        val method = this.getClass.getMethod(identifier)
        sb.append(method.invoke(this))
      } catch {
        case _: NoSuchMethodException =>
        sb.append(matchString)
      }
      prev = m.end
    }
    sb.append(s.substring(prev, s.length))
    sb.toString
  }
}

object Test extends InterpolationContext {
  val a = "3"
  def main(args: Array[String]) {
    println("expanded: ${a}; not expanded: ${b}; more text".i)
  }
}

How does this work? If you call the i method on a String, it will force implicit conversion to a InterpolatedString, similar to how the r method works for converting to a Regex. The interpolate method uses reflection to find a method with the name equal to the identifier delimited by "${" and "}". This works both for vals and defs due to Scala's adhering to the uniform access principle. If the delimited string is not a valid identifier or a parameterless method with this name doesn't exist, it is not substituted, but stays as it is.

How do you use it? Just extend or mix in the InterpolationContext trait in the class where you want this to work, and then call i on the Strings where you want interpolation to work. The InterpolationContext trait serves two purposes- first of all, it imports the implicit conversion to the interpolated string, and second it provides access to the methods of the current object via reflection.

The limitations of this method are that interpolation only works on member vals and defs of the current object only. I rather like this, because you know you can't accidentally construct a String from user input in an html form like "Name: ${System.exec(\"echo echo pwned >> .bashrc\")}". Also, interpolation doesn't work for private members as well as local variables. Finally, you have to both mix in or extend the trait and call a method on every String (even though it's a one-letter method). This is not too bad, because you can control the scope where interpolation works and therefore avoid and debug nasty surprises easier.

I don't see this as a pattern, which can be used in real-life projects, and I don't see the use of implicits here worthy of emulation. Nevertheless, trying to copy features from other languages is instructive about the limitations of your current language of choice and the tradeoffs you get for these features. Last but not least- even if it's a gimmick feature, implementing it was fun.

Error detection with pattern matching

2009-11-23T22:36:00.005+02:00

Scala's pattern matching is one of the most powerful features in the language. It not only helps write concise and very readable code, but also helps prevent trivial errors.

Let's say you want to save a line or two of code when comparing for some empty data structure like a List. You decide to use comparison for equality:


val items = Nil
if (items == Nil) println("No items")

Then you decide to refactor and turn the items collection into a Set. The Scala compiler is clever and should give you a warning, right? Well, not quite:


val items = Set()
if (items == Nil) println("No items")

This results in nothing printed, as the expression evaluates to false. Of course, given the types it's perfectly clear at compile-time that this will never be true. Can't the compiler give you a hint? Indeed, it will if you use pattern matching:


items match {
  case Nil => println("No items")
  case _ =>
}

This will result in the following error message:


error: pattern type is incompatible with expected type;
 found   : object Nil
 required: scala.collection.immutable.Set[Nothing]
         case Nil => println("No items")
              ^

The examples are contrived (unparameterized Nil and Set?), but you get the point. Yes, Nil can never be a value of the Set type. You might think this is fairly obvious, but pattern matching can be a life saver when implicit conversions are involved. Consider this:


"heh".reverse == "heh"

What does this evaluate to? This should be obvious, right? But the value is false! If you used pattern matching, you would easily see why:


"heh".reverse match {
  case "heh" => "obvious?"
}

This will make the compiler very nervous, and this is the reason why:


error: type mismatch;
 found   : java.lang.String("heh")
 required: scala.runtime.RichString
         case "heh" => "obvious?"
              ^

So reverse converts the String to the wrapper RichString, which is not the same type as String.

I have had similar problems detecting a bug where I was checking for equality with None a variable which was of type net.liftweb.common.Box (a type very similar conceptually to Scala's built-in Option).

This made me adopt a general rule to prefer pattern matching rather than equality comparison. The bugs it catches are sometimes subtle and hard to see, and that's exactly what Scala's rich static type checker tries to avoid. Use it to your advantage.

Since we're talking about bugs caught by pattern matching, there's one subtle bug, which is often (though not always) caught by the compiler (another contrived example follows):


val items = Set()
Set() match {
  case items => println("empty")
  case _ => println("full")
}

This will result in an error, which looks a little bit unusual to the newbie:


error: unreachable code
         case _ => println("full")
                   ^

This error is usually crying out loud: hey, you're inadvertently using variable binding instead of constant matching! The newly bound variable items shadows the existing variable with the same name and all other cases after it will never match.

One way to fix it is to use backticks to prevent the name to be bound to a new variable:


Set() match {
  case `items` => println("empty")
  case _ => println("full")
}

As a rule of thumb it is advised to use CapitalLetters for case classes and constants which you intend to pattern match.

This error wouldn't have occurred if you used equality comparison in the first place, but even in mildly complex cases pattern matching trumps plain equality checking in readability and detecting errors. Apparently, there are cases where pattern matching fails (for instance, matching structural types), so there's still no reason to deprecate good old "==". But there are many more errors, which pattern matching catches, like checking if the match is exhaustive. So there's no point in saving a couple of characters but lose the type safety you expect from Scala.

Embedded Scala interpreter

2009-11-04T23:04:00.004+02:00

The Scala interpreter is proof that a statically typed language can have features most people only expect from a dynamically-typed language. One of the cool uses of an interpreter is embedding it within your application. This allows you to conveniently experiment with Scala and probably even interact with object instances in your running system. I won't explain how the interpreter works here, but I will try to show you a simple way of embedding the interpreter.

As it usually happens, someone beat me to it. Josh Suereth explains in great detail how to embed an interpreter, but he has done so many customizations that his solution would probably fit on several pages.

I wanted a simpler solution which one could understand at a glance. The code for Lord of the REPLs is much shorter although it doesn't customize much of what the standard interpreter offers.

I tried to come up with the shortest working version you could possibly get away with. Provided I create the settings properly, this is what I could muster:


val out = new java.io.StringWriter()
val interpreter = new scala.tools.nsc.Interpreter(settings, new PrintWriter(out))
interpreter.interpret(code)

Not too much code, is it? (Half of it is probably due to the full package names). Now you could collect your output from the "out" stream and probably convert to String if you need using "out.toString".

Not so fast, though. I said this works if I have the appropriate settings:


val settings = new scala.tools.nsc.Settings()

The problem here is that the interpreter doesn't find two of the crucial jars needed for its proper functioning: scala-compiler.jar and scala-library.jar. When it doesn't it spits out the following error:


scala.tools.nsc.FatalError: object scala not found.

Thanks to the following discussion by Eric Torreborre (author of Specs) I managed to find out that one needs to add to the bootclasspath of the settings object:


val origBootclasspath = settings.bootclasspath.value
val pathList = List(jarPathOfClass(compilerPath),
                         jarPathOfClass(libPath))
settings.bootclasspath.value = (origBootclasspath :: pathList).mkString(java.io.File.separator)

One could hardcode the path to these two jars, but that's not too flexible. If we want to do it right, we might create a function which discovers the path to the jar from the name of a class that's in it:


def jarPathOfClass(className: String) = {
  Class.forName(className).getProtectionDomain.getCodeSource.getLocation
}

Now you could find the paths to these jars like this:


val compilerPath = jarPathOfClass("scala.tools.nsc.Interpreter")
val libPath = jarPathOfClass("scala.ScalaObject")

I've read that getProtectionDomain.getCodeSource returns null in some classloaders and might have problems specifically with OSGi. In that case, one might need to resort to the following hack:


def jarPathOfClass(className: String) = {
  val resource = className.split('.').mkString("/", "/", ".class")
  val path = getClass.getResource(resource).getPath
  val indexOfFile = path.indexOf("file:")
  val indexOfSeparator = path.lastIndexOf('!')
  path.substring(indexOfFile, indexOfSeparator)
}

With the last ugly piece of code creating an interpreter is not so concise anymore, but sometimes you can't be both robust and concise.

If you want to see the above snippets assembled in one piece you can check out Apache ESME's source code for the ScalaInterpreter action.

Warning: interpreting code directly in your application is a huge security risk and might not always be a good idea.

Scala closures as poor man's mocks

2009-10-13T12:07:00.008+03:00

Groovy has this feature that you can use a closure whenever you need an interface with one method only. A class implementing the interface is automatically created, and the closure provides the implementation of this single method. This process is called closure coercion and is particularly convenient to make tests readable and concise.

I'm not yet sure about the relative advantages of such code everywhere, since there might be ambiguities or the intent might be obscured. Tests usually contain a lot of boilerplate, though, so I'm all for making them more concise. Except for readability, people would be more likely to write a test if it doesn't take much effort to create yet another trivial mock.

Who tests your tests though? You don't need to answer. Still, the page about implementing interfaces in Groovy warns of this trap: "Be careful that you don't accidentally define the map with { }". Type safety could help sometimes when writing tests. Or at least that's a very good excuse to try to emulate this Groovy feature in Scala.

In Scala there are implicit conversions, which help define a controlled way to coerce a value to a different type. We cannot automatically convert all closures to all possible single-method interfaces in scope, but we can choose several we are using especially often (as it happens in tests). For instance, if we want to use a mock of java.lang.Readable often enough, we could define the following implicit conversion:


import java.nio.CharBuffer

implicit def fun2readable(f: Function1[CharBuffer,Int]): Readable =
  new Readable { def read(cb: CharBuffer) = f(cb) }

So now everywhere we need a Readable, we might just drop in the appropriate closure instead (technically, this is not a mock, but a stub):


val readable: Readable = {cb: CharBuffer => cb.put("12 34"); 5}
val s = new java.util.Scanner(readable)
assert (s.nextInt == 12)

Compare this with the Groovy version:


def readable = { it.put("12 34"); 5 } as Readable
def s = new Scanner(readable)
assert s.nextInt() == 12

In the Scala version, we are explicitly defining the type of the variable in order to force the implicit conversion. In methods, where Readable is expected as an argument, explicitly naming the type will not be necessary, whereas in Groovy you always need to coerce using "as Readable":


val s = new java.util.Scanner({cb: CharBuffer => cb.put("12 34"); 5})

So in Scala you're trading some verbosity up front for some conciseness in all of the code where the implicit conversion is visible. Furthermore, the type checker guarantees that only closures with the correct signature are converted to the interface in question (i.e. with an input of CharBuffer and output of Int) and will flag a compile-time error otherwise.

The reader might be tempted to also emulate map coercion, but beware of an unexpected difficulty: it's not easy to define both a closure (with one argument) and map coercion. Since a Map in scala is a function and due to type erasure, the compiler won't know whether to substitute the Map or the Function!

Can you generate these implicits automatically without defining them? Maybe with a Scala compiler plugin, but then you lose some of the transparency of knowing which implicits you've defined.

Scala interpreter in Firefox' location bar

2009-09-26T16:05:00.003+03:00

SimplyScala is awesome for trying out some quick Scala tricks. However, you can go one step further, and execute Scala one-liners in your Firefox location bar!

1. Create a bookmark to URL http://www.simplyscala.com/interp?code=%s
2. Associate a keyword to the bookmark. Mine is "scala", you can try something shorter like "s" or ">".
3. Now when you type "scala <expression>", the expression will be sent to SimlpyScala and evaluated. For instance, "scala List(1,2,3).reverse"

Why does this work? Whenever you have a keyword associated with a bookmark Firefox expands "%s" in the URL to the text following the keyword. Check this Lifehacker article for details.

The nice thing is that your bindings are kept in your session, so you can use variable names defined in previous invocations. There are inconveniences, for instance you cannot easily preview your history and it's hard to review inputs which are more than a couple of lines long.

I also find that in combination with URL-shortening services it's an acceptable alternative to using code snippet sharing sites like gist.github.com or paste.pocoo.org. The code is actually part of the URL, which makes sense for shorter code snippets. You don't have syntax highlighting, but you have the evaluated result readily available.

3 things you didn't know Scala pattern matching can(not) do

2009-09-16T23:27:00.007+03:00

Since I started learning Scala, pattern matching has become a favorite feature. But powerful as it is, pattern matching has some limitations. On the other hand, there are some unexpected ways to use pattern matching.

Fake multiple assignment of tuples

You already knew you can initialize multiple variables using the tuple syntax:

val (a, b) = (1, 2)

I want to do the same with closure parameters, though. For example, to swap the elements of 2-tuples in a list:


List(1 -> "one") map {
  t => (t._2, t._1)
}

Still, using numbered slots for tuples doesn't seem as readable. It would be great if we could easily assign variables with meaningful names to document the purpose of the elements of the tuple. Parameters are already packed in a tuple, so why can't I do this:


List(1 -> "one") map {
  val (num, str) = _; (str, num)
}

Unfortunately this is not legal syntax. Of course, I can explicitly bind the tuple to a temporary variable and then decompose it:


List(1 -> "one") map {
  t => val (num, str) = t; (str, num)
}

But this seems too verbose for Scala. Fortunately, there's a trick which can help. Pattern matches are actually instances of partial functions! The closures used for map, filter, etc. are functions, too. This means we can use a pattern match to bind variables to the closure parameters:


List(1 -> "one") map {
  case (num, str) => (str, num)
}

Match the last element of a list

Decomposing lists in pattern matching is another power feature of Scala. You can match the first element of a list:


List(1, 2, 3) match {
  case List(1, _*) => "yeah!"
}

The underscore-star (_*) symbol serves as a placeholder for the rest of the elements of the list. You can also do this:


List(1, 2, 3) match {
  case 1 :: _ => "hurray!"
}

Which mimicks construction of the list using the cons operator. So what if I want to match on the last element instead of the first?


(1 to 9).toList match {
  case List(_*, 8, 9) => "sure"
}

Hey, this seems like it's working! Not so fast, though:


(1 to 9).toList match {
  case List(_*, 18) => "yeah, right"
}

In fact, in Scala 2.8 this syntax will trigger an error message: "_* may only come last"

How about using the alternative notation?


(1 to 9).toList match {
  case _ :: 9 :: Nil=> "no way"
}

Unfortunately Scala expects the placeholder to be an element, not a list. The triple colon expects a list, will it work?


(1 to 9).toList match {
  case _ ::: 9 :: Nil => "absolutely not"
}

Nope, this isn't even recognized by the compiler. While you're wondering why there is a ::: operator, but it's unknown to Scala in pattern matching, remember that ::, which is used in pattern matching, is actually a class. Furthermore, the syntax "a :: b" is in fact a more readable expression of "::(a, b)", which in turn is a short form of "call unapply on the matching expression and check if it decomposes to a and b".

And this, in short, is how the Scala black magic, called "extractors", works. But can we also use it to match on the last element of a list? Sure! Just define an object (let's call it "::>") and define its unapply method. The method expects a list and must return a tuple of the "init" part of the list and the "last".


object ::> {def unapply[A] (l: List[A]) = Some( (l.init, l.last) )}

List(1, 2, 3) match {
  case _ ::> last => println(last)
}

(1 to 9).toList match {
  case List(1, 2, 3, 4, 5, 6, 7, 8) ::> 9 => "woah!"
}
(1 to 9).toList match {
  case List(1, 2, 3, 4, 5, 6, 7) ::> 8 ::> 9 => "w00t!"
}

Match structural types

Since you've heard of structural types, didn't you wish you could do this to find out whether a class has a certain method?


List(1) match {
  case t: {def length: Int} => "cool!"
}

Unfortunately, here again we are fooled in thinking method checking happens when matching the class. It's not:


List(1) match {
  case t: {def aoeu: Int} => "really?"
}

The reason for this is that the class type is erased to AnyRef. Unfortunately, there's no clean workaround for this- yet. This looks positively ugly:


List(1) match {
  case t: {def aoeu: Int}
          if t.getClass.getMethods.exists{_.getName == "aoeu"} => "not really!"
}

Do not despair, though- there's a chance that this feature is going to land in Scala sooner or later.

The ultimate Twitter client

2009-06-23T16:33:00.002+03:00

...is a Feed reader!

OK, not quite, but after some thinking about the top features the ideal Twitter client should have, I found feed readers have most. For the minor drawback of not being able to post messages, you get:

Marking messages as read

Fixed replies

Favorites of friends

others

friends

recommend

Tracking

No follow/unfollow counting

But some of these "features" require you to import the feed of every single one of your users. Noone in their right mind would do that manually, but there's a way to generate a list of feeds in the form of an OPML file (which many readers can import). First of all, get a list of your friends. Then apply the following XSL stylesheet:


<xsl:stylesheet version="1.0"
 xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

 <xsl:output method="xml" encoding="ISO-8859-1" indent="yes"/>
 <xsl:strip-space elements="*"/>

 <xsl:template match="/">
  <opml>
   <body>
    <outline title="twitter">
    <xsl:for-each select="/ids/id">
     <outline>
      <xsl:attribute name="title"><xsl:value-of select="."/></xsl:attribute>
      <xsl:attribute name="xmlUrl">http://twitter.com/statuses/user_timeline/<xsl:value-of select="."/>.atom</xsl:attribute>
     </outline>
    </xsl:for-each>
    </outline>
   </body>
  </opml>
 </xsl:template>

</xsl:stylesheet>

This contains only the most essential elements of a feed, and the name of each feed is the ID, not the name, but Gooogle Reader imports the feed successfully and some readers can rename the feed from information provided in the RSS/Atom format.

For an OPML of your friends' favorites, replace "statuses/user_timeline" with favorites.

3 Scala gotchas

2009-02-09T18:57:00.008+02:00

It's been over a year since I bought my "Programming In Scala" book by Martin Odersky. Since then, a lot has changed. There are now 3 more Scala books in the works: a "Programming Scala" by Venkat Subramaniam (Pragmatic Bookshelf), "Beginning Scala" by David Pollak (Apress) and another "Programming Scala" by Alex Payne (of Twitter fame) and Dean Wampler (by O'Reilly). Scala is the key actor in a couple of new software projects (pun intended), including one at Apache (ESME, to which yours truly is a committer; ok, I have no shame).

There's no perfect language, though, and you can't say you know a language until you know its warts. So here's my share of traps, which wasted a lot of my time- I certainly hope you discover them before you waste yours. Of course, these don't mean the language is ill-designed; if only minor corner cases like these can be discovered, this would suggest that a lot of thought has been put in the language.

Main method in companion objects

object Test {
  def main(args: Array[String]) {
    println("Running!")
  }
}

You copy & paste a simple example and compile it. Great, everything works and you're happy. So now you read about companion objects and decide to use your main method so you can test the companion class. You start with a dummy and (if you're cautious), try to compile right away:


class Test {
}

Good, it compiles. Running it, though, results in a NoSuchMethodException for main. Huh? Well, there's a main method only in singleton objects, not in companion objects.

Variables and constants in pattern matching


val Extension = "xml"
"json" match {
  case Extension => "Wrong!"
}

Here, you want to match on the contents of a variable. It works as expected: in the above example a MatchError is thrown, as expected. So suddenly you decide that you want to rename the variable...


val fileExtension = "xml"
"json" match {
  case fileExtension => "Wrong!"
}

Boom! Now pattern matching matches things it shouldn't! Actually, it matches everything you throw at it. What has happened? Well, Scala checks the name in order to decide if it's an existing constant (capitalized first letter) or a new variable name it should bind (lower-case first letter). So in the last example a new variable called "fileExtension" was created, which shadows the existing variable name.

Always returning Unit from methods


def test1 {
  println("test1")
}

def test2 = {
  "test"
}

No surprises here- one method prints a String, the other returns one. So then you suddenly decide that you want only one method to this (contrived) and merge these two:


def test {
  println("test1")
  "test2"
}

println(test)

...and the output is not what is expected: only "test1" is printed. What's wrong here? In Scala, if you omit the equals sign after the method declaration and before the method body, the result is always Unit (same as void in Java).

Scala is static- is Scala dynamic?

2009-01-28T10:32:00.005+02:00

Despite being statically typed, Scala is not an inflexible language. Smarter folks than me have commented on extending the behavior of classes dynamically using implicits.

I want to focus on another feature of Scala which allows for dynamic behavior. In concurrent programming with actors, the default handling involves pattern matching on the messages received. Pattern matching and actor concurrency have been borrowed from Erlang- a dynamic language by nature- and have been one of the key differentiating concepts compared to languages I've learned before.

When calling a method on a class, you have to know the method signature at compile-time. The types of the parameters you pass must be defined (even through an implicit type conversion). But when you pass a message to an actor (something of an asynchronous method call), you don't need to care what type of class the actor is or what type of messages it can process- at least at compile time:

import scala.actors._
import scala.actors.Actor._

val a = actor {
  loop {
    react {
      case i: Int => println("Got an integer")
      case o: AnyRef => println(o.getClass)
    }
  }
}

a ! 2
a ! 2.8
a ! "what is this?"

There's an unassuming case denoted by the underscore character (_). This is the so-called catch-all case, the moral equivalent of Ruby's method_missing. As you can see, you can send any object to the actor, and all undefined classes can be handled dynamically by this placeholder case.

Of course, the spirit of Scala doesn't encourage handling patterns using the catch-all method. You can send messages to actors only through a typed scala.actors.Channel:

import scala.actors._
import scala.actors.Actor._

val c = new Channel[Int](actor {
  loop {
    react {
      case _ ! _ => println("test")
    }
  }
})

c ! 2

Once you use this Int-typed Channel, you can only send Int messages to it (unless there are any implicit conversions). For instance this would be caught by the compiler and fail with an error message:


c ! "2"
:10: error: type mismatch;
 found   : java.lang.String("2")
 required: Int
       c ! "2"
           ^

The Channel is constructed using an actor as an argument. This actor can then match by the scala.actors.! class (yes, it's the name of the class) and check which messages have arrived through which channels.

This took awhile to discover, since the best documentation is in the source itself. If there are any other (and better) idioms for using scala.actors.Channel, I'd be happy to learn them!

Apply search to following

2009-01-21T00:14:00.003+02:00

There are still some things that don't make sense in search.twitter.com. For instance, it would be perfectly logical to search in just your followees' timelines. Why, TwitterSpy is already doing it. You could, of course, generate a huge query with "from:" and every possible person you're following, but search.twitter.com restricts the length of the query. Good luck if you have more than 10 friends. Uh, are there people like this after 1 month of using Twitter?

You could also write a console script, except that you don't want to type your password and reauthenticate once per follower. So doing it from the browser might be a better fit for a quick & dirty approach. This way your session would be reused.

Here's an educational bookmarklet. You can paste it in the URL bar of Firefox (didn't bother with other browsers) when you've opened twitter.com (won't work if window has a link open under another domain). And you can bookmark it- you thought it would be called a bookmarklet for nothing?

javascript:query = prompt("Twitter search:");
xmlhttp = new XMLHttpRequest();
xmlhttp.open("GET", "http://twitter.com/statuses/friends.json");
xmlhttp.onload=function (){
        list=eval(xmlhttp.responseText);
        for (i=0; i<list.length; i++)
                window.open("http://search.twitter.com/search?q="
                        + query + "+from:" + list[i].screen_name)
};
xmlhttp.send(null)

This will open one search window per followee- so don't try this if you have loads of'em. You will hit a limit in Firefox' default config, and you can increase it using the property:


browser.tabs.maxOpenBeforeWarn

Still, you will hit a hardware/hardcoded limit if you're not careful with how many people you're following. I told you this only has educational value, right?

SliTaz: smaller than you think, more useful than you think

2009-01-09T23:28:00.005+02:00

I started using SliTaz on a USB stick half an year ago just as a workaround to make my laptop with crashed hard-disk usable and try to rescue some of the disk contents. Eventually I ended up keeping the distro as my regular OS. There's a lot to like about it.

First of all, it's unbelievably small- 25 MB. Yes, in times of Vista, there are graphical OS's and software distros that small. Yes, they cheat, but with style- the OS is uncompressed using lzma, one of the most efficient compression algorithms. And it has an amazing amount of software, too- Openbox provides a slick GUI and even graphical effects like transparency, Geany is a powerful programmer's editor, Firefox is there, a mail client, torrent client, CD/DVD burner, PDF viewer, mplayer, even Sudoku. I like the choice of lightweight, but stylish programs. Finally, there's a system information applet, which gives you just what you need, and a Debian-like package manager- with a GUI.

The lack of an office suite is one of the key decisions in SliTaz. These days it is not uncommon to rely on Google Docs as a no-frills office suite. And getting rid of one of the bulkiest pieces of software has a lot of unexpected consequences. The OS is loaded in RAM, which makes it exceptionally fast and responsive, and eliminates a lot of reading/writing to the USB stick, which prolongs its life.

Why is SliTaz significant? Small is fashionable again due to the resurgent interest in mobile devices and ultra-mini PCs. Especially the Asus EEE PC and similar devices would be a perfect fit.

In this age of mobility it becomes increasingly important to have a stable environment so you can maximize your productivity. It becomes annoying if you have to set up a new PC every couple of weeks or switch between two different environments. By carrying around a USB stick with your OS, you have not only your documents and portable programs with you- the window manager is the same, you can use the familiar key bindings, the shortcuts and menus are the same. It feels like you're on the same machine.

Finally, you can regain this lost feeling that you understand what's going on with your OS. You keep it simple, you have just what you need and use everyday, and when you need a rarely used package, just install it from the repository. It won't be there when you reboot (unless you explicitly write RAM to the USB), and security-wise, I like the idea- it's as if you update your software for security vulnerabilities when you need it. It's also harder for malware to set itself up to start on reboot.

But why did I choose this specific distro? I used DamnSmallLinux for a while, and it was really cool for awhile, but showing its age- most of the software is hopelessly old- kernel 2.4 and... only last year switched to Firefox 2? No, that's not a typo- in times when Firefox 3 is stable, lagging one major version behind seems backwards. If you want to be small, get Firefox 3, use Google Docs, get rid of the office suite.

More importantly, in DSL there was a 50 MB core of applications, which could not be easily modified and customized. That makes it harder to upgrade as well. In SliTaz you can change everything- remove or upgrade to your heart's content. I can remove more programs, making SliTaz even slimmer.

The only nitpick I have is that a lot of the configuration utilities assume you have certain base programs. For instance, I wanted to have mrxvt instead of xterm, and get rid of xterm. Eventually wrote a small wrapper for xterm which invokes mrxvt.

#!/bin/sh
while [ $# -gt 0 ]; do
 if [ $1 == "-fa" ]; then
  ARGS="$ARGS -fn"
 elif [ $1 == "-fs" ]; then
  shift
 elif [ $1 == "-e" ]; then
  ARGS="$ARGS -e sh -c '"
  while [ $# -gt 0 ]; do
   shift
   ARGS="$ARGS $1"
  done
  ARGS="$ARGS'"
  break
 else
  ARGS="$ARGS \"$1\""
 fi
 shift
done
eval exec mrxvt $ARGS

The browser as a platform

2008-09-30T12:07:00.003+03:00

Google certainly know how to draw attention. They've even created a comic for dramatic effect to present their new browser. I remember it took a couple of hours to read it because it was constantly timing out because of the immense buzz-generated load on the servers.

Now after things have calmed down a little, there was a need to have a sober look at Chrome's competitor: Firefox. There was even an article explaining why Mozilla won't change their Gecko rendering engine with Webkit, the engine used in Chrome.

I'm not saying that Firefox is a better browser, but I think ArsTechnica is onto something. In my humble opinion (I hate abbreviations outside of chat), the fact that XUL can make it so much easier to write extensions is a key factor in its success. Even Steve Yegge mentions, half-jokingly, that Firefox is trying to be Emacs, and if it manages to edit files, he would dump Emacs (and if you know Steve Yegge, he lives for Emacs). Extensibility is again a key criteria here.

So now even Google has announced that Crome will support extensions. It will surely need them if it is to compete with Firefox.

I noticed a trend with my personal computing lately. I am using a very thin OS on an USB stick, which is 25-30MB (compressed). A lot of this is due to Firefox, and my externally kept profile is comparable to the OS size- mostly due to addons. Many of these addons are applications in their own right, comfortably using the networking capabilities of Firefox. Here are some of the extensions (some I use, some I don't), which are more like standalone applications than extensions to Firefox' functionality:

SamePlace

instant messenger

Extend Firefox 2

TwitterFox

FireFTP

mozImage

Firefly

I'm not even mentioning diffent mail checkers, bookmark synchronizers, etc. Is there a tendency towards extensions which are more independent from the functionality of the browser?

Eventually, the browser is just a platform, an OS of sorts, another abstraction layer to make networking transparent for your Web-enabled apps.

Ruby for mp3 file organizing

2008-06-27T09:32:00.005+03:00

So there we are, me and my friend Oliver, caught in a business trip. We're already bored of the only decent pub in the village, our families are a long distance away, so what's a developer to do? Code in a programming language he/she's not allowed on the job, of course! Oliver seemed interested in Ruby and I've already done a couple of small scripts with it, so we were curious to see what the fuss is about.

It goes without saying that if you want to learn to program (in a particular language) you should not rely too much on books. The only way is to find a task you want to have automated, and then code it using your language of choice. Surely, you must pick the task (or the language) carefully, since not all languages are suitable for all tasks.

One of the things which Oliver has struggled with was organizing all of his podcasts in his player, sorted neatly by directories of author and title. Having found both a hammer and a nail, we were ready to start pounding.

After a bit of research we found the mp3info and id3tag Ruby libraries. id3tag had different fields for ID3v1 and ID3v2 data and didn't have write support (not that we needed it). mp3info didn't have ID3v2.2 support, but I found an interesting link about ID3 internals- the format of the fields was something that could be useful.

After a while our pair programming session has reached a milestone- our script works. It doesn't seem very modular though, so we spend some time making classes and discussing what is the responsibility of each class. Should there be a manager-class? Or should the objects manage themselves? I go with the second approach, and here's the result:


# Class for handling information of the mp3 file
class Mp3File
  attr_reader :title, :artist, :album

  def initialize filename
    @artist = @album = "unknown"
    @filename = filename
    @title = File.basename(filename, ".mp3")

    read_attributes
  end

  def title
    sanitize(@title)
    if @title == "unknown" then @title = File.basename(@filename, ".mp3") end
    @title
  end

  def read_attributes
    begin
      Mp3Info.open(@filename) do |mp3info|
        (@title, @artist, @album) = %w{title artist album}.collect { |attrib|
          begin
            (result = mp3info.tag.send(attrib)).empty? ? "unknown" : result
          rescue
            "unknown"
          end
        }
      end
    rescue
    end
  end

  def sanitize str
    str.tr_s!("?'","_")
  end

  def transfer(newPath)
    newPath = eval('"' + newPath + '"')
    FileUtils.mkdir_p File.dirname(newPath)
    FileUtils.cp @filename, newPath
  end
end

This is the class which is initialized with the location of the file and then extracts information about the artist, title and track name. The read_attributes method is meant to show off our new knowledge about the dynamic nature of Ruby- we build a list of methods to invoke on the Mp3Info object, and if no meaningful result, return "unknown". Finally, as the class knows about the current location and mp3 meta-info, it has a method for copying the file to a new location. The new path is passed as a template, where the #@artist, #@album, #@title are substituted with the value of these fields.


class Mp3List

  attr_reader :files

  def files
    @files.map {|file| Mp3File.new(file) }
  end

  def initialize(sourcePath, days = 7)
    @sourcePath = sourcePath
    @days = days
    @files = read_new
  end

  def read_new
      Dir["#@sourcePath/**/*.mp3"].find_all do |path|
        test(?M, path) > (Time.now - (@days * 60 * 60 * 24))
      end
  end

  def to_s
    @files.inspect
  end
end

Here comes the class, which represents a list of mp3 files in a certain directory (and subdirectories), which satisfies some criteria- in this case, how long ago the files were created (modified). Could it be made more general? Certainly, but in a 80-line script? Maybe next time.


  list = Mp3List.new("/home/whoami/Music", 730)
  list.files.each do |mp3|
    #~ puts "Processing #{filename}"
    mp3.transfer('/tmp/music/#@artist/#@album/#@title.mp3')
  end

What's left was an example of how to use these classes. Seems good to me- and best of all, it works.

The only thing left was to prepare a patch for the mp3info library for ID3v2.2 support. I actually implemented one (still not incorporated in base), and it also initializes the common fields with either the v2 or v1 data, whatever present (v2 still has precedence, if both are present).

Conclusions from our short session:

Ruby is neat for quick hack jobs

mp3info does not provide an exhaustive ID3 handling support, but is good enough and workable

Pair programming might not be smooth from the start, but you will learn a lot about yourself

Organizing your music can sometimes take longer than total time spent looking for your tracks

You should choose your business trip accomodation place carefully if you can

command-not-found handler not only in Ubuntu

2008-06-11T12:04:00.005+03:00

One of Ubuntu's strengths is that it pays attention to the little things. One of these things, which were important to me lately, was the command_not_found_handler in bash.

In layman's terms, when you open a terminal in Ubuntu and type a command, which is not on your system, it looks in a database and in a helpful way suggests that you can get this command if you install a certain package. I was using apt-file to provide the same functionality for a while, but this is both easier and more informative (apt-file scans for substrings of the whole path, so spits out a lot of false positives, which are not commands).

Now as happy as I am in using Ubuntu on my personal machine, I wanted to have this at work, too. I wanted to make it easy for my colleagues who forget to ssh/sudo before executing a command and prepend an appropriate string if the command doesn't work on this machine. Ubuntu can accomplish this trick thanks to a patch of bash, which adds a hook, called command_not_found_handler, but OpenSUSE/SLES is installed at work and this function is missing.

I had the feeling I can do this in bash only, and then I found this blog entry. Only I didn't quite like how it's implemented, though I liked the idea. Benjamin (the blog author) suggested that the check if the last command was successful is done at each prompt, so it is evaluated even if you press enter on an empty line. Besides, to obtain the last command, the whole history was retrieved only to extract the last entry.

The reason Benjamin's hook was done like that is that history expansion didn't work in noninteractive shells. I also tried to enable it by using set -H, but with no luck.

I thought that using a trap would be better than a prompt command. There was a trap, which would be evaluated only on error, so my first draft was based on the trap ERR hook. Then I would check if the exit status of the last command was 127 ("command not found" for bash) and... use the command. By chance I found out quickly about BASH_COMMAND, which in traps is evaluated to the currently executing command. But after a lot of failed attempts I found out that it only worked with the DEBUG trap.

So I used the DEBUG trap and evaluated the last command- if it could be resolved using type, then it is found. Slower, but more accurate. And it worked!


trap 'if ! type -t $BASH_COMMAND >/dev/null; then echo Do something with $BASH_COMMAND; fi' DEBUG

Now my colleagues are happily executing commands, which are only found on remote servers, oblivious to the fact that this command is transparently executing on a host different than their own.

Java Portable Apps

2008-06-10T14:04:00.004+03:00

portable
–adjective

capable of being transported or conveyed: a portable stage.

easily carried or conveyed by hand: a portable typewriter.

Computers. (of data sets, software, etc.) capable of being used on different computer systems.

...

Dictionary.com

So how many definitions of "portable" do your applications of choice satisfy? Having grown up profesionally in the spirit of "write once, run everywhere", I've always wondered how a "portable" application could run only on Windows, as is the case with most of the PortableApps.com. Not that I have anything against portableapps.com, it's a cool idea and a very pragmatic software suite is offered.

But I happen to work on Linux and my colleagues usually work on Windows, so if I want my applications-on-a-stick to be truly portable, they need to run on any platform. I need portable in the sense of "running on any operating system". Thus an obvious choice is Java- it's usually not the first choice for a desktop application, but here its intended purpose fits the bill nicely. And I don't need all operating systems- honestly, who would lend me their Mac? Come on, it's too personal to give to anyone ;-) This means I can have a JRE for Linux and Windows on my stick (in case it's not installed), and I'm set.

Of course, the task of finding the actual applications is the difficult one. They are not all completely portable in the sense of "not modifying anything on the hard disk", but most of them could be configured to run with settings from the USB. I also needed to run all of these on machines without administrator privileges. Most (with a few exceptions) are open-source:

Organizer: ThinkingRock

Getting Things Done

Con

Editor: jEdit

Mind Mapper: FreeMind

Outliners

JOE

Jreepad

File Manager: muCommander

Disk space manager: JDiskReport

Media organizer: MediaSort

jhead

File Synchronization: JFileSync

Version Control: SmartSVN

and

Port forwarders

JPortForwarder

PlugProxy

jzbd

Port Scanners

JAPS

JMap

post-mortem memory analysis tool

This set of programs proved especially useful when my laptop's hard drive reached the end of its intense life. Using a Damn Small Linux to boot from the USB disk along with these applications really decreased the time I could get productive again- without a hard disk- until I got a new hard-drive. To top it off, I could boot DSL in Windows, emulated by qemu- but DSL can really deserves a blog post on its own.

Update: This blog prompted me to do some more research and I found this great thread on Javalobby about Java Desktop Apps. There's at least one new cool app I am going away with- ekspos image viewer. It even has Picasa integration!

SSH key-based authentication in Java

2008-05-08T14:08:00.004+03:00

Yet again the next project of the week was the intersection of work and play (kinda). There was the upcoming project where I would need to exercise remote control using SSH. And there was the next program I was toying with: a nice lightweight commander clone written in Java. What's the common theme here? Well, muCommander has several virtual filesystems, including one for SFTP (and if you didn't know it before, that's a file transfer protocol for SSH).

But muCommander didn't use public key authentication, and I have disabled password-based authentication on my home SSH server. So there was the opportunity to play around with J2SSH (which muCommander was using) and both implement key-based authentication in my current file manager and explore one of the libraries for my next project. OK, maybe not applicable for my work project, because J2SSH is GPL, but I can get to play, OK?

With license questions out of the way, we can get down to business. And it is not difficult at all, as the j2ssh/examples directory contains a PublicKeyConnect.java. You just need to instantiate a com.sshtools.j2ssh.SshClient, call connect, passing the hostname as a String and authenticate using the method... which was it... ah yes authenticate, passing a com.sshtools.j2ssh.authentication.SshAuthenticationClient.

Now getting this SshAuthenticationClient is the fun part. In our case, we are using the com.sshtools.j2ssh.authentication.PublicKeyAuthenticationClient and we need several more steps to prep it up. After instantiating it, we need to call setUsername with a String (can't tell you what it stands for). Then we need to set the private key itself using the setKey method and we're ~~ready~~.

What? setKey expects a parameter? Well yes it does, and it's another of those j2ssh classes: com.sshtools.j2ssh.transport.publickey.SshPrivateKey. But we seem to be stuck here, as you can't instantiate SshPrivateKey- it's abstract. What does this mean? Argh, never mind, you can't instantiate it. Try it. See? Told ya so.

So then we see there's this similarly named com.sshtools.j2ssh.transport.publickey.SshPrivateKeyFile. Might it have something to do with our SshPrivateKey? Yes, as it turns out, it has a method toPrivateKey (takes a passphrase as a String- you do use passphrases, don't you?), and returns our most wanted SshPrivateKey. What's that you're saying? You can't instantiate SshPrivateKeyFile either? Fear not, because it has a static method parse, which overloaded to accept both a File and a byte array with the actual private key file or data.

If you couldn't follow this convoluted line of thought (I couldn't), here's the code, as taken from the patch of muCommander:


PublicKeyAuthenticationClient pk = new PublicKeyAuthenticationClient();
pk.setUsername(credentials.getLogin());

SshPrivateKey key = null;
// Throw an AuthException if problems with private key file
try {
    SshPrivateKeyFile file = SshPrivateKeyFile.parse(new File(privateKeyPath));
    key = file.toPrivateKey(credentials.getPassword());
} catch (InvalidSshKeyException iske) {
    throw new AuthException(realm, "Invalid private key file or passphrase");  // Todo: localize this entry
} catch (IOException ioe) {
    throw new AuthException(realm, "Error reading private key file");  // Todo: localize this entry
}

pk.setKey(key);

Now if sshClient.authenticate returns AuthenticationProtocolState.COMPLETE, you're ready to go. You can open a shell or an SftpClient. Mission accomplished.

If you had the urge to download the muCommander source code and apply this patch, no need to: it's already implemented in CVS.

Twitter: do you follow me?

2008-04-18T11:48:00.002+03:00

This week's hacking task was to implement a "follow all" function for Twitter.

Even for Twitter users, this needs some explanation: the follow functionality now means "enable notifications". However, the command interface in IM/SMS wasn't changed, so the command name remains "follow". For brevity, I will use the word "follow" instead of "enable notifications".

The reason for having this command is that there used to be a function "follow all" in Twitter. It used to instantly turn on notifications for all your friends (users you're following in new terminology). Now there's a user, called "all" and the function doesn't work (ok, maybe that's not the real reason). This put an end to a very useful feature for users who rely often on the Twitter IM integration.

Having a quick look at the Twitter API it seemed pretty straightforward to fetch all users and enable notifications for all of them one by one. It would be fairly slow, but there was no information in the user list whether notifications are enabled for a user or not. This would have eliminated the need to send requests for users, for whom we already have notifications enabled. Ah well...

The first tool I reach in my toolbox is Ruby. I tried using JSON, but had to give up- I simply couldn't handle Unicode issues:

/usr/lib/ruby/1.8/json.rb:288:in `chr': 1090 out of char range (RangeError)

It turned that it was much smoother with REXML, and it really is a superior library for XML processing (Python's are either easy or full-featured, REXML seems to be both).

I initially took the path of using 'open-uri' for fetching the data over http. After all, it handled even http base authentication and abstracted the nitty-gritty details, and so was easy to use.

But it isn't meant to be used for more fine-grained control, and I soon ran into performance problems, which required special treatment. I found that I quickly exhausted the rate limit of the Twitter API- it's only 70 requests per hour, and with one request per user... you get the picture. The web interface wasn't actually subject to such restrictions, so I wanted to check how it's doing it. A slightly different URL, but worked like a charm, and rate limits seemed to be no problem now!

This time, though, the script ran much longer- 80 seconds compared to about 30 before the change. I analyzed the requests and found out that each received a 302 response, forwarding back to the home page. That meant that open-uri was downloading the whole home page for each user!

At that point open-uri had to go and make way for Net::HTTP. It took more lines to rewrite it, but now I had the choice not to follow redirect responses. I only needed to toggle notifications and didn't care what I got back (as long as it's not an error code). In addition, I could use the same Net::HTTP object, meaning that I use the same HTTP keep-alive connection (not sure if open-uri can do this).

And here's the result- dirty, but still quick. You can configure the action to "follow" or "leave" (to disable all notifications). You need to configure the user and password. Putting the configuration options as command-line arguments is left as an exercise to the reader.

#!/usr/bin/env ruby

require 'uri'
require 'net/http'
require 'rexml/document'
include REXML

user = "lazyuser"
pass = "notmypassword"
action = "follow"
PAGE_USERS = 100

Net::HTTP.start("twitter.com") do |http|
    page = 0
    begin
        page += 1
        req = Net::HTTP::Get.new("/statuses/friends.xml?lite=true&page=#{page}")
        req.basic_auth(user, pass)

        doc = Document.new(http.request(req).body)
        ids = doc.elements.to_a("/users/user/id")
        ids.each do |entry|
            req_follow = Net::HTTP::Get.new("/friends/#{action}/" + entry.text)
            req_follow.basic_auth(user, pass)
            http.request(req_follow)
        end
    end while ids.size == PAGE_USERS
end

XML escape

2008-04-16T14:40:00.003+03:00

Friday afternoon is a prime time for blitz-tasks and a rich opportunity for your hacking one-liner ~~5K1LLz~~ skills.

This Friday the finish-up task came from a colleague, who had to leave in half an hour to catch the plane. There's a big several-megabyte XML file and all characters in it had to be escaped, I presume for preparation to be sent over the wire (read http).

Problem is, a certain Windows editor (does it really qualify that definition?) hangs when opening files bigger than 1234KB, and writing a Java program would take a fairly long time compared to the alternatives. Not to mention that many programmers could write the Java program even less memory-efficiently than the joke of an editor that Notepad is (there, I said it). And Java is not very forgiving on memory problems.

But what are the alternatives? As the Perl manual page says, "The three principal virtues of a programmer are Laziness, Impatience, and Hubris." Being a lazy programmer, I tried to see if somebody else had already written a utility to do this (there was zero chance that there wasn't one), and if it was available. First I found this eclipse plugin. However, pasting megabytes into a text box didn't make me confident that it would work.

There was also the xmlstarlet package, which would have done a wonderful job, had it been installed on the old servers where the file could be easily transferred. But it wasn't, while it would take too long to copy it to my machine and back just to convert the file. It would also be hard to find an appropriate package for that old Linux version. No, that's not for impatient programmers.

The next thought I had was: why spend effort on trying to install a package when with Ruby I could do this in a one-liner. Of course, I have nothing against Python, but if there's one thing nobody would argue is that it doesn't fare well against Ruby when it comes to writing one-liners. Anyway, Ruby wasn't installed there either (note to self: this must be amended).

The clock was ticking. So Python it is, and instead of an obfuscated one-liner I convinced myself to write many short readable lines. I hadn't done serious XML processing in Python for a while, but a google search away the answer came to me. It was really insultingly simple, but given enough hubris one could turn even this meager piece of code into a rambling blog post:


#!/usr/bin/env python

from xml.sax.saxutils import escape
from sys import stdin

for line in stdin:
        print escape(line)

Instant evaluation of Java code in OpenOffice

2008-04-10T18:28:00.007+03:00

At the end of last year I had agreed to update a 4-day Java learning course to reflect the changes in Java 5 and 6. The problems with that were:

The course material was long- over 300 pages already

There was a deadline

My first son was going to be born before the deadline

The document was in Word .doc format

I really considered moving the contents to another format- be it LaTeX, docbook or a lightweight markup language like reStructuredText or asciidoc (a topic for another post). Now being out of time meant that I couldn't.

I should have also updated each and every single example for the new language syntax (where relevant) and test that it works. Now last I counted that was 254 snippets of code. OK, it could happen that one example was split across several snippets, but that's nonetheless tons of work- copy the code, paste it in a text editor, save it, compile it, run it, see if the results are the ones expected. If any step fails, rinse and repeat. Dirty work.

The reason I miss simple text formats is that it's so much easier to automate things. For instance, in the document you could include all your source files, which are located separately and tested automatically in one go.

But I didn't have this option. I needed to find some way to automate this. And the answer came from the very material I was going to present- Java Compiler API.

A frequently neglected feature in Java 6, Compiler API provided language libraries to control the process of compiling right in your Java code. So far you needed to save into a file and invoke a separate process to do that- really a workaround. If you've ever used eval in scripting language, you're going to miss instant evaluation sorely. With the new compiler API, well, you're still gonna miss it, but at least compiling is no longer a hack. Besides, control of the compiler now meant that your performance is going to be much better as you can even compile from a string in memory.

The idea began to form- I could define a BeanShell macro in OpenOffice. If we're using OpenOffice integrated with Java 6, then BeanShell will have access to the compiler APIs as well. The macro would compile a class from the selected text in the document (in memory) and load it (still in memory), then run it and display the results. This would certainly make testing the examples faster.

I'm usually lazy enough to first search for a similar solution (even if finding that solution takes more effort than writing it). The first source I came across about in-memory compilation was in the API documentation of JavaCompiler. It was a good start, it used SimpleJavaFileObject to read source from a String, but the class file was still compiled and saved to disk.

Along the same lines was the detailed article in JavaBeat- it showed how to compile from String using a SimpleJavaFileObject.

I knew there had to be more you need to do. The class file always appeared on disk. I was not very keen on the idea of writing a file to disk (implicitly or explicitly). It's slower, it's less secure and often more effort. I found what I was looking for in the velocityreviews forums. Bot I really struck gold with this really detailed document, which described almost exactly what I wanted to do. It's about visualizing the Java bytecode by the way.

There was one more point. I was wondering which graphical widget to use, and it turns out I could have a popup box using both the OpenOffice APIs or normal Java AWT/Swing. I chose OpenOffice, because the look and feel was better integrated- and because it was different than the Java libraries, which I already knew. I had to read a bit in the OpenOffice developer's guide, but it finally worked out.

So my final macro began to take shape. I first had to create the familiar SimpleJavaFileObject. I had to construct it with a String as an argument. The crucial point was to override the getCharContent method so it returns the class field with the String.


class JavaObjectFromString extends SimpleJavaFileObject{
    private String contents = null;
    public JavaObjectFromString(String className, String contents) throws Exception{
        super(new URI(className + Kind.SOURCE.extension), Kind.SOURCE);
        this.contents = contents;
    }
    public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException {
        return contents;
    }
}

What I discovered in the last two sources was that I would need to implement a file manager, preferrably by extending ForwardingJavaFileManager. However, it needs another reimplementation of SimpleJavaFileObject, but this time for the class data itself, not the source. The important thing here is to override openInputStream and openOutputStream to correct the notion of the class about where its data is located:


static class RAMJavaFileObject extends SimpleJavaFileObject {

    RAMJavaFileObject(String name, Kind kind) {
        super(URI.create("string:///" + name.replace('.','/') + Kind.SOURCE.extension), kind);
    }

    ByteArrayOutputStream baos;

    public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException, IllegalStateException, UnsupportedOperationException {
        throw new UnsupportedOperationException();
    }

    public InputStream openInputStream() throws IOException, IllegalStateException, UnsupportedOperationException {
        return new ByteArrayInputStream(baos.toByteArray());
    }

    public OutputStream openOutputStream() throws IOException, IllegalStateException, UnsupportedOperationException {
        return baos = new ByteArrayOutputStream();
    }

}

Now we have everything necessary to extend on our ForwardinJavaFileManager so that it uses our implementation of in-memory class data. Note that we create a HashMap to be used later:


output = new HashMap();

class RAMFileManager extends ForwardingJavaFileManager{
    RAMFileManager (JavaCompiler compiler){
        super(compiler.getStandardFileManager(null,null,null));
    }
    public JavaFileObject getJavaFileForOutput(Location location, String name, Kind kind, FileObject sibling) throws java.io.IOException {
        JavaFileObject jfo = new RAMJavaFileObject(name, kind);
        output.put(name, jfo);
        return jfo;
    }
}

The last thing I define is a small utility method which just returns the String of the class. As you can see, I've made it convenient to select code fragments and the macro will create the necessary framework around them- a container class and a main method, if need be:


SimpleJavaFileObject getJavaFileContentsAsString(String outside, String inClass, String inMethod){
    StringBuilder javaFileContents = new StringBuilder(outside +
            "\npublic class TestClass{\n" +
            inClass +
            "\n    public void testMethod() throws Throwable {\n" +
            inMethod +
            "try{this.getClass().getMethod(\"main\", String[].class).invoke(null, new Object[] {new String[]{}});} catch (NoSuchMethodException nsme) {}" +
            "\n}\n" +
            "}");
    JavaObjectFromString javaFileObject = null;
    try{
        javaFileObject = new JavaObjectFromString("TestClass", javaFileContents.toString());
    }catch(Exception exception){
        exception.printStackTrace();
    }
    return javaFileObject;
}

Now that we have the main infrastructure in place it is time for the action to unfold. This means creating the compiler object, a new task and invoking the task. I do not really want to parse the java fragments in the document to find where they belong, so I use a trick. I assume the fragment belongs either outside of the class I'm generating (import statements, other classes), in the class definition (fields, methods) or in the main method (instructions). I try to put each selection (you haven't forgotten that you can have multiple selections in OpenOffice, right?) in one of these three areas and invoke the task to see if it compiles without error. If it does, I go ahead with the next one:


JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
JavaFileManager jfm = new RAMFileManager (compiler);

outside = inClass = inMethod = "";
for(i=0;i<count;i++) {
    xTextRange = (XTextRange) UnoRuntime.queryInterface(XTextRange.class, xIndexAccess.getByIndex(i));
    newText = xTextRange.getString();

    JavaFileObject javaObjectFromString = getJavaFileContentsAsString(outside + newText, inClass, inMethod);
    Iterable fileObjects = Arrays.asList(new Object[] {javaObjectFromString});
    out = new StringWriter();
    CompilationTask task = compiler.getTask(out, jfm, null, null, null, fileObjects);
    Boolean result = task.call();

    if(result){
        outside += newText;
    } else {
        javaObjectFromString = getJavaFileContentsAsString(outside, inClass + newText, inMethod);
        fileObjects = Arrays.asList(new Object[] {javaObjectFromString});
        task = compiler.getTask(out, jfm, null, null, null, fileObjects);
        result = task.call();
        if (result){
            inClass += newText;
        } else {
            javaObjectFromString = getJavaFileContentsAsString(outside, inClass, inMethod + newText);
            fileObjects = Arrays.asList(new Object[] {javaObjectFromString});
            task = compiler.getTask(out, jfm, null, null, null, fileObjects);
            result = task.call();
            if (result){
                inMethod += newText;
            } else {
                message = "Compilation fails:\n" + out.toString();
                msgtype = "errorbox";
                title = "Compilation error";
                showMessage();
                return 0;
            }
        }
    }
}

At this stage, we already have compiled the class, but how to execute it? We need to define a class loader which knows specifically where to find and how to define the class (note the HashMap we used to store the classes):


ClassLoader cl = new ClassLoader() {
    protected Class findClass(String name) throws ClassNotFoundException {
        JavaFileObject jfo = output.get(name);
        if (jfo != null) {
            byte[] bytes = ((RAMJavaFileObject) jfo).baos.toByteArray();
            return defineClass(name, bytes, 0, bytes.length);
        }
        return super.findClass(name);
    }

};

Now that we have the class, we can get the method and run it. Here I'm also doing some hocus-pocus in order to kidnap the standard output and show it in the message box- I am really interested in what actually is printed and if there are any exceptions:


Class c = Class.forName("TestClass", true, cl);
constructor = c.getConstructor(new Class[]{});
object = constructor.newInstance(new Object[]{});
method = c.getMethod("testMethod", new Class[]{});

sysOut = System.out;
sysErr = System.err;
// System.setOut(out);
PipedOutputStream pout = new PipedOutputStream();
BufferedReader brout = new BufferedReader(new InputStreamReader(new PipedInputStream(pout)));
System.setOut(new PrintStream(pout));

PipedOutputStream perr = new PipedOutputStream();
BufferedReader brerr = new BufferedReader(new InputStreamReader(new PipedInputStream(perr)));
System.setErr(new PrintStream(perr));

message = "Compiled and ran successfully:\n";
msgtype = "infobox";
title = "Success";

try {
    method.invoke(object, new Object[]{});
} catch (Throwable t) {
    message = "Runtime error:\n" + t;
    msgtype = "errorbox";
    title = "Runtime error";
}
sb = new StringBuilder();
while (brout.ready()) {
    sb.append((char) brout.read());
}
message += (sb.length() == 0 ? "" : "\nOutput:\n") + sb.toString();

sb = new StringBuilder();
while (brerr.ready()) {
    sb.append((char) brerr.read());
}
message += (sb.length() == 0 ? "" : "\nError:\n") + sb.toString();

System.setOut(sysOut);
System.setErr(sysErr);

showMessage();

And that's it! What's that you ask? We don't know anything about the showMessage method? You're really curious, aren't you? OK, here it is, mostly taken from the OpenOffice UNO interface:


showMessage() {
    import com.sun.star.awt.XToolkit;
    import com.sun.star.awt.XMessageBoxFactory;
    import com.sun.star.awt.XMessageBox;
    import com.sun.star.awt.XWindowPeer;

    m_xContext = XSCRIPTCONTEXT.getComponentContext();
    m_xMCF = m_xContext.getServiceManager();

    Object oToolkit = m_xMCF.createInstanceWithContext("com.sun.star.awt.Toolkit", m_xContext);
    XToolkit xToolkit = (XToolkit) UnoRuntime.queryInterface(XToolkit.class, oToolkit);

    windowPeer = xTextDoc.currentController.frame.containerWindow;
    XWindowPeer xWindowPeer = (XWindowPeer) UnoRuntime.queryInterface(XWindowPeer.class, windowPeer);

    com.sun.star.awt.Rectangle aRectangle = new com.sun.star.awt.Rectangle();

    XMessageBoxFactory xMessageBoxFactory = (XMessageBoxFactory) UnoRuntime.queryInterface(XMessageBoxFactory.class, oToolkit);
    XMessageBox xMessageBox = xMessageBoxFactory.createMessageBox(xWindowPeer, aRectangle, msgtype, com.sun.star.awt.MessageBoxButtons.BUTTONS_OK, title, message);
    if (xMessageBox != null){
        short nResult = xMessageBox.execute();
    }
}

What's missing is a couple of import statements and standard initializing lines found in every OpenOffice macro template, but this blog post is already too long. And it's better that you ask if you need it rather than get bored and not even get to the end.

Playing with Javascript or what binds Greasemonkey, Twitter and Ambient Avatars together

2008-04-02T12:53:00.020+03:00

It's been a while since I tried JavaScript hacking (almost 2 years). This time I had the haunting idea to create a Greasemonkey mashup so I can see my twitter page with the avatar next to each tweet exactly as it looked at the time the tweet was posted.

To do this the avatar history must be stored somewhere. That's where chinposin.com comes in. Initially originated as a refreshing avatar on Friday, it evolved into the Ambient Avatar Platform (TM) (credit goes to @monkchips and @yellowpark- you're great). In simple words- you follow @chinposin on twitter, and when you change your avatar, the old one is saved. So you have a gallery of all of your previous avatars for your previewing pleasure and along with the dates they were changed.

For those of you wondering what's twitter, that's a topic for an entire new blog post... or a whole blog, so start at wikipedia, so we can continue with the interesting stuff, shall we?

So there we are- we want to include info from one site into another- a task where Greasemonkey excels (normally JavaScript cannot just fetch info from any other site at whim).

I've obviously lost some of my JavaScript knowledge since it took me an obscene amount of time to get this tiny piece of code working. To start off, I had forgotten that Greasemonkey had also some restrictions, not only enhancements. For security purposes, a lot of objects were wrapped in XPCNativeWrapper and I had to use loads of wrappedJSObject as a workaround. Yes, I know it's not secure, and you should know this too.

Another issue I had a problem with was passing an argument to a closure. I eventually remembered that the closure is an object and you can just assign any field to an object, because each object is also an associative array. Accessing the function object from itself also took some googling- arguments.callee did the trick.

So is there anything that can be improved in this shoddy script? You bet. For starters, it loads the chinposin site a lot, sending 20 simultaneous requests right off the bat, even for duplicate user pages. I could cache the avatar history, but that would require that I synchronize the requests. This script could be modified into a Firefox extension, which has less restrictions than Greasemonkey. And I really should use a prototype for those twenty closures I create, but I gotta have something to do for next time, right?

Without further ado, here's the script. Copy it and paste it into twitteravatarhistory.user.js (OK, you can come up with a longer name if you're so inclined). Then open it with Firefox and if Greasemonkey is installed you will be presented with a dialog prompting you to install it. It's tested with Firefox 2.0.0.13, 3a9, 3b4 and Greasemonkey 0.6.6.20061017 and 0.7.20080121.0. Considering the rate of change, I would be surprised it works in 1 year.

// ==UserScript==
// @name          TwitterAvatarHistory
// @description   Shows tweets with the avatar at time of posting
// @include       http://twitter.com/*
// ==/UserScript==

// Assumptions:
// -chinposin.com has a special date string under the pic
// -avatars are listed chronologically
// -many others regarding DOM position

const avatar_home = "http://www.chinposin.com/home/";
var twitter_images = document.evaluate('//.[contains(@class, "hentry")]', document, null, XPathResult.UNORDERED_NODE_ITERATOR_TYPE, null);
while (message = twitter_images.iterateNext()) {
 message = message.wrappedJSObject;

 // Read user name
 var url = message.getElementsByClassName("url")[0];
 if (!url) continue;
 var username = url.getAttribute("href").match("[^/]*$");

 // Read date of message and extract fields with a regexp
 var date_string = message.getElementsByClassName("published")[0].getAttribute("title");
 var match = date_string.match(/(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})\+(\d{2}):(\d{2})/);
 var date = new Date(match[1], match[2], match[3], match[4], match[5], match[6]);

 var http = function(responseDetails) {
  // add dummy element so we can operate on its DOM
  var elem = document.createElement("html");
  document.body.appendChild(elem);
  elem.innerHTML = responseDetails.responseText;

  // getElementById is only found in document object, will use XPath
  var gallery = document.evaluate('//.[@id="gallery"]', elem.wrappedJSObject, null,
                       XPathResult.ANY_UNORDERED_NODE_TYPE, null).singleNodeValue.wrappedJSObject;

  // Might be better to couple these more tightly than creating two separate arrays
  var images = gallery.getElementsByTagName("img");
  var dates = gallery.getElementsByClassName("mainText");

  // Find avatar date not more recent than message date
  for (i = 0; i < dates.length; i++) {
   var match = dates[i].textContent.match(/(\d{4})-(\d{2})-(\d{2}) +(\d{2}):(\d{2}):(\d{2})/);
   var avatar_date = new Date(match[1], match[2], match[3], match[4], match[5], match[6]);

   if (avatar_date < arguments.callee.date) {
    // Replace message pic with avatar corresponding to date
    arguments.callee.img.firstChild.setAttribute("src", images[i].getAttribute("src"))
    // TMTOWTDI:
    //~ arguments.callee.img.replaceChild(images[i].cloneNode(false), arguments.callee.img.firstChild);
    break;
   }
  }

  // clean up temp structure
  document.body.removeChild(elem);

 }

 // Trick to pass data to the closure
 http.date = date;
 http.img = message.getElementsByClassName("url")[0];

 // Reach list of pix from user page
 GM_xmlhttpRequest({method : "GET", url : avatar_home + username, onload : http});

}

Update: code formatting had munched some of the Greasemonkey header, that should be fixed now.

Update 10 April 2008: New code's on Greasemonkey repository since last week, today a fix was issued that adapts to twitter interface changes.

A New Year's resolution

2008-03-20T18:53:00.006+02:00

The start of each year invariably makes us look back and wish to bring more purpose to the next year. So did I have an improved New Year's Resolution? Of course- 1280x1024. A bit more significant, though, was to find another programming language to learn this year (following the sound advice from "Pragmatic" Dave Thomas).

Since we geeks can sometimes be programming language junkies, this was harder for me to do than previous years. I've previously improved my knowledge about Perl(2004), Python (2005), Tcl (2006) and Ruby (2007). They served their purpose but this year I felt it's about time to depart a bit from my comfort zone and try a new concept. As someone with much more experience noted, "A language that doesn't affect the way you think about programming, is not worth knowing".

So I did my homework and did a bit of research and nominated the following languages:

Scheme: yes I've done some functional programming in the university and I've edited an Emacs Lisp macro or two, but if you're not using it, you're losing it. Besides, I've always wanted (and postponed forever) reading "The Little Schemer", which has an almost cult status.
Logo: This elegant language is sometimes referred to as "LISP without the parentheses". I did some research about which programming language I should start teaching my son first (yeah I know it's ridiculously early to do that before your son is even born!) and decided that LOGO is the all-time favorite kids' programming language. Of course, being famous for its graph-drawing turtle does not mean it can't be used as a serious language. Anyway, it would be a pity if I don't learn it some day, given my relatives with some years of research on Logo.
Haskell: One of the most exotic in this mix, Haskell's lazy evaluation and monads sound novel enough to be fun to learn, besides it's getting some moderate popularity and there is even real-world software written in Haskell, like the Perl 6 compiler/interpreter and revision-control system Darcs (OK, Perl 6 is not a real-world software yet).
Erlang: One of the promises of Erlang is that it can deliver one of the easiest concurrency programming models with message passing between lightweight processes. Besides, it seems to be a fairly simple (compared to Haskell at least) functional language, while still preserving power features like pattern matching. Last, but not least, I really like hot class (module?) reloading and compared to the hot swap feature in the JVM it is really much more pervasive, so much easier for debugging, monitoring, quick fixes, etc.
Scala: It's not just object-oriented and it's not just functional, because it's a language with many influences- it even has pattern matching and Erlang's message passing model for concurrency. One of the best advantages is that it works on the JVM and integrates seamlessly with existing Java libraries.
Groovy: This is another promising dynamic language on the JVM, which seems to draw a lot of its influence from Ruby, including a Rails-like webapp framework, called Grails (surprising choice of name, huh?). Since my professional work is related to Java, I might learn this sooner or later because I think we would be seeing more from this (we already are). Especially for working with XML builders seem to be making things much smoother.

Interestingly, along with my new big-resolution display came a dual-core processor. Everyone knows that multiple cores will be all over the place soon. So my curiosity about how you can make these multiple processors scale based on the programming language model itself, took over. That leaves Erlang and Scala.

Of course that's not all of it. Logo's time just hadn't come yet, as my son is too young for me to start seriously teaching him (hell, he doesn't even know a human language to start with). Groovy could wait as well and it shouldn't be hard at all to pick up with Python, Ruby and Java behind my back. Erlang might be an easier first step into serious functional programming than Haskell and Scheme... well I guess I can postpone it for one more year and make do with Emacs Lisp.

But which one should it be- Erlang or Scala? Erlang is recommended by the same Pragmatic Dave who is responsible for me learning Ruby. And Scala's creator, Martin Odersky, is the man behind Java Generics and the current Java compiler.

But you know what? I didn't have to decide (or I might have learned none of these). My brother has the uncanny ability to guess what I want for a present, so he got me Programming Erlang from the Pragmatic bookshelf. And my father has the great ability to ask me what I want for a present, so he got me the Programming in Scala from Artima.

Eventually, I must have done a fairly good job on my language research because my list was not very different from the one in an article about new languages on Dr.Dobb's Journal.

On a side note- ordering PDF books is great- you can search easily, you don't destroy trees and you get the updates of new revisions easily- go for it!

So there I am, with two languages instead of one for this year, although Erlang and Scala have a lot in common. With the surge of interest in new programming languages one language per year only might take me forever to cover all I wanted to learn.