Wednesday, January 28, 2009

Scala is static- is Scala dynamic?

Despite being statically typed, Scala is not an inflexible language. Smarter folks than me have commented on extending the behavior of classes dynamically using implicits.

I want to focus on another feature of Scala which allows for dynamic behavior. In concurrent programming with actors, the default handling involves pattern matching on the messages received. Pattern matching and actor concurrency have been borrowed from Erlang- a dynamic language by nature- and have been one of the key differentiating concepts compared to languages I've learned before.

When calling a method on a class, you have to know the method signature at compile-time. The types of the parameters you pass must be defined (even through an implicit type conversion). But when you pass a message to an actor (something of an asynchronous method call), you don't need to care what type of class the actor is or what type of messages it can process- at least at compile time:

import scala.actors._
import scala.actors.Actor._

val a = actor {
loop {
react {
case i: Int => println("Got an integer")
case o: AnyRef => println(o.getClass)
}
}
}

a ! 2
a ! 2.8
a ! "what is this?"


There's an unassuming case denoted by the underscore character (_). This is the so-called catch-all case, the moral equivalent of Ruby's method_missing. As you can see, you can send any object to the actor, and all undefined classes can be handled dynamically by this placeholder case.

Of course, the spirit of Scala doesn't encourage handling patterns using the catch-all method. You can send messages to actors only through a typed scala.actors.Channel:

import scala.actors._
import scala.actors.Actor._

val c = new Channel[Int](actor {
loop {
react {
case _ ! _ => println("test")
}
}
})

c ! 2



Once you use this Int-typed Channel, you can only send Int messages to it (unless there are any implicit conversions). For instance this would be caught by the compiler and fail with an error message:


c ! "2"
:10: error: type mismatch;
found : java.lang.String("2")
required: Int
c ! "2"
^


The Channel is constructed using an actor as an argument. This actor can then match by the scala.actors.! class (yes, it's the name of the class) and check which messages have arrived through which channels.

This took awhile to discover, since the best documentation is in the source itself. If there are any other (and better) idioms for using scala.actors.Channel, I'd be happy to learn them!

Wednesday, January 21, 2009

Apply search to following

There are still some things that don't make sense in search.twitter.com. For instance, it would be perfectly logical to search in just your followees' timelines. Why, TwitterSpy is already doing it. You could, of course, generate a huge query with "from:" and every possible person you're following, but search.twitter.com restricts the length of the query. Good luck if you have more than 10 friends. Uh, are there people like this after 1 month of using Twitter?

You could also write a console script, except that you don't want to type your password and reauthenticate once per follower. So doing it from the browser might be a better fit for a quick & dirty approach. This way your session would be reused.

Here's an educational bookmarklet. You can paste it in the URL bar of Firefox (didn't bother with other browsers) when you've opened twitter.com (won't work if window has a link open under another domain). And you can bookmark it- you thought it would be called a bookmarklet for nothing?

javascript:query = prompt("Twitter search:");
xmlhttp = new XMLHttpRequest();
xmlhttp.open("GET", "http://twitter.com/statuses/friends.json");
xmlhttp.onload=function (){
list=eval(xmlhttp.responseText);
for (i=0; i<list.length; i++)
window.open("http://search.twitter.com/search?q="
+ query + "+from:" + list[i].screen_name)
};
xmlhttp.send(null)


This will open one search window per followee- so don't try this if you have loads of'em. You will hit a limit in Firefox' default config, and you can increase it using the property:


browser.tabs.maxOpenBeforeWarn


Still, you will hit a hardware/hardcoded limit if you're not careful with how many people you're following. I told you this only has educational value, right?

Friday, January 9, 2009

SliTaz: smaller than you think, more useful than you think

I started using SliTaz on a USB stick half an year ago just as a workaround to make my laptop with crashed hard-disk usable and try to rescue some of the disk contents. Eventually I ended up keeping the distro as my regular OS. There's a lot to like about it.

First of all, it's unbelievably small- 25 MB. Yes, in times of Vista, there are graphical OS's and software distros that small. Yes, they cheat, but with style- the OS is uncompressed using lzma, one of the most efficient compression algorithms. And it has an amazing amount of software, too- Openbox provides a slick GUI and even graphical effects like transparency, Geany is a powerful programmer's editor, Firefox is there, a mail client, torrent client, CD/DVD burner, PDF viewer, mplayer, even Sudoku. I like the choice of lightweight, but stylish programs. Finally, there's a system information applet, which gives you just what you need, and a Debian-like package manager- with a GUI.

The lack of an office suite is one of the key decisions in SliTaz. These days it is not uncommon to rely on Google Docs as a no-frills office suite. And getting rid of one of the bulkiest pieces of software has a lot of unexpected consequences. The OS is loaded in RAM, which makes it exceptionally fast and responsive, and eliminates a lot of reading/writing to the USB stick, which prolongs its life.

Why is SliTaz significant? Small is fashionable again due to the resurgent interest in mobile devices and ultra-mini PCs. Especially the Asus EEE PC and similar devices would be a perfect fit.

In this age of mobility it becomes increasingly important to have a stable environment so you can maximize your productivity. It becomes annoying if you have to set up a new PC every couple of weeks or switch between two different environments. By carrying around a USB stick with your OS, you have not only your documents and portable programs with you- the window manager is the same, you can use the familiar key bindings, the shortcuts and menus are the same. It feels like you're on the same machine.

Finally, you can regain this lost feeling that you understand what's going on with your OS. You keep it simple, you have just what you need and use everyday, and when you need a rarely used package, just install it from the repository. It won't be there when you reboot (unless you explicitly write RAM to the USB), and security-wise, I like the idea- it's as if you update your software for security vulnerabilities when you need it. It's also harder for malware to set itself up to start on reboot.

But why did I choose this specific distro? I used DamnSmallLinux for a while, and it was really cool for awhile, but showing its age- most of the software is hopelessly old- kernel 2.4 and... only last year switched to Firefox 2? No, that's not a typo- in times when Firefox 3 is stable, lagging one major version behind seems backwards. If you want to be small, get Firefox 3, use Google Docs, get rid of the office suite.

More importantly, in DSL there was a 50 MB core of applications, which could not be easily modified and customized. That makes it harder to upgrade as well. In SliTaz you can change everything- remove or upgrade to your heart's content. I can remove more programs, making SliTaz even slimmer.

The only nitpick I have is that a lot of the configuration utilities assume you have certain base programs. For instance, I wanted to have mrxvt instead of xterm, and get rid of xterm. Eventually wrote a small wrapper for xterm which invokes mrxvt.

#!/bin/sh
while [ $# -gt 0 ]; do
if [ $1 == "-fa" ]; then
ARGS="$ARGS -fn"
elif [ $1 == "-fs" ]; then
shift
elif [ $1 == "-e" ]; then
ARGS="$ARGS -e sh -c '"
while [ $# -gt 0 ]; do
shift
ARGS="$ARGS $1"
done
ARGS="$ARGS'"
break
else
ARGS="$ARGS \"$1\""
fi
shift
done
eval exec mrxvt $ARGS

Tuesday, September 30, 2008

The browser as a platform

Google certainly know how to draw attention. They've even created a comic for dramatic effect to present their new browser. I remember it took a couple of hours to read it because it was constantly timing out because of the immense buzz-generated load on the servers.

Now after things have calmed down a little, there was a need to have a sober look at Chrome's competitor: Firefox. There was even an article explaining why Mozilla won't change their Gecko rendering engine with Webkit, the engine used in Chrome.

I'm not saying that Firefox is a better browser, but I think ArsTechnica is onto something. In my humble opinion (I hate abbreviations outside of chat), the fact that XUL can make it so much easier to write extensions is a key factor in its success. Even Steve Yegge mentions, half-jokingly, that Firefox is trying to be Emacs, and if it manages to edit files, he would dump Emacs (and if you know Steve Yegge, he lives for Emacs). Extensibility is again a key criteria here.

So now even Google has announced that Crome will support extensions. It will surely need them if it is to compete with Firefox.

I noticed a trend with my personal computing lately. I am using a very thin OS on an USB stick, which is 25-30MB (compressed). A lot of this is due to Firefox, and my externally kept profile is comparable to the OS size- mostly due to addons. Many of these addons are applications in their own right, comfortably using the networking capabilities of Firefox. Here are some of the extensions (some I use, some I don't), which are more like standalone applications than extensions to Firefox' functionality:


  • SamePlace
  • an XMPP instant messenger, which won the Extend Firefox 2 contest
  • TwitterFox
  • a Twitter client
  • FireFTP
  • an FTP client
  • mozImage
  • an image viewer
  • Firefly
  • a file manager


I'm not even mentioning diffent mail checkers, bookmark synchronizers, etc. Is there a tendency towards extensions which are more independent from the functionality of the browser?

Eventually, the browser is just a platform, an OS of sorts, another abstraction layer to make networking transparent for your Web-enabled apps.

Friday, June 27, 2008

Ruby for mp3 file organizing

So there we are, me and my friend Oliver, caught in a business trip. We're already bored of the only decent pub in the village, our families are a long distance away, so what's a developer to do? Code in a programming language he/she's not allowed on the job, of course! Oliver seemed interested in Ruby and I've already done a couple of small scripts with it, so we were curious to see what the fuss is about.

It goes without saying that if you want to learn to program (in a particular language) you should not rely too much on books. The only way is to find a task you want to have automated, and then code it using your language of choice. Surely, you must pick the task (or the language) carefully, since not all languages are suitable for all tasks.

One of the things which Oliver has struggled with was organizing all of his podcasts in his player, sorted neatly by directories of author and title. Having found both a hammer and a nail, we were ready to start pounding.

After a bit of research we found the mp3info and id3tag Ruby libraries. id3tag had different fields for ID3v1 and ID3v2 data and didn't have write support (not that we needed it). mp3info didn't have ID3v2.2 support, but I found an interesting link about ID3 internals- the format of the fields was something that could be useful.

After a while our pair programming session has reached a milestone- our script works. It doesn't seem very modular though, so we spend some time making classes and discussing what is the responsibility of each class. Should there be a manager-class? Or should the objects manage themselves? I go with the second approach, and here's the result:


# Class for handling information of the mp3 file
class Mp3File
attr_reader :title, :artist, :album

def initialize filename
@artist = @album = "unknown"
@filename = filename
@title = File.basename(filename, ".mp3")

read_attributes
end

def title
sanitize(@title)
if @title == "unknown" then @title = File.basename(@filename, ".mp3") end
@title
end

def read_attributes
begin
Mp3Info.open(@filename) do |mp3info|
(@title, @artist, @album) = %w{title artist album}.collect { |attrib|
begin
(result = mp3info.tag.send(attrib)).empty? ? "unknown" : result
rescue
"unknown"
end
}
end
rescue
end
end

def sanitize str
str.tr_s!("?'","_")
end

def transfer(newPath)
newPath = eval('"' + newPath + '"')
FileUtils.mkdir_p File.dirname(newPath)
FileUtils.cp @filename, newPath
end
end


This is the class which is initialized with the location of the file and then extracts information about the artist, title and track name. The read_attributes method is meant to show off our new knowledge about the dynamic nature of Ruby- we build a list of methods to invoke on the Mp3Info object, and if no meaningful result, return "unknown". Finally, as the class knows about the current location and mp3 meta-info, it has a method for copying the file to a new location. The new path is passed as a template, where the #@artist, #@album, #@title are substituted with the value of these fields.


class Mp3List

attr_reader :files

def files
@files.map {|file| Mp3File.new(file) }
end

def initialize(sourcePath, days = 7)
@sourcePath = sourcePath
@days = days
@files = read_new
end

def read_new
Dir["#@sourcePath/**/*.mp3"].find_all do |path|
test(?M, path) > (Time.now - (@days * 60 * 60 * 24))
end
end

def to_s
@files.inspect
end
end


Here comes the class, which represents a list of mp3 files in a certain directory (and subdirectories), which satisfies some criteria- in this case, how long ago the files were created (modified). Could it be made more general? Certainly, but in a 80-line script? Maybe next time.


list = Mp3List.new("/home/whoami/Music", 730)
list.files.each do |mp3|
#~ puts "Processing #{filename}"
mp3.transfer('/tmp/music/#@artist/#@album/#@title.mp3')
end


What's left was an example of how to use these classes. Seems good to me- and best of all, it works.

The only thing left was to prepare a patch for the mp3info library for ID3v2.2 support. I actually implemented one (still not incorporated in base), and it also initializes the common fields with either the v2 or v1 data, whatever present (v2 still has precedence, if both are present).

Conclusions from our short session:

  • Ruby is neat for quick hack jobs

  • mp3info does not provide an exhaustive ID3 handling support, but is good enough and workable

  • Pair programming might not be smooth from the start, but you will learn a lot about yourself

  • Organizing your music can sometimes take longer than total time spent looking for your tracks

  • You should choose your business trip accomodation place carefully if you can

Wednesday, June 11, 2008

command-not-found handler not only in Ubuntu

One of Ubuntu's strengths is that it pays attention to the little things. One of these things, which were important to me lately, was the command_not_found_handler in bash.

In layman's terms, when you open a terminal in Ubuntu and type a command, which is not on your system, it looks in a database and in a helpful way suggests that you can get this command if you install a certain package. I was using apt-file to provide the same functionality for a while, but this is both easier and more informative (apt-file scans for substrings of the whole path, so spits out a lot of false positives, which are not commands).

Now as happy as I am in using Ubuntu on my personal machine, I wanted to have this at work, too. I wanted to make it easy for my colleagues who forget to ssh/sudo before executing a command and prepend an appropriate string if the command doesn't work on this machine. Ubuntu can accomplish this trick thanks to a patch of bash, which adds a hook, called command_not_found_handler, but OpenSUSE/SLES is installed at work and this function is missing.

I had the feeling I can do this in bash only, and then I found this blog entry. Only I didn't quite like how it's implemented, though I liked the idea. Benjamin (the blog author) suggested that the check if the last command was successful is done at each prompt, so it is evaluated even if you press enter on an empty line. Besides, to obtain the last command, the whole history was retrieved only to extract the last entry.

The reason Benjamin's hook was done like that is that history expansion didn't work in noninteractive shells. I also tried to enable it by using set -H, but with no luck.

I thought that using a trap would be better than a prompt command. There was a trap, which would be evaluated only on error, so my first draft was based on the trap ERR hook. Then I would check if the exit status of the last command was 127 ("command not found" for bash) and... use the command. By chance I found out quickly about BASH_COMMAND, which in traps is evaluated to the currently executing command. But after a lot of failed attempts I found out that it only worked with the DEBUG trap.

So I used the DEBUG trap and evaluated the last command- if it could be resolved using type, then it is found. Slower, but more accurate. And it worked!


trap 'if ! type -t $BASH_COMMAND >/dev/null; then echo Do something with $BASH_COMMAND; fi' DEBUG


Now my colleagues are happily executing commands, which are only found on remote servers, oblivious to the fact that this command is transparently executing on a host different than their own.

Tuesday, June 10, 2008

Java Portable Apps


portable
–adjective

  1. capable of being transported or conveyed: a portable stage.

  2. easily carried or conveyed by hand: a portable typewriter.

  3. Computers. (of data sets, software, etc.) capable of being used on different computer systems.

  4. ...


Dictionary.com



So how many definitions of "portable" do your applications of choice satisfy? Having grown up profesionally in the spirit of "write once, run everywhere", I've always wondered how a "portable" application could run only on Windows, as is the case with most of the PortableApps.com. Not that I have anything against portableapps.com, it's a cool idea and a very pragmatic software suite is offered.

But I happen to work on Linux and my colleagues usually work on Windows, so if I want my applications-on-a-stick to be truly portable, they need to run on any platform. I need portable in the sense of "running on any operating system". Thus an obvious choice is Java- it's usually not the first choice for a desktop application, but here its intended purpose fits the bill nicely. And I don't need all operating systems- honestly, who would lend me their Mac? Come on, it's too personal to give to anyone ;-) This means I can have a JRE for Linux and Windows on my stick (in case it's not installed), and I'm set.

Of course, the task of finding the actual applications is the difficult one. They are not all completely portable in the sense of "not modifying anything on the hard disk", but most of them could be configured to run with settings from the USB. I also needed to run all of these on machines without administrator privileges. Most (with a few exceptions) are open-source:


  • Organizer: ThinkingRock
  • An outstanding application for organizing your tasks according to the Getting Things Done method. I am now so addicted to this app that I can't leave anywhere without it- I feel lost withot my next actions list. Con- it is the least portable in the sense that it creates some .java entries, but mostly related to layout- something I can live without.
  • Editor: jEdit
  • One of my all-time favorite editors, I really doubt there's much this editor can't do. With all of its useful plugins, it doubles as a mini-IDE. I run it with a command-line option to use the settings on the USB stick and the settings include the option to download my plugins there, so I have them with me as well.
  • Mind Mapper: FreeMind
  • Another immensely useful program, FreeMind is an open-source mind-mapping application, indispensable for loads of stuff like notes, brainstorming, personal database, task-manager...
  • Outliners
  • I couldn't choose between the extensibility and import/export capabilities of JOE (Java Outline Editor) and the rich text support of Jreepad (including Textile markup), so I have both. I really wish JOE's default shortcuts were a bit less awkward, but it's a great tool even though there's no development on it for several years now.
  • File Manager: muCommander
  • A couple of years ago I would be really bothered by the idea of using a Java file manager (Java doesn't have the best OS integration, you know). Now I'm happy to use this really nice lightweight commander clone. It has transparent filesystem support for popular archives and remote protocols (FTP, Windows shares, SFTP).
  • Disk space manager: JDiskReport
  • Warning: not open-source, but free for personal use. Generates very nice pie charts about filesystem usage though, and even has a Web Start version. I still want to be able to check my disk space when I'm not online, so it has a convenient place on my USB thumb drive.
  • Media organizer: MediaSort
  • Organizes music and pictures into directories/filenames based on tags inside these media files. Pretty nifty. I used to do that with a clunky one-liner script calling jhead for JPEG files and a custom Ruby script for MP3 files, but this proved to be a nicer general solution.
  • File Synchronization: JFileSync
  • I carry this around in case I don't have rsync handy. Its GUI looks very nice for syncing directories to and from my USB key.
  • Version Control: SmartSVN
  • Version control is must-have for a developer. SmartSVN is not open-source, but has a version, which is free for personal use. This was the only Subversion client I could find, which would work on Windows without administrative privileges, and use Windows authentication.
  • Port forwarders
  • Network connectivity is important to me so I have an assortment of port forwarding applications for different purposes. JPortForwarder is a simple port forwarder (site says it's multithreaded), so I don't have to rewrite one every time. PlugProxy is a really cool way to debug network applications, as it shows the network traffic as it transparently redirects it. jzbd adds encryption to forwarding when I'm worried about security. These are not updated in a long time, but what could you improve in a port forwarder?
  • Port Scanners
  • Yeah, I know the low-level scanning options of NMap are out of Java's reach, but still. JAPS makes a fast concurrent scan of a single host, while JMap can scan a subnet (no, it's got nothing to do with Sun's post-mortem memory analysis tool).


This set of programs proved especially useful when my laptop's hard drive reached the end of its intense life. Using a Damn Small Linux to boot from the USB disk along with these applications really decreased the time I could get productive again- without a hard disk- until I got a new hard-drive. To top it off, I could boot DSL in Windows, emulated by qemu- but DSL can really deserves a blog post on its own.

Update: This blog prompted me to do some more research and I found this great thread on Javalobby about Java Desktop Apps. There's at least one new cool app I am going away with- ekspos image viewer. It even has Picasa integration!