Tuesday, September 30, 2008

The browser as a platform

Google certainly know how to draw attention. They've even created a comic for dramatic effect to present their new browser. I remember it took a couple of hours to read it because it was constantly timing out because of the immense buzz-generated load on the servers.

Now after things have calmed down a little, there was a need to have a sober look at Chrome's competitor: Firefox. There was even an article explaining why Mozilla won't change their Gecko rendering engine with Webkit, the engine used in Chrome.

I'm not saying that Firefox is a better browser, but I think ArsTechnica is onto something. In my humble opinion (I hate abbreviations outside of chat), the fact that XUL can make it so much easier to write extensions is a key factor in its success. Even Steve Yegge mentions, half-jokingly, that Firefox is trying to be Emacs, and if it manages to edit files, he would dump Emacs (and if you know Steve Yegge, he lives for Emacs). Extensibility is again a key criteria here.

So now even Google has announced that Crome will support extensions. It will surely need them if it is to compete with Firefox.

I noticed a trend with my personal computing lately. I am using a very thin OS on an USB stick, which is 25-30MB (compressed). A lot of this is due to Firefox, and my externally kept profile is comparable to the OS size- mostly due to addons. Many of these addons are applications in their own right, comfortably using the networking capabilities of Firefox. Here are some of the extensions (some I use, some I don't), which are more like standalone applications than extensions to Firefox' functionality:

  • SamePlace
  • an XMPP instant messenger, which won the Extend Firefox 2 contest
  • TwitterFox
  • a Twitter client
  • FireFTP
  • an FTP client
  • mozImage
  • an image viewer
  • Firefly
  • a file manager

I'm not even mentioning diffent mail checkers, bookmark synchronizers, etc. Is there a tendency towards extensions which are more independent from the functionality of the browser?

Eventually, the browser is just a platform, an OS of sorts, another abstraction layer to make networking transparent for your Web-enabled apps.

Friday, June 27, 2008

Ruby for mp3 file organizing

So there we are, me and my friend Oliver, caught in a business trip. We're already bored of the only decent pub in the village, our families are a long distance away, so what's a developer to do? Code in a programming language he/she's not allowed on the job, of course! Oliver seemed interested in Ruby and I've already done a couple of small scripts with it, so we were curious to see what the fuss is about.

It goes without saying that if you want to learn to program (in a particular language) you should not rely too much on books. The only way is to find a task you want to have automated, and then code it using your language of choice. Surely, you must pick the task (or the language) carefully, since not all languages are suitable for all tasks.

One of the things which Oliver has struggled with was organizing all of his podcasts in his player, sorted neatly by directories of author and title. Having found both a hammer and a nail, we were ready to start pounding.

After a bit of research we found the mp3info and id3tag Ruby libraries. id3tag had different fields for ID3v1 and ID3v2 data and didn't have write support (not that we needed it). mp3info didn't have ID3v2.2 support, but I found an interesting link about ID3 internals- the format of the fields was something that could be useful.

After a while our pair programming session has reached a milestone- our script works. It doesn't seem very modular though, so we spend some time making classes and discussing what is the responsibility of each class. Should there be a manager-class? Or should the objects manage themselves? I go with the second approach, and here's the result:

# Class for handling information of the mp3 file
class Mp3File
attr_reader :title, :artist, :album

def initialize filename
@artist = @album = "unknown"
@filename = filename
@title = File.basename(filename, ".mp3")


def title
if @title == "unknown" then @title = File.basename(@filename, ".mp3") end

def read_attributes
Mp3Info.open(@filename) do |mp3info|
(@title, @artist, @album) = %w{title artist album}.collect { |attrib|
(result = mp3info.tag.send(attrib)).empty? ? "unknown" : result

def sanitize str

def transfer(newPath)
newPath = eval('"' + newPath + '"')
FileUtils.mkdir_p File.dirname(newPath)
FileUtils.cp @filename, newPath

This is the class which is initialized with the location of the file and then extracts information about the artist, title and track name. The read_attributes method is meant to show off our new knowledge about the dynamic nature of Ruby- we build a list of methods to invoke on the Mp3Info object, and if no meaningful result, return "unknown". Finally, as the class knows about the current location and mp3 meta-info, it has a method for copying the file to a new location. The new path is passed as a template, where the #@artist, #@album, #@title are substituted with the value of these fields.

class Mp3List

attr_reader :files

def files
@files.map {|file| Mp3File.new(file) }

def initialize(sourcePath, days = 7)
@sourcePath = sourcePath
@days = days
@files = read_new

def read_new
Dir["#@sourcePath/**/*.mp3"].find_all do |path|
test(?M, path) > (Time.now - (@days * 60 * 60 * 24))

def to_s

Here comes the class, which represents a list of mp3 files in a certain directory (and subdirectories), which satisfies some criteria- in this case, how long ago the files were created (modified). Could it be made more general? Certainly, but in a 80-line script? Maybe next time.

list = Mp3List.new("/home/whoami/Music", 730)
list.files.each do |mp3|
#~ puts "Processing #{filename}"

What's left was an example of how to use these classes. Seems good to me- and best of all, it works.

The only thing left was to prepare a patch for the mp3info library for ID3v2.2 support. I actually implemented one (still not incorporated in base), and it also initializes the common fields with either the v2 or v1 data, whatever present (v2 still has precedence, if both are present).

Conclusions from our short session:

  • Ruby is neat for quick hack jobs

  • mp3info does not provide an exhaustive ID3 handling support, but is good enough and workable

  • Pair programming might not be smooth from the start, but you will learn a lot about yourself

  • Organizing your music can sometimes take longer than total time spent looking for your tracks

  • You should choose your business trip accomodation place carefully if you can

Wednesday, June 11, 2008

command-not-found handler not only in Ubuntu

One of Ubuntu's strengths is that it pays attention to the little things. One of these things, which were important to me lately, was the command_not_found_handler in bash.

In layman's terms, when you open a terminal in Ubuntu and type a command, which is not on your system, it looks in a database and in a helpful way suggests that you can get this command if you install a certain package. I was using apt-file to provide the same functionality for a while, but this is both easier and more informative (apt-file scans for substrings of the whole path, so spits out a lot of false positives, which are not commands).

Now as happy as I am in using Ubuntu on my personal machine, I wanted to have this at work, too. I wanted to make it easy for my colleagues who forget to ssh/sudo before executing a command and prepend an appropriate string if the command doesn't work on this machine. Ubuntu can accomplish this trick thanks to a patch of bash, which adds a hook, called command_not_found_handler, but OpenSUSE/SLES is installed at work and this function is missing.

I had the feeling I can do this in bash only, and then I found this blog entry. Only I didn't quite like how it's implemented, though I liked the idea. Benjamin (the blog author) suggested that the check if the last command was successful is done at each prompt, so it is evaluated even if you press enter on an empty line. Besides, to obtain the last command, the whole history was retrieved only to extract the last entry.

The reason Benjamin's hook was done like that is that history expansion didn't work in noninteractive shells. I also tried to enable it by using set -H, but with no luck.

I thought that using a trap would be better than a prompt command. There was a trap, which would be evaluated only on error, so my first draft was based on the trap ERR hook. Then I would check if the exit status of the last command was 127 ("command not found" for bash) and... use the command. By chance I found out quickly about BASH_COMMAND, which in traps is evaluated to the currently executing command. But after a lot of failed attempts I found out that it only worked with the DEBUG trap.

So I used the DEBUG trap and evaluated the last command- if it could be resolved using type, then it is found. Slower, but more accurate. And it worked!

trap 'if ! type -t $BASH_COMMAND >/dev/null; then echo Do something with $BASH_COMMAND; fi' DEBUG

Now my colleagues are happily executing commands, which are only found on remote servers, oblivious to the fact that this command is transparently executing on a host different than their own.

Tuesday, June 10, 2008

Java Portable Apps


  1. capable of being transported or conveyed: a portable stage.

  2. easily carried or conveyed by hand: a portable typewriter.

  3. Computers. (of data sets, software, etc.) capable of being used on different computer systems.

  4. ...


So how many definitions of "portable" do your applications of choice satisfy? Having grown up profesionally in the spirit of "write once, run everywhere", I've always wondered how a "portable" application could run only on Windows, as is the case with most of the PortableApps.com. Not that I have anything against portableapps.com, it's a cool idea and a very pragmatic software suite is offered.

But I happen to work on Linux and my colleagues usually work on Windows, so if I want my applications-on-a-stick to be truly portable, they need to run on any platform. I need portable in the sense of "running on any operating system". Thus an obvious choice is Java- it's usually not the first choice for a desktop application, but here its intended purpose fits the bill nicely. And I don't need all operating systems- honestly, who would lend me their Mac? Come on, it's too personal to give to anyone ;-) This means I can have a JRE for Linux and Windows on my stick (in case it's not installed), and I'm set.

Of course, the task of finding the actual applications is the difficult one. They are not all completely portable in the sense of "not modifying anything on the hard disk", but most of them could be configured to run with settings from the USB. I also needed to run all of these on machines without administrator privileges. Most (with a few exceptions) are open-source:

  • Organizer: ThinkingRock
  • An outstanding application for organizing your tasks according to the Getting Things Done method. I am now so addicted to this app that I can't leave anywhere without it- I feel lost withot my next actions list. Con- it is the least portable in the sense that it creates some .java entries, but mostly related to layout- something I can live without.
  • Editor: jEdit
  • One of my all-time favorite editors, I really doubt there's much this editor can't do. With all of its useful plugins, it doubles as a mini-IDE. I run it with a command-line option to use the settings on the USB stick and the settings include the option to download my plugins there, so I have them with me as well.
  • Mind Mapper: FreeMind
  • Another immensely useful program, FreeMind is an open-source mind-mapping application, indispensable for loads of stuff like notes, brainstorming, personal database, task-manager...
  • Outliners
  • I couldn't choose between the extensibility and import/export capabilities of JOE (Java Outline Editor) and the rich text support of Jreepad (including Textile markup), so I have both. I really wish JOE's default shortcuts were a bit less awkward, but it's a great tool even though there's no development on it for several years now.
  • File Manager: muCommander
  • A couple of years ago I would be really bothered by the idea of using a Java file manager (Java doesn't have the best OS integration, you know). Now I'm happy to use this really nice lightweight commander clone. It has transparent filesystem support for popular archives and remote protocols (FTP, Windows shares, SFTP).
  • Disk space manager: JDiskReport
  • Warning: not open-source, but free for personal use. Generates very nice pie charts about filesystem usage though, and even has a Web Start version. I still want to be able to check my disk space when I'm not online, so it has a convenient place on my USB thumb drive.
  • Media organizer: MediaSort
  • Organizes music and pictures into directories/filenames based on tags inside these media files. Pretty nifty. I used to do that with a clunky one-liner script calling jhead for JPEG files and a custom Ruby script for MP3 files, but this proved to be a nicer general solution.
  • File Synchronization: JFileSync
  • I carry this around in case I don't have rsync handy. Its GUI looks very nice for syncing directories to and from my USB key.
  • Version Control: SmartSVN
  • Version control is must-have for a developer. SmartSVN is not open-source, but has a version, which is free for personal use. This was the only Subversion client I could find, which would work on Windows without administrative privileges, and use Windows authentication.
  • Port forwarders
  • Network connectivity is important to me so I have an assortment of port forwarding applications for different purposes. JPortForwarder is a simple port forwarder (site says it's multithreaded), so I don't have to rewrite one every time. PlugProxy is a really cool way to debug network applications, as it shows the network traffic as it transparently redirects it. jzbd adds encryption to forwarding when I'm worried about security. These are not updated in a long time, but what could you improve in a port forwarder?
  • Port Scanners
  • Yeah, I know the low-level scanning options of NMap are out of Java's reach, but still. JAPS makes a fast concurrent scan of a single host, while JMap can scan a subnet (no, it's got nothing to do with Sun's post-mortem memory analysis tool).

This set of programs proved especially useful when my laptop's hard drive reached the end of its intense life. Using a Damn Small Linux to boot from the USB disk along with these applications really decreased the time I could get productive again- without a hard disk- until I got a new hard-drive. To top it off, I could boot DSL in Windows, emulated by qemu- but DSL can really deserves a blog post on its own.

Update: This blog prompted me to do some more research and I found this great thread on Javalobby about Java Desktop Apps. There's at least one new cool app I am going away with- ekspos image viewer. It even has Picasa integration!

Thursday, May 8, 2008

SSH key-based authentication in Java

Yet again the next project of the week was the intersection of work and play (kinda). There was the upcoming project where I would need to exercise remote control using SSH. And there was the next program I was toying with: a nice lightweight commander clone written in Java. What's the common theme here? Well, muCommander has several virtual filesystems, including one for SFTP (and if you didn't know it before, that's a file transfer protocol for SSH).

But muCommander didn't use public key authentication, and I have disabled password-based authentication on my home SSH server. So there was the opportunity to play around with J2SSH (which muCommander was using) and both implement key-based authentication in my current file manager and explore one of the libraries for my next project. OK, maybe not applicable for my work project, because J2SSH is GPL, but I can get to play, OK?

With license questions out of the way, we can get down to business. And it is not difficult at all, as the j2ssh/examples directory contains a PublicKeyConnect.java. You just need to instantiate a com.sshtools.j2ssh.SshClient, call connect, passing the hostname as a String and authenticate using the method... which was it... ah yes authenticate, passing a com.sshtools.j2ssh.authentication.SshAuthenticationClient.

Now getting this SshAuthenticationClient is the fun part. In our case, we are using the com.sshtools.j2ssh.authentication.PublicKeyAuthenticationClient and we need several more steps to prep it up. After instantiating it, we need to call setUsername with a String (can't tell you what it stands for). Then we need to set the private key itself using the setKey method and we're ready.

What? setKey expects a parameter? Well yes it does, and it's another of those j2ssh classes: com.sshtools.j2ssh.transport.publickey.SshPrivateKey. But we seem to be stuck here, as you can't instantiate SshPrivateKey- it's abstract. What does this mean? Argh, never mind, you can't instantiate it. Try it. See? Told ya so.

So then we see there's this similarly named com.sshtools.j2ssh.transport.publickey.SshPrivateKeyFile. Might it have something to do with our SshPrivateKey? Yes, as it turns out, it has a method toPrivateKey (takes a passphrase as a String- you do use passphrases, don't you?), and returns our most wanted SshPrivateKey. What's that you're saying? You can't instantiate SshPrivateKeyFile either? Fear not, because it has a static method parse, which overloaded to accept both a File and a byte array with the actual private key file or data.

If you couldn't follow this convoluted line of thought (I couldn't), here's the code, as taken from the patch of muCommander:

PublicKeyAuthenticationClient pk = new PublicKeyAuthenticationClient();

SshPrivateKey key = null;
// Throw an AuthException if problems with private key file
try {
SshPrivateKeyFile file = SshPrivateKeyFile.parse(new File(privateKeyPath));
key = file.toPrivateKey(credentials.getPassword());
} catch (InvalidSshKeyException iske) {
throw new AuthException(realm, "Invalid private key file or passphrase"); // Todo: localize this entry
} catch (IOException ioe) {
throw new AuthException(realm, "Error reading private key file"); // Todo: localize this entry


Now if sshClient.authenticate returns AuthenticationProtocolState.COMPLETE, you're ready to go. You can open a shell or an SftpClient. Mission accomplished.

If you had the urge to download the muCommander source code and apply this patch, no need to: it's already implemented in CVS.

Friday, April 18, 2008

Twitter: do you follow me?

This week's hacking task was to implement a "follow all" function for Twitter.

Even for Twitter users, this needs some explanation: the follow functionality now means "enable notifications". However, the command interface in IM/SMS wasn't changed, so the command name remains "follow". For brevity, I will use the word "follow" instead of "enable notifications".

The reason for having this command is that there used to be a function "follow all" in Twitter. It used to instantly turn on notifications for all your friends (users you're following in new terminology). Now there's a user, called "all" and the function doesn't work (ok, maybe that's not the real reason). This put an end to a very useful feature for users who rely often on the Twitter IM integration.

Having a quick look at the Twitter API it seemed pretty straightforward to fetch all users and enable notifications for all of them one by one. It would be fairly slow, but there was no information in the user list whether notifications are enabled for a user or not. This would have eliminated the need to send requests for users, for whom we already have notifications enabled. Ah well...

The first tool I reach in my toolbox is Ruby. I tried using JSON, but had to give up- I simply couldn't handle Unicode issues:

/usr/lib/ruby/1.8/json.rb:288:in `chr': 1090 out of char range (RangeError)

It turned that it was much smoother with REXML, and it really is a superior library for XML processing (Python's are either easy or full-featured, REXML seems to be both).

I initially took the path of using 'open-uri' for fetching the data over http. After all, it handled even http base authentication and abstracted the nitty-gritty details, and so was easy to use.

But it isn't meant to be used for more fine-grained control, and I soon ran into performance problems, which required special treatment. I found that I quickly exhausted the rate limit of the Twitter API- it's only 70 requests per hour, and with one request per user... you get the picture. The web interface wasn't actually subject to such restrictions, so I wanted to check how it's doing it. A slightly different URL, but worked like a charm, and rate limits seemed to be no problem now!

This time, though, the script ran much longer- 80 seconds compared to about 30 before the change. I analyzed the requests and found out that each received a 302 response, forwarding back to the home page. That meant that open-uri was downloading the whole home page for each user!

At that point open-uri had to go and make way for Net::HTTP. It took more lines to rewrite it, but now I had the choice not to follow redirect responses. I only needed to toggle notifications and didn't care what I got back (as long as it's not an error code). In addition, I could use the same Net::HTTP object, meaning that I use the same HTTP keep-alive connection (not sure if open-uri can do this).

And here's the result- dirty, but still quick. You can configure the action to "follow" or "leave" (to disable all notifications). You need to configure the user and password. Putting the configuration options as command-line arguments is left as an exercise to the reader.

#!/usr/bin/env ruby

require 'uri'
require 'net/http'
require 'rexml/document'
include REXML

user = "lazyuser"
pass = "notmypassword"
action = "follow"

Net::HTTP.start("twitter.com") do |http|
page = 0
page += 1
req = Net::HTTP::Get.new("/statuses/friends.xml?lite=true&page=#{page}")
req.basic_auth(user, pass)

doc = Document.new(http.request(req).body)
ids = doc.elements.to_a("/users/user/id")
ids.each do |entry|
req_follow = Net::HTTP::Get.new("/friends/#{action}/" + entry.text)
req_follow.basic_auth(user, pass)
end while ids.size == PAGE_USERS

Wednesday, April 16, 2008

XML escape

Friday afternoon is a prime time for blitz-tasks and a rich opportunity for your hacking one-liner 5K1LLz skills.

This Friday the finish-up task came from a colleague, who had to leave in half an hour to catch the plane. There's a big several-megabyte XML file and all characters in it had to be escaped, I presume for preparation to be sent over the wire (read http).

Problem is, a certain Windows editor (does it really qualify that definition?) hangs when opening files bigger than 1234KB, and writing a Java program would take a fairly long time compared to the alternatives. Not to mention that many programmers could write the Java program even less memory-efficiently than the joke of an editor that Notepad is (there, I said it). And Java is not very forgiving on memory problems.

But what are the alternatives? As the Perl manual page says, "The three principal virtues of a programmer are Laziness, Impatience, and Hubris." Being a lazy programmer, I tried to see if somebody else had already written a utility to do this (there was zero chance that there wasn't one), and if it was available. First I found this eclipse plugin. However, pasting megabytes into a text box didn't make me confident that it would work.

There was also the xmlstarlet package, which would have done a wonderful job, had it been installed on the old servers where the file could be easily transferred. But it wasn't, while it would take too long to copy it to my machine and back just to convert the file. It would also be hard to find an appropriate package for that old Linux version. No, that's not for impatient programmers.

The next thought I had was: why spend effort on trying to install a package when with Ruby I could do this in a one-liner. Of course, I have nothing against Python, but if there's one thing nobody would argue is that it doesn't fare well against Ruby when it comes to writing one-liners. Anyway, Ruby wasn't installed there either (note to self: this must be amended).

The clock was ticking. So Python it is, and instead of an obfuscated one-liner I convinced myself to write many short readable lines. I hadn't done serious XML processing in Python for a while, but a google search away the answer came to me. It was really insultingly simple, but given enough hubris one could turn even this meager piece of code into a rambling blog post:

#!/usr/bin/env python

from xml.sax.saxutils import escape
from sys import stdin

for line in stdin:
print escape(line)

Thursday, April 10, 2008

Instant evaluation of Java code in OpenOffice

At the end of last year I had agreed to update a 4-day Java learning course to reflect the changes in Java 5 and 6. The problems with that were:

  • The course material was long- over 300 pages already
  • I know what you're thinking- that's 75 pages per freakin' day (not counting the exercises). All right, next time I'll publish a book
  • There was a deadline
  • ...tight as it usually is, considering it had to be done in off-work hours.
  • My first son was going to be born before the deadline
  • Uh-oh, now that should have made me worry.
  • The document was in Word .doc format
  • A big no-no. Maybe you have your reasons, but I have mine- I will never, ever write or maintain a significant piece of documentation like that (or insignificant, for that matter) ever again. And I mean it.

I really considered moving the contents to another format- be it LaTeX, docbook or a lightweight markup language like reStructuredText or asciidoc (a topic for another post). Now being out of time meant that I couldn't.

I should have also updated each and every single example for the new language syntax (where relevant) and test that it works. Now last I counted that was 254 snippets of code. OK, it could happen that one example was split across several snippets, but that's nonetheless tons of work- copy the code, paste it in a text editor, save it, compile it, run it, see if the results are the ones expected. If any step fails, rinse and repeat. Dirty work.

The reason I miss simple text formats is that it's so much easier to automate things. For instance, in the document you could include all your source files, which are located separately and tested automatically in one go.

But I didn't have this option. I needed to find some way to automate this. And the answer came from the very material I was going to present- Java Compiler API.

A frequently neglected feature in Java 6, Compiler API provided language libraries to control the process of compiling right in your Java code. So far you needed to save into a file and invoke a separate process to do that- really a workaround. If you've ever used eval in scripting language, you're going to miss instant evaluation sorely. With the new compiler API, well, you're still gonna miss it, but at least compiling is no longer a hack. Besides, control of the compiler now meant that your performance is going to be much better as you can even compile from a string in memory.

The idea began to form- I could define a BeanShell macro in OpenOffice. If we're using OpenOffice integrated with Java 6, then BeanShell will have access to the compiler APIs as well. The macro would compile a class from the selected text in the document (in memory) and load it (still in memory), then run it and display the results. This would certainly make testing the examples faster.

I'm usually lazy enough to first search for a similar solution (even if finding that solution takes more effort than writing it). The first source I came across about in-memory compilation was in the API documentation of JavaCompiler. It was a good start, it used SimpleJavaFileObject to read source from a String, but the class file was still compiled and saved to disk.

Along the same lines was the detailed article in JavaBeat- it showed how to compile from String using a SimpleJavaFileObject.

I knew there had to be more you need to do. The class file always appeared on disk. I was not very keen on the idea of writing a file to disk (implicitly or explicitly). It's slower, it's less secure and often more effort. I found what I was looking for in the velocityreviews forums. Bot I really struck gold with this really detailed document, which described almost exactly what I wanted to do. It's about visualizing the Java bytecode by the way.

There was one more point. I was wondering which graphical widget to use, and it turns out I could have a popup box using both the OpenOffice APIs or normal Java AWT/Swing. I chose OpenOffice, because the look and feel was better integrated- and because it was different than the Java libraries, which I already knew. I had to read a bit in the OpenOffice developer's guide, but it finally worked out.

So my final macro began to take shape. I first had to create the familiar SimpleJavaFileObject. I had to construct it with a String as an argument. The crucial point was to override the getCharContent method so it returns the class field with the String.

class JavaObjectFromString extends SimpleJavaFileObject{
private String contents = null;
public JavaObjectFromString(String className, String contents) throws Exception{
super(new URI(className + Kind.SOURCE.extension), Kind.SOURCE);
this.contents = contents;
public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException {
return contents;

What I discovered in the last two sources was that I would need to implement a file manager, preferrably by extending ForwardingJavaFileManager. However, it needs another reimplementation of SimpleJavaFileObject, but this time for the class data itself, not the source. The important thing here is to override openInputStream and openOutputStream to correct the notion of the class about where its data is located:

static class RAMJavaFileObject extends SimpleJavaFileObject {

RAMJavaFileObject(String name, Kind kind) {
super(URI.create("string:///" + name.replace('.','/') + Kind.SOURCE.extension), kind);

ByteArrayOutputStream baos;

public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException, IllegalStateException, UnsupportedOperationException {
throw new UnsupportedOperationException();

public InputStream openInputStream() throws IOException, IllegalStateException, UnsupportedOperationException {
return new ByteArrayInputStream(baos.toByteArray());

public OutputStream openOutputStream() throws IOException, IllegalStateException, UnsupportedOperationException {
return baos = new ByteArrayOutputStream();


Now we have everything necessary to extend on our ForwardinJavaFileManager so that it uses our implementation of in-memory class data. Note that we create a HashMap to be used later:

output = new HashMap();

class RAMFileManager extends ForwardingJavaFileManager{
RAMFileManager (JavaCompiler compiler){
public JavaFileObject getJavaFileForOutput(Location location, String name, Kind kind, FileObject sibling) throws java.io.IOException {
JavaFileObject jfo = new RAMJavaFileObject(name, kind);
output.put(name, jfo);
return jfo;

The last thing I define is a small utility method which just returns the String of the class. As you can see, I've made it convenient to select code fragments and the macro will create the necessary framework around them- a container class and a main method, if need be:

SimpleJavaFileObject getJavaFileContentsAsString(String outside, String inClass, String inMethod){
StringBuilder javaFileContents = new StringBuilder(outside +
"\npublic class TestClass{\n" +
inClass +
"\n public void testMethod() throws Throwable {\n" +
inMethod +
"try{this.getClass().getMethod(\"main\", String[].class).invoke(null, new Object[] {new String[]{}});} catch (NoSuchMethodException nsme) {}" +
"\n}\n" +
JavaObjectFromString javaFileObject = null;
javaFileObject = new JavaObjectFromString("TestClass", javaFileContents.toString());
}catch(Exception exception){
return javaFileObject;

Now that we have the main infrastructure in place it is time for the action to unfold. This means creating the compiler object, a new task and invoking the task. I do not really want to parse the java fragments in the document to find where they belong, so I use a trick. I assume the fragment belongs either outside of the class I'm generating (import statements, other classes), in the class definition (fields, methods) or in the main method (instructions). I try to put each selection (you haven't forgotten that you can have multiple selections in OpenOffice, right?) in one of these three areas and invoke the task to see if it compiles without error. If it does, I go ahead with the next one:

JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
JavaFileManager jfm = new RAMFileManager (compiler);

outside = inClass = inMethod = "";
for(i=0;i<count;i++) {
xTextRange = (XTextRange) UnoRuntime.queryInterface(XTextRange.class, xIndexAccess.getByIndex(i));
newText = xTextRange.getString();

JavaFileObject javaObjectFromString = getJavaFileContentsAsString(outside + newText, inClass, inMethod);
Iterable fileObjects = Arrays.asList(new Object[] {javaObjectFromString});
out = new StringWriter();
CompilationTask task = compiler.getTask(out, jfm, null, null, null, fileObjects);
Boolean result = task.call();

outside += newText;
} else {
javaObjectFromString = getJavaFileContentsAsString(outside, inClass + newText, inMethod);
fileObjects = Arrays.asList(new Object[] {javaObjectFromString});
task = compiler.getTask(out, jfm, null, null, null, fileObjects);
result = task.call();
if (result){
inClass += newText;
} else {
javaObjectFromString = getJavaFileContentsAsString(outside, inClass, inMethod + newText);
fileObjects = Arrays.asList(new Object[] {javaObjectFromString});
task = compiler.getTask(out, jfm, null, null, null, fileObjects);
result = task.call();
if (result){
inMethod += newText;
} else {
message = "Compilation fails:\n" + out.toString();
msgtype = "errorbox";
title = "Compilation error";
return 0;

At this stage, we already have compiled the class, but how to execute it? We need to define a class loader which knows specifically where to find and how to define the class (note the HashMap we used to store the classes):

ClassLoader cl = new ClassLoader() {
protected Class findClass(String name) throws ClassNotFoundException {
JavaFileObject jfo = output.get(name);
if (jfo != null) {
byte[] bytes = ((RAMJavaFileObject) jfo).baos.toByteArray();
return defineClass(name, bytes, 0, bytes.length);
return super.findClass(name);


Now that we have the class, we can get the method and run it. Here I'm also doing some hocus-pocus in order to kidnap the standard output and show it in the message box- I am really interested in what actually is printed and if there are any exceptions:

Class c = Class.forName("TestClass", true, cl);
constructor = c.getConstructor(new Class[]{});
object = constructor.newInstance(new Object[]{});
method = c.getMethod("testMethod", new Class[]{});

sysOut = System.out;
sysErr = System.err;
// System.setOut(out);
PipedOutputStream pout = new PipedOutputStream();
BufferedReader brout = new BufferedReader(new InputStreamReader(new PipedInputStream(pout)));
System.setOut(new PrintStream(pout));

PipedOutputStream perr = new PipedOutputStream();
BufferedReader brerr = new BufferedReader(new InputStreamReader(new PipedInputStream(perr)));
System.setErr(new PrintStream(perr));

message = "Compiled and ran successfully:\n";
msgtype = "infobox";
title = "Success";

try {
method.invoke(object, new Object[]{});
} catch (Throwable t) {
message = "Runtime error:\n" + t;
msgtype = "errorbox";
title = "Runtime error";
sb = new StringBuilder();
while (brout.ready()) {
sb.append((char) brout.read());
message += (sb.length() == 0 ? "" : "\nOutput:\n") + sb.toString();

sb = new StringBuilder();
while (brerr.ready()) {
sb.append((char) brerr.read());
message += (sb.length() == 0 ? "" : "\nError:\n") + sb.toString();



And that's it! What's that you ask? We don't know anything about the showMessage method? You're really curious, aren't you? OK, here it is, mostly taken from the OpenOffice UNO interface:

showMessage() {
import com.sun.star.awt.XToolkit;
import com.sun.star.awt.XMessageBoxFactory;
import com.sun.star.awt.XMessageBox;
import com.sun.star.awt.XWindowPeer;

m_xContext = XSCRIPTCONTEXT.getComponentContext();
m_xMCF = m_xContext.getServiceManager();

Object oToolkit = m_xMCF.createInstanceWithContext("com.sun.star.awt.Toolkit", m_xContext);
XToolkit xToolkit = (XToolkit) UnoRuntime.queryInterface(XToolkit.class, oToolkit);

windowPeer = xTextDoc.currentController.frame.containerWindow;
XWindowPeer xWindowPeer = (XWindowPeer) UnoRuntime.queryInterface(XWindowPeer.class, windowPeer);

com.sun.star.awt.Rectangle aRectangle = new com.sun.star.awt.Rectangle();

XMessageBoxFactory xMessageBoxFactory = (XMessageBoxFactory) UnoRuntime.queryInterface(XMessageBoxFactory.class, oToolkit);
XMessageBox xMessageBox = xMessageBoxFactory.createMessageBox(xWindowPeer, aRectangle, msgtype, com.sun.star.awt.MessageBoxButtons.BUTTONS_OK, title, message);
if (xMessageBox != null){
short nResult = xMessageBox.execute();

What's missing is a couple of import statements and standard initializing lines found in every OpenOffice macro template, but this blog post is already too long. And it's better that you ask if you need it rather than get bored and not even get to the end.

Wednesday, April 2, 2008

Playing with Javascript or what binds Greasemonkey, Twitter and Ambient Avatars together

It's been a while since I tried JavaScript hacking (almost 2 years). This time I had the haunting idea to create a Greasemonkey mashup so I can see my twitter page with the avatar next to each tweet exactly as it looked at the time the tweet was posted.

To do this the avatar history must be stored somewhere. That's where chinposin.com comes in. Initially originated as a refreshing avatar on Friday, it evolved into the Ambient Avatar Platform (TM) (credit goes to @monkchips and @yellowpark- you're great). In simple words- you follow @chinposin on twitter, and when you change your avatar, the old one is saved. So you have a gallery of all of your previous avatars for your previewing pleasure and along with the dates they were changed.

For those of you wondering what's twitter, that's a topic for an entire new blog post... or a whole blog, so start at wikipedia, so we can continue with the interesting stuff, shall we?

So there we are- we want to include info from one site into another- a task where Greasemonkey excels (normally JavaScript cannot just fetch info from any other site at whim).

I've obviously lost some of my JavaScript knowledge since it took me an obscene amount of time to get this tiny piece of code working. To start off, I had forgotten that Greasemonkey had also some restrictions, not only enhancements. For security purposes, a lot of objects were wrapped in XPCNativeWrapper and I had to use loads of wrappedJSObject as a workaround. Yes, I know it's not secure, and you should know this too.

Another issue I had a problem with was passing an argument to a closure. I eventually remembered that the closure is an object and you can just assign any field to an object, because each object is also an associative array. Accessing the function object from itself also took some googling- arguments.callee did the trick.

So is there anything that can be improved in this shoddy script? You bet. For starters, it loads the chinposin site a lot, sending 20 simultaneous requests right off the bat, even for duplicate user pages. I could cache the avatar history, but that would require that I synchronize the requests. This script could be modified into a Firefox extension, which has less restrictions than Greasemonkey. And I really should use a prototype for those twenty closures I create, but I gotta have something to do for next time, right?

Without further ado, here's the script. Copy it and paste it into twitteravatarhistory.user.js (OK, you can come up with a longer name if you're so inclined). Then open it with Firefox and if Greasemonkey is installed you will be presented with a dialog prompting you to install it. It's tested with Firefox, 3a9, 3b4 and Greasemonkey and 0.7.20080121.0. Considering the rate of change, I would be surprised it works in 1 year.

// ==UserScript==
// @name TwitterAvatarHistory
// @description Shows tweets with the avatar at time of posting
// @include http://twitter.com/*
// ==/UserScript==

// Assumptions:
// -chinposin.com has a special date string under the pic
// -avatars are listed chronologically
// -many others regarding DOM position

const avatar_home = "http://www.chinposin.com/home/";
var twitter_images = document.evaluate('//.[contains(@class, "hentry")]', document, null, XPathResult.UNORDERED_NODE_ITERATOR_TYPE, null);
while (message = twitter_images.iterateNext()) {
message = message.wrappedJSObject;

// Read user name
var url = message.getElementsByClassName("url")[0];
if (!url) continue;
var username = url.getAttribute("href").match("[^/]*$");

// Read date of message and extract fields with a regexp
var date_string = message.getElementsByClassName("published")[0].getAttribute("title");
var match = date_string.match(/(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2}):(\d{2})\+(\d{2}):(\d{2})/);
var date = new Date(match[1], match[2], match[3], match[4], match[5], match[6]);

var http = function(responseDetails) {
// add dummy element so we can operate on its DOM
var elem = document.createElement("html");
elem.innerHTML = responseDetails.responseText;

// getElementById is only found in document object, will use XPath
var gallery = document.evaluate('//.[@id="gallery"]', elem.wrappedJSObject, null,
XPathResult.ANY_UNORDERED_NODE_TYPE, null).singleNodeValue.wrappedJSObject;

// Might be better to couple these more tightly than creating two separate arrays
var images = gallery.getElementsByTagName("img");
var dates = gallery.getElementsByClassName("mainText");

// Find avatar date not more recent than message date
for (i = 0; i < dates.length; i++) {
var match = dates[i].textContent.match(/(\d{4})-(\d{2})-(\d{2}) +(\d{2}):(\d{2}):(\d{2})/);
var avatar_date = new Date(match[1], match[2], match[3], match[4], match[5], match[6]);

if (avatar_date < arguments.callee.date) {
// Replace message pic with avatar corresponding to date
arguments.callee.img.firstChild.setAttribute("src", images[i].getAttribute("src"))
//~ arguments.callee.img.replaceChild(images[i].cloneNode(false), arguments.callee.img.firstChild);

// clean up temp structure


// Trick to pass data to the closure
http.date = date;
http.img = message.getElementsByClassName("url")[0];

// Reach list of pix from user page
GM_xmlhttpRequest({method : "GET", url : avatar_home + username, onload : http});


Update: code formatting had munched some of the Greasemonkey header, that should be fixed now.

Update 10 April 2008: New code's on Greasemonkey repository since last week, today a fix was issued that adapts to twitter interface changes.

Thursday, March 20, 2008

A New Year's resolution

The start of each year invariably makes us look back and wish to bring more purpose to the next year. So did I have an improved New Year's Resolution? Of course- 1280x1024. A bit more significant, though, was to find another programming language to learn this year (following the sound advice from "Pragmatic" Dave Thomas).

Since we geeks can sometimes be programming language junkies, this was harder for me to do than previous years. I've previously improved my knowledge about Perl(2004), Python (2005), Tcl (2006) and Ruby (2007). They served their purpose but this year I felt it's about time to depart a bit from my comfort zone and try a new concept. As someone with much more experience noted, "A language that doesn't affect the way you think about programming, is not worth knowing".

So I did my homework and did a bit of research and nominated the following languages:
  • Scheme: yes I've done some functional programming in the university and I've edited an Emacs Lisp macro or two, but if you're not using it, you're losing it. Besides, I've always wanted (and postponed forever) reading "The Little Schemer", which has an almost cult status.
  • Logo: This elegant language is sometimes referred to as "LISP without the parentheses". I did some research about which programming language I should start teaching my son first (yeah I know it's ridiculously early to do that before your son is even born!) and decided that LOGO is the all-time favorite kids' programming language. Of course, being famous for its graph-drawing turtle does not mean it can't be used as a serious language. Anyway, it would be a pity if I don't learn it some day, given my relatives with some years of research on Logo.
  • Haskell: One of the most exotic in this mix, Haskell's lazy evaluation and monads sound novel enough to be fun to learn, besides it's getting some moderate popularity and there is even real-world software written in Haskell, like the Perl 6 compiler/interpreter and revision-control system Darcs (OK, Perl 6 is not a real-world software yet).
  • Erlang: One of the promises of Erlang is that it can deliver one of the easiest concurrency programming models with message passing between lightweight processes. Besides, it seems to be a fairly simple (compared to Haskell at least) functional language, while still preserving power features like pattern matching. Last, but not least, I really like hot class (module?) reloading and compared to the hot swap feature in the JVM it is really much more pervasive, so much easier for debugging, monitoring, quick fixes, etc.
  • Scala: It's not just object-oriented and it's not just functional, because it's a language with many influences- it even has pattern matching and Erlang's message passing model for concurrency. One of the best advantages is that it works on the JVM and integrates seamlessly with existing Java libraries.
  • Groovy: This is another promising dynamic language on the JVM, which seems to draw a lot of its influence from Ruby, including a Rails-like webapp framework, called Grails (surprising choice of name, huh?). Since my professional work is related to Java, I might learn this sooner or later because I think we would be seeing more from this (we already are). Especially for working with XML builders seem to be making things much smoother.
Interestingly, along with my new big-resolution display came a dual-core processor. Everyone knows that multiple cores will be all over the place soon. So my curiosity about how you can make these multiple processors scale based on the programming language model itself, took over. That leaves Erlang and Scala.

Of course that's not all of it. Logo's time just hadn't come yet, as my son is too young for me to start seriously teaching him (hell, he doesn't even know a human language to start with). Groovy could wait as well and it shouldn't be hard at all to pick up with Python, Ruby and Java behind my back. Erlang might be an easier first step into serious functional programming than Haskell and Scheme... well I guess I can postpone it for one more year and make do with Emacs Lisp.

But which one should it be- Erlang or Scala? Erlang is recommended by the same Pragmatic Dave who is responsible for me learning Ruby. And Scala's creator, Martin Odersky, is the man behind Java Generics and the current Java compiler.

But you know what? I didn't have to decide (or I might have learned none of these). My brother has the uncanny ability to guess what I want for a present, so he got me Programming Erlang from the Pragmatic bookshelf. And my father has the great ability to ask me what I want for a present, so he got me the Programming in Scala from Artima.

Eventually, I must have done a fairly good job on my language research because my list was not very different from the one in an article about new languages on Dr.Dobb's Journal.

On a side note- ordering PDF books is great- you can search easily, you don't destroy trees and you get the updates of new revisions easily- go for it!

So there I am, with two languages instead of one for this year, although Erlang and Scala have a lot in common. With the surge of interest in new programming languages one language per year only might take me forever to cover all I wanted to learn.