Is Scala more complicated than what Java tries to become?

Is Scala more complicated than Java? My last post did not tell the whole truth. I've only listed Scala features, which have a Java analog. There is a glaring omission of advanced Scala features like implicit conversions, operator overloading, call-by-name parameters and pattern matching. These Scala features are more complicated than what Java has. There, I said it. But then Scala is more complicated in the way a calculator is more complicated than an abacus- sure you can do some of the same stuff with an abacus, but trying to calculate the square root of a number is much more cumbersome.

However, this complexity pays off, because it lets us simplify many day-to-day features. This post will try a different angle by comparing where Java wants to be and where Scala is right now. I hope after reading it you will at least question your assumptions whether this trade-off is worth it.

Upon its creation, Java was a fairly simple language. A major reason it took over C++ is because it was specifically designed to steer away from multiple inheritance, automatic memory management and pointer arithmetic. But it's not a simple language anymore, and it's getting more and more complicated.

Why? Java wasn't designed to be too extensible. Scala, on the other hand, was designed to be scalable, in the sense of flexible syntax. The very creators of Java knew very well that a "main goal in designing a language should be to plan for growth" (Guy Steele's famous words from Growing a Language)

We need to expand our definition of language complexity. The language needs to be able to abstract away accidental complexity, or using it will be difficult. Examples of accidental complexity: jumping to a position in your program with goto, and then remembering to go back (before procedural programming); or allocating memory, and then remembering to deallocate it (before garbage collectors). Another example: using a counter to access collections, and remembering to initialize and increment it correctly, not to mention checking when we're done.

Creating extensions of the language in order to hide these complexities doesn't happen often. When it does, it offers huge rewards. On the other hand, if a language is rigid, even though it looks simple, this forces you to invent your own arcane workarounds. When the language leaves you to deal with complexity on your own, the resulting code will necessarily be complicated.

Let's see what special new language features Java tries to add to the language, which Scala can do because of its flexibility and extensibility.

Pattern matching

Pattern matching is often compared with Java's switch/case statement. I have listed pattern matching as something which doesn't have an analog in Java, because comparing it to "switch" really doesn't do it justice. Pattern matching can be used for arbitrary types, it can be used to assign variables and check preconditions; the compiler will check if the match is exhaustive and if the types make sense. Meanwhile Java has only recently accepted Strings in switch statements, which is only scratching the surface of Scala's pattern matching.

Furthermore, Scala is using pattern matching all through the language- from variable assignment to exception handling. To compare, the proposal for handling multiple exceptions in Java is postponed yet again.

Case classes

In order to get rid of Java's verbose getters, setters, hashCode and equals, one solution is to muck with the javac compiler, like the folks from Project Lombok have done. Is going to the guts of javac complicated? I'm sure it is.

In Scala, you can do it if you just define your classes as case classes.

Implicit conversions

In short, implicit conversions help transparently convert one type to another if the original type doesn't support the operations requested.

There are many examples where this is useful.

What in Java is hardcoded in the language as conversions and promotions, in Scala is defined using implicit conversions. This is another example where Java can get quite complicated. In most cases where you need to decide how to convert a method argument, for instance, you must have in mind narrowing and widening conversions, promotions, autoboxing, varargs and overriding (whew!). In Scala, the advantage of having implicit conversions is that you can inspect the code, where no ambiguity can result. You can analyze the conversions taking place in the interpreter by supplying the "-Xprint:typer" parameter. You can even disable these implicits, if you don't like them, by shadowing the import.

Another example of what implicits can do is adding methods and functionality to existing classes. Dynamic languages already do that easily using open classes and "missing method" handlers. In Java one way to do this using bytecode manipulation trickery via libraries like cglib, bcel, asm or javassist.

Bytecode manipulation in Java is required for popular libraries like Hibernate, Spring and AspectJ. Few "enterprise" Java developers can imagine development without Hibernate and Spring. Although there are many more things you can do with AspectJ, it can be used to emulate implicits with type member declarations. However, even though using AspectJ is a more high-level way to solve the problem, it adds even more complexity, as it defines additional keywords and constructs.

If you're new to Scala, you don't lose much if you don't know how implicit conversions work, just like you don't need to know about the magic that happens behind the scenes when Hibernate persists objects or when Spring creates its proxies. Just as with bytecode generation, you're not advised to use this feature often, as it is difficult to use. Still, you'll be glad it exists, because someone will create a library which will make your life and the life of many developers so much easier.

Operator overloading

The line between operators and methods in Scala is blurred- you can use the symbols +, -, /, *, etc. as method names. In fact, that's exactly how arithmetic operators work in Scala- they are method invocations (relax, everything is optimized by the compiler).

Some people object that operator overloading adds unnecessary complexity, because they can be abused. Still, you can also abuse method naming in much the same way. For instance, some hapless folk can define methods with visually similar symbols, like method1, methodl and methodI. They can use inconsistent capitalization, like addJar or addJAR. One could use meaningless identifiers like ahgsl. Why should operator best practices be different than method naming best practices?

What is complicated is treating numeric types like ints and BigInteger differently. Not only that, but operations with BigInteger are very verbose and barely readable even with simple expressions. To compare, this is how a recursive definition of factorial looks like in Scala with BigInteger:

def factorial (x: BigInt): BigInt =
  if (x == 0) 1 else x * factorial(x - 1)

This is how it would look if Scala didn't support operator overloading:

def factorial (x: BigInteger): BigInteger =
  if (x == BigInteger.ZERO)
    BigInteger.ONE
  else
    x.multiply(factorial(x.subtract(BigInteger.ONE)))

Call by name

One of the proposals for Java 7 language extension was automatic resource management. This is one more rule to the language, which you need to remember. Without this feature, code is also unnecessarily complicated, because it forces you to remember to always close resources after using them- if you slip up, subtle bugs with leaking files or connections can result.

In Scala, it's easy to add language constructs like this. Using function blocks, which are evaluated only when they are invoked, one can emulate almost any language construct, including while, if, etc..

Existential types

Existential types are roughly an alternative to Java wildcards, only more powerful.

Martin Odersky: If Java had reified types and no raw types or wildcards, I don't think we would have that much use for existential types and I doubt they would be in Scala.

If Martin Odersky says that existential types wouldn't be in the language if it wasn't for Java compatibility, why would you even need to know about them? Mostly if you need to interoperate with Java generics.

Conclusion

Scala tries to define fewer language rules, which are however more universal. Many of these advanced features are not often used, but they pay off by allowing to create constructs, which in Java would require specific hardcoded additions to the language. In Scala, they can be defined simply as libraries.

Why does it matter that it's in the libraries, and not hardcoded in the language? You can more easily evolve and adapt these features, you can add your own extensions, and you can even disable some of the library parts or replace them.

The conclusion is that if a language is not designed to be extended, it will eventually develop features, which are not well-integrated and this language will collapse under the weight of its own complexity.

Finally, learning something so that you avoid a lot of routine error-prone operations reduces effort by increasing the level of abstraction, at the cost of additional complexity. When you were in school, it was complicated to learn multiplication, but if you got over it, it would save you from quite a bit of repetition than if you just used addition.

P.S. I realize it's not possible to resolve the issue once and for all which language is more complicated- Java or Scala- in a post or two. First of all, have in mind that simple is not the same as easy to use. There are also many topics which are open for discussion. I haven't touched on Scala traits; I haven't mentioned functions as first-class constructs compared to the Java 7 closure proposal; and there's a lot that can be said about how Scala obviates many Java design patterns. Extending the Scala syntax via compiler plugins is another interesting advanced topic.

I suppose someone could even write a blog post about these topics some day.

Is Scala more complicated than Java?

One Scala-related thread on Artima drew a lot of attention: "Is Scala really more complicated than Java?". This post really struck a nerve. Whoever claims that Scala is much more complicated than Java has clearly not seen a Java Programmer Certification in a while and is probably not using many new features since Java 5 came out.

What I'll try to prove in this post is not that Scala is not a complicated language. There are certainly many languages which are simpler. The core features which are used reasonably often are indeed a simplification over Java. Scala also has features which are more complicated than what Java has. However, the complicated Scala features are more specialized at extending the language while the complexity of Java is usually imposed on everyone including the beginner.

This post will also not try to describe the language features in exhaustive detail- that's what the language specification tries to achieve, and the blog post is already long enough. I will assume that you know about the core language rules or can easily look them up.

What is complexity? Many conflate it with unreadability, some say it's the opposite to ease of use. Let's start with the following definition of complexity: it's the many special exceptions (pun not intended) to the rules, making the whole system difficult to understand and remember.

Based on that definition, let's run a comparison of language features of Java and Scala.

Keywords

Java has more reserved words than Scala. Even when we remove the words of primitive types (boolean, int, float, etc.) Scala still has less keywords!

• Scala misses some of Java's control structures
Yes, continue and break (well, at least until Scala 2.8) are not part of the language, as they are not deemed a best-practice way to solve programming problems.

• if/then/else
in Scala returns a value, thus eliminating the need for the only ternary operator in Java, "?:", the use of which has long been discouraged.

• for loop
Java folks discovered late in the game that the enhanced for loop is much less complicated to use in cases when you don't need the counter index. And they were right- it's one more detail which you (or at least newbies) can get wrong. But why stop there- Scala has a much more universal for loop, and there aren't two different syntaxes as in Java.

• Scala keywords not in Java
one might argue that the override keyword in Scala complicates things as you might do the same thing in Java with an @Override annotation. That's not quite the case, as you still might override a method by accident and forget to put the annotation (as it's not mandatory), and then the compiler will not give as much as a warning! So that's one more special case you need to worry about and keep in your head. When you start using traits, you definitely start to appreciate that override is a keyword.

Access modifiers

Java has four access modifiers: default (package), private, public and protected. Scala has three, but it can combine them with a scope for protection. This flexibility allows you to define access control, for which Java has no analogs. They are also readable because they look self explanatory. For instance, if you have the class org.apache.esme.actor.UserActor, these are the Scala equivalents for Java's access modifiers:

private[actor]

Same as package access in Java

private[UserActor]

Same as private in Java

Scala's default access is public (yay, one less keyword!).

On the other hand, by defining the scope, Scala allows visibility types, which Java doesn't have:

private[this]

the member is accessible only from this instance. Not even other instances from the same class can access it

private[esme]

access for a package and subpackages. How many times did you have to use clumsy workarounds in Java because subpackages couldn't access the parent package?

protected

only subclasses can access the field. This is more consistent than Java's protected keyword. In Java, you have to remember that both subclasses and classes in the same package can access the field. How's that for remembering arbitrary rules?

private

This will not allow you to access inner objects. This is also more consistent than Java's private access, which is perhaps indicative of the fact that inner objects in Java were bolted on after the first version of the language was created

Namespaces

Java has four namespaces (fields, methods, packages, types), Scala has two (one for types, one for everything else). This has important implications when overriding. In Scala, you can start with a parameterless def (method definition) and then override with a val (immutable variable). This helps enforce the uniform access principle: whether you access a field or a method, this doesn't restrict you later on, because the syntax is the same everywhere you access it.

One thing which you cannot do is define a variable and a method with the same method. The "Programming in Scala" book explicitly mentions that allowing this would be a code smell because of possible code duplication and the ambiguity which could arise.

Types and the type system

Scala's type system has been criticized for being too complex, but let's have a look at Java's type system.

Primitive types

There are many exceptions in Java's type system, which make the language hard to use and evolve. Java has primitive types, while in Scala everything is an object, making this valid code:

(1 to 3) foreach println

Java's primitive types make a lot of things harder- before Java 5, manual wrapping in collections, e.g. int in Integer, boolean in Boolean, etc. was a pain. After auto-boxing and unboxing came out, the syntax is cleaner, but the overhead remains, as well as some very subtle bugs. For example, auto-unboxing a null wrapper type will cause a NullPointerException.

There is a lot of code duplication because of primitive types- you always have to specify special cases and can't be truly generic.

Generics

Scala has generics done right. For instance, you can define covariance at the declaration site, whereas Java requires you to do this at the call site. This, combined with Scala's type inference allows one to use generified libraries without having to know or define the complete type signature.

Java has yet another "special" type: arrays. As a result the rules for the underlying array type and the generified ArrayList are quite different and inconsistent. The type of arrays is checked at runtime, while the genericity of ArrayList is checked at compile-time. As a result, inappropriate assignment to an array element results in a ArrayStoreException only at runtime.

Constructors

Java initialization order in object construction is a pain to get right. The puzzlers on the certification exam use the most bizarre mix of static and instance initializer blocks and constructors calling or inheriting other constructors.

In Scala, any code which isn't part of a method or member variable declaration is executed as the primary constructor. You can define auxiliary constructors which call either the primary one or another auxiliary one defined in the same class before it. Can you come up with anything simpler?

Uniform syntax

Scala is sometimes accused of using too many symbols. Whatever you've seen, it's mostly not part of the language, but of the libraries. You can override them and even disable them. What does Java have to say about special symbols?

Arrays

Arrays in Java are accessed via square brackets. In Scala, parentheses are used, because accessing an array index is a method call (called apply). Don't worry though, the compiler optimizes this away.

Collection literals

Java has a lot of special syntax for array instantiation and will soon have ones for instantiating lists, sets and maps. Scala, again, creates these special collections using factory methods in the companion objects: Array(1,2,3), List(1,2,3), Map((1,2)). Lists can also be created using the cons "operator", but here's the trick: it's not actually an operator. It's a method, appending to the list. You can also create a map using the arrow tuple syntax: Map(1->2). And again, this is not "special" syntax, which is part of the language- it's a method, constructing a tuple!

Now someone might smirk and think: "Ha, gotcha! Do you mean to say that simply because you've pushed the complexity out of the language and into the libraries you don't need to deal with it?". True, but let's have a look at Java and its ever growing standard libraries. It has AWT and Swing. It has old I/O and new I/O (gosh, which one do I use?). It has RMI (do you remember RMI?) and OMG, it even has CORBA. These libraries will never die. Methods in Thread have been deprecated for ages. There's also no sign that the ill-conceived Date/Calendar classes will ever be removed, but you still must know JodaTime if you hope to get any job with dates done.

More importantly, extending the language easily helps abstract away the details and evolve the language without creating tons of special cases. As per our definition, special cases add up to increase complexity. We'll explore the topic of extending the language in the followup post.

Java Portable Apps

portable
–adjective

1. capable of being transported or conveyed: a portable stage.


2. easily carried or conveyed by hand: a portable typewriter.


3. Computers. (of data sets, software, etc.) capable of being used on different computer systems.


4. ...

Dictionary.com

So how many definitions of "portable" do your applications of choice satisfy? Having grown up profesionally in the spirit of "write once, run everywhere", I've always wondered how a "portable" application could run only on Windows, as is the case with most of the PortableApps.com. Not that I have anything against portableapps.com, it's a cool idea and a very pragmatic software suite is offered.

But I happen to work on Linux and my colleagues usually work on Windows, so if I want my applications-on-a-stick to be truly portable, they need to run on any platform. I need portable in the sense of "running on any operating system". Thus an obvious choice is Java- it's usually not the first choice for a desktop application, but here its intended purpose fits the bill nicely. And I don't need all operating systems- honestly, who would lend me their Mac? Come on, it's too personal to give to anyone ;-) This means I can have a JRE for Linux and Windows on my stick (in case it's not installed), and I'm set.

Of course, the task of finding the actual applications is the difficult one. They are not all completely portable in the sense of "not modifying anything on the hard disk", but most of them could be configured to run with settings from the USB. I also needed to run all of these on machines without administrator privileges. Most (with a few exceptions) are open-source:

• Organizer: ThinkingRock
An outstanding application for organizing your tasks according to the Getting Things Done method. I am now so addicted to this app that I can't leave anywhere without it- I feel lost withot my next actions list. Con- it is the least portable in the sense that it creates some .java entries, but mostly related to layout- something I can live without.

• Editor: jEdit
One of my all-time favorite editors, I really doubt there's much this editor can't do. With all of its useful plugins, it doubles as a mini-IDE. I run it with a command-line option to use the settings on the USB stick and the settings include the option to download my plugins there, so I have them with me as well.

• Mind Mapper: FreeMind
Another immensely useful program, FreeMind is an open-source mind-mapping application, indispensable for loads of stuff like notes, brainstorming, personal database, task-manager...

• Outliners
I couldn't choose between the extensibility and import/export capabilities of JOE (Java Outline Editor) and the rich text support of Jreepad (including Textile markup), so I have both. I really wish JOE's default shortcuts were a bit less awkward, but it's a great tool even though there's no development on it for several years now.

• File Manager: muCommander
A couple of years ago I would be really bothered by the idea of using a Java file manager (Java doesn't have the best OS integration, you know). Now I'm happy to use this really nice lightweight commander clone. It has transparent filesystem support for popular archives and remote protocols (FTP, Windows shares, SFTP).

• Disk space manager: JDiskReport
Warning: not open-source, but free for personal use. Generates very nice pie charts about filesystem usage though, and even has a Web Start version. I still want to be able to check my disk space when I'm not online, so it has a convenient place on my USB thumb drive.

• Media organizer: MediaSort
Organizes music and pictures into directories/filenames based on tags inside these media files. Pretty nifty. I used to do that with a clunky one-liner script calling jhead for JPEG files and a custom Ruby script for MP3 files, but this proved to be a nicer general solution.

• File Synchronization: JFileSync
I carry this around in case I don't have rsync handy. Its GUI looks very nice for syncing directories to and from my USB key.

• Version Control: SmartSVN
Version control is must-have for a developer. SmartSVN is not open-source, but has a version, which is free for personal use. This was the only Subversion client I could find, which would work on Windows without administrative privileges, and use Windows authentication.

• Port forwarders
Network connectivity is important to me so I have an assortment of port forwarding applications for different purposes. JPortForwarder is a simple port forwarder (site says it's multithreaded), so I don't have to rewrite one every time. PlugProxy is a really cool way to debug network applications, as it shows the network traffic as it transparently redirects it. jzbd adds encryption to forwarding when I'm worried about security. These are not updated in a long time, but what could you improve in a port forwarder?

• Port Scanners
Yeah, I know the low-level scanning options of NMap are out of Java's reach, but still. JAPS makes a fast concurrent scan of a single host, while JMap can scan a subnet (no, it's got nothing to do with Sun's post-mortem memory analysis tool).

This set of programs proved especially useful when my laptop's hard drive reached the end of its intense life. Using a Damn Small Linux to boot from the USB disk along with these applications really decreased the time I could get productive again- without a hard disk- until I got a new hard-drive. To top it off, I could boot DSL in Windows, emulated by qemu- but DSL can really deserves a blog post on its own.

Update: This blog prompted me to do some more research and I found this great thread on Javalobby about Java Desktop Apps. There's at least one new cool app I am going away with- ekspos image viewer. It even has Picasa integration!

SSH key-based authentication in Java

Yet again the next project of the week was the intersection of work and play (kinda). There was the upcoming project where I would need to exercise remote control using SSH. And there was the next program I was toying with: a nice lightweight commander clone written in Java. What's the common theme here? Well, muCommander has several virtual filesystems, including one for SFTP (and if you didn't know it before, that's a file transfer protocol for SSH).

But muCommander didn't use public key authentication, and I have disabled password-based authentication on my home SSH server. So there was the opportunity to play around with J2SSH (which muCommander was using) and both implement key-based authentication in my current file manager and explore one of the libraries for my next project. OK, maybe not applicable for my work project, because J2SSH is GPL, but I can get to play, OK?

With license questions out of the way, we can get down to business. And it is not difficult at all, as the j2ssh/examples directory contains a PublicKeyConnect.java. You just need to instantiate a com.sshtools.j2ssh.SshClient, call connect, passing the hostname as a String and authenticate using the method... which was it... ah yes authenticate, passing a com.sshtools.j2ssh.authentication.SshAuthenticationClient.

Now getting this SshAuthenticationClient is the fun part. In our case, we are using the com.sshtools.j2ssh.authentication.PublicKeyAuthenticationClient and we need several more steps to prep it up. After instantiating it, we need to call setUsername with a String (can't tell you what it stands for). Then we need to set the private key itself using the setKey method and we're ready.

What? setKey expects a parameter? Well yes it does, and it's another of those j2ssh classes: com.sshtools.j2ssh.transport.publickey.SshPrivateKey. But we seem to be stuck here, as you can't instantiate SshPrivateKey- it's abstract. What does this mean? Argh, never mind, you can't instantiate it. Try it. See? Told ya so.

So then we see there's this similarly named com.sshtools.j2ssh.transport.publickey.SshPrivateKeyFile. Might it have something to do with our SshPrivateKey? Yes, as it turns out, it has a method toPrivateKey (takes a passphrase as a String- you do use passphrases, don't you?), and returns our most wanted SshPrivateKey. What's that you're saying? You can't instantiate SshPrivateKeyFile either? Fear not, because it has a static method parse, which overloaded to accept both a File and a byte array with the actual private key file or data.

If you couldn't follow this convoluted line of thought (I couldn't), here's the code, as taken from the patch of muCommander:

PublicKeyAuthenticationClient pk = new PublicKeyAuthenticationClient();
pk.setUsername(credentials.getLogin());

SshPrivateKey key = null;
// Throw an AuthException if problems with private key file
try {
    SshPrivateKeyFile file = SshPrivateKeyFile.parse(new File(privateKeyPath));
    key = file.toPrivateKey(credentials.getPassword());
} catch (InvalidSshKeyException iske) {
    throw new AuthException(realm, "Invalid private key file or passphrase");  // Todo: localize this entry
} catch (IOException ioe) {
    throw new AuthException(realm, "Error reading private key file");  // Todo: localize this entry
}

pk.setKey(key);

Now if sshClient.authenticate returns AuthenticationProtocolState.COMPLETE, you're ready to go. You can open a shell or an SftpClient. Mission accomplished.

If you had the urge to download the muCommander source code and apply this patch, no need to: it's already implemented in CVS.

Instant evaluation of Java code in OpenOffice

At the end of last year I had agreed to update a 4-day Java learning course to reflect the changes in Java 5 and 6. The problems with that were:

• The course material was long- over 300 pages already
I know what you're thinking- that's 75 pages per freakin' day (not counting the exercises). All right, next time I'll publish a book

• There was a deadline
...tight as it usually is, considering it had to be done in off-work hours.

• My first son was going to be born before the deadline
Uh-oh, now that should have made me worry.

• The document was in Word .doc format
A big no-no. Maybe you have your reasons, but I have mine- I will never, ever write or maintain a significant piece of documentation like that (or insignificant, for that matter) ever again. And I mean it.

I really considered moving the contents to another format- be it LaTeX, docbook or a lightweight markup language like reStructuredText or asciidoc (a topic for another post). Now being out of time meant that I couldn't.

I should have also updated each and every single example for the new language syntax (where relevant) and test that it works. Now last I counted that was 254 snippets of code. OK, it could happen that one example was split across several snippets, but that's nonetheless tons of work- copy the code, paste it in a text editor, save it, compile it, run it, see if the results are the ones expected. If any step fails, rinse and repeat. Dirty work.

The reason I miss simple text formats is that it's so much easier to automate things. For instance, in the document you could include all your source files, which are located separately and tested automatically in one go.

But I didn't have this option. I needed to find some way to automate this. And the answer came from the very material I was going to present- Java Compiler API.

A frequently neglected feature in Java 6, Compiler API provided language libraries to control the process of compiling right in your Java code. So far you needed to save into a file and invoke a separate process to do that- really a workaround. If you've ever used eval in scripting language, you're going to miss instant evaluation sorely. With the new compiler API, well, you're still gonna miss it, but at least compiling is no longer a hack. Besides, control of the compiler now meant that your performance is going to be much better as you can even compile from a string in memory.

The idea began to form- I could define a BeanShell macro in OpenOffice. If we're using OpenOffice integrated with Java 6, then BeanShell will have access to the compiler APIs as well. The macro would compile a class from the selected text in the document (in memory) and load it (still in memory), then run it and display the results. This would certainly make testing the examples faster.

I'm usually lazy enough to first search for a similar solution (even if finding that solution takes more effort than writing it). The first source I came across about in-memory compilation was in the API documentation of JavaCompiler. It was a good start, it used SimpleJavaFileObject to read source from a String, but the class file was still compiled and saved to disk.

Along the same lines was the detailed article in JavaBeat- it showed how to compile from String using a SimpleJavaFileObject.

I knew there had to be more you need to do. The class file always appeared on disk. I was not very keen on the idea of writing a file to disk (implicitly or explicitly). It's slower, it's less secure and often more effort. I found what I was looking for in the velocityreviews forums. Bot I really struck gold with this really detailed document, which described almost exactly what I wanted to do. It's about visualizing the Java bytecode by the way.

There was one more point. I was wondering which graphical widget to use, and it turns out I could have a popup box using both the OpenOffice APIs or normal Java AWT/Swing. I chose OpenOffice, because the look and feel was better integrated- and because it was different than the Java libraries, which I already knew. I had to read a bit in the OpenOffice developer's guide, but it finally worked out.

So my final macro began to take shape. I first had to create the familiar SimpleJavaFileObject. I had to construct it with a String as an argument. The crucial point was to override the getCharContent method so it returns the class field with the String.

class JavaObjectFromString extends SimpleJavaFileObject{
    private String contents = null;
    public JavaObjectFromString(String className, String contents) throws Exception{
        super(new URI(className + Kind.SOURCE.extension), Kind.SOURCE);
        this.contents = contents;
    }
    public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException {
        return contents;
    }
}

What I discovered in the last two sources was that I would need to implement a file manager, preferrably by extending ForwardingJavaFileManager. However, it needs another reimplementation of SimpleJavaFileObject, but this time for the class data itself, not the source. The important thing here is to override openInputStream and openOutputStream to correct the notion of the class about where its data is located:

static class RAMJavaFileObject extends SimpleJavaFileObject {

    RAMJavaFileObject(String name, Kind kind) {
        super(URI.create("string:///" + name.replace('.','/') + Kind.SOURCE.extension), kind);
    }

    ByteArrayOutputStream baos;

    public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException, IllegalStateException, UnsupportedOperationException {
        throw new UnsupportedOperationException();
    }

    public InputStream openInputStream() throws IOException, IllegalStateException, UnsupportedOperationException {
        return new ByteArrayInputStream(baos.toByteArray());
    }

    public OutputStream openOutputStream() throws IOException, IllegalStateException, UnsupportedOperationException {
        return baos = new ByteArrayOutputStream();
    }

}

Now we have everything necessary to extend on our ForwardinJavaFileManager so that it uses our implementation of in-memory class data. Note that we create a HashMap to be used later:

output = new HashMap();

class RAMFileManager extends ForwardingJavaFileManager{
    RAMFileManager (JavaCompiler compiler){
        super(compiler.getStandardFileManager(null,null,null));
    }
    public JavaFileObject getJavaFileForOutput(Location location, String name, Kind kind, FileObject sibling) throws java.io.IOException {
        JavaFileObject jfo = new RAMJavaFileObject(name, kind);
        output.put(name, jfo);
        return jfo;
    }
}

The last thing I define is a small utility method which just returns the String of the class. As you can see, I've made it convenient to select code fragments and the macro will create the necessary framework around them- a container class and a main method, if need be:

SimpleJavaFileObject getJavaFileContentsAsString(String outside, String inClass, String inMethod){
    StringBuilder javaFileContents = new StringBuilder(outside +
            "\npublic class TestClass{\n" +
            inClass +
            "\n    public void testMethod() throws Throwable {\n" +
            inMethod +
            "try{this.getClass().getMethod(\"main\", String[].class).invoke(null, new Object[] {new String[]{}});} catch (NoSuchMethodException nsme) {}" +
            "\n}\n" +
            "}");
    JavaObjectFromString javaFileObject = null;
    try{
        javaFileObject = new JavaObjectFromString("TestClass", javaFileContents.toString());
    }catch(Exception exception){
        exception.printStackTrace();
    }
    return javaFileObject;
}

Now that we have the main infrastructure in place it is time for the action to unfold. This means creating the compiler object, a new task and invoking the task. I do not really want to parse the java fragments in the document to find where they belong, so I use a trick. I assume the fragment belongs either outside of the class I'm generating (import statements, other classes), in the class definition (fields, methods) or in the main method (instructions). I try to put each selection (you haven't forgotten that you can have multiple selections in OpenOffice, right?) in one of these three areas and invoke the task to see if it compiles without error. If it does, I go ahead with the next one:

JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
JavaFileManager jfm = new RAMFileManager (compiler);

outside = inClass = inMethod = "";
for(i=0;i<count;i++) {
    xTextRange = (XTextRange) UnoRuntime.queryInterface(XTextRange.class, xIndexAccess.getByIndex(i));
    newText = xTextRange.getString();

    JavaFileObject javaObjectFromString = getJavaFileContentsAsString(outside + newText, inClass, inMethod);
    Iterable fileObjects = Arrays.asList(new Object[] {javaObjectFromString});
    out = new StringWriter();
    CompilationTask task = compiler.getTask(out, jfm, null, null, null, fileObjects);
    Boolean result = task.call();

    if(result){
        outside += newText;
    } else {
        javaObjectFromString = getJavaFileContentsAsString(outside, inClass + newText, inMethod);
        fileObjects = Arrays.asList(new Object[] {javaObjectFromString});
        task = compiler.getTask(out, jfm, null, null, null, fileObjects);
        result = task.call();
        if (result){
            inClass += newText;
        } else {
            javaObjectFromString = getJavaFileContentsAsString(outside, inClass, inMethod + newText);
            fileObjects = Arrays.asList(new Object[] {javaObjectFromString});
            task = compiler.getTask(out, jfm, null, null, null, fileObjects);
            result = task.call();
            if (result){
                inMethod += newText;
            } else {
                message = "Compilation fails:\n" + out.toString();
                msgtype = "errorbox";
                title = "Compilation error";
                showMessage();
                return 0;
            }
        }
    }
}

At this stage, we already have compiled the class, but how to execute it? We need to define a class loader which knows specifically where to find and how to define the class (note the HashMap we used to store the classes):

ClassLoader cl = new ClassLoader() {
    protected Class findClass(String name) throws ClassNotFoundException {
        JavaFileObject jfo = output.get(name);
        if (jfo != null) {
            byte[] bytes = ((RAMJavaFileObject) jfo).baos.toByteArray();
            return defineClass(name, bytes, 0, bytes.length);
        }
        return super.findClass(name);
    }

};

Now that we have the class, we can get the method and run it. Here I'm also doing some hocus-pocus in order to kidnap the standard output and show it in the message box- I am really interested in what actually is printed and if there are any exceptions:

Class c = Class.forName("TestClass", true, cl);
constructor = c.getConstructor(new Class[]{});
object = constructor.newInstance(new Object[]{});
method = c.getMethod("testMethod", new Class[]{});

sysOut = System.out;
sysErr = System.err;
// System.setOut(out);
PipedOutputStream pout = new PipedOutputStream();
BufferedReader brout = new BufferedReader(new InputStreamReader(new PipedInputStream(pout)));
System.setOut(new PrintStream(pout));

PipedOutputStream perr = new PipedOutputStream();
BufferedReader brerr = new BufferedReader(new InputStreamReader(new PipedInputStream(perr)));
System.setErr(new PrintStream(perr));

message = "Compiled and ran successfully:\n";
msgtype = "infobox";
title = "Success";

try {
    method.invoke(object, new Object[]{});
} catch (Throwable t) {
    message = "Runtime error:\n" + t;
    msgtype = "errorbox";
    title = "Runtime error";
}
sb = new StringBuilder();
while (brout.ready()) {
    sb.append((char) brout.read());
}
message += (sb.length() == 0 ? "" : "\nOutput:\n") + sb.toString();

sb = new StringBuilder();
while (brerr.ready()) {
    sb.append((char) brerr.read());
}
message += (sb.length() == 0 ? "" : "\nError:\n") + sb.toString();

System.setOut(sysOut);
System.setErr(sysErr);

showMessage();

And that's it! What's that you ask? We don't know anything about the showMessage method? You're really curious, aren't you? OK, here it is, mostly taken from the OpenOffice UNO interface:

showMessage() {
    import com.sun.star.awt.XToolkit;
    import com.sun.star.awt.XMessageBoxFactory;
    import com.sun.star.awt.XMessageBox;
    import com.sun.star.awt.XWindowPeer;

    m_xContext = XSCRIPTCONTEXT.getComponentContext();
    m_xMCF = m_xContext.getServiceManager();

    Object oToolkit = m_xMCF.createInstanceWithContext("com.sun.star.awt.Toolkit", m_xContext);
    XToolkit xToolkit = (XToolkit) UnoRuntime.queryInterface(XToolkit.class, oToolkit);

    windowPeer = xTextDoc.currentController.frame.containerWindow;
    XWindowPeer xWindowPeer = (XWindowPeer) UnoRuntime.queryInterface(XWindowPeer.class, windowPeer);

    com.sun.star.awt.Rectangle aRectangle = new com.sun.star.awt.Rectangle();

    XMessageBoxFactory xMessageBoxFactory = (XMessageBoxFactory) UnoRuntime.queryInterface(XMessageBoxFactory.class, oToolkit);
    XMessageBox xMessageBox = xMessageBoxFactory.createMessageBox(xWindowPeer, aRectangle, msgtype, com.sun.star.awt.MessageBoxButtons.BUTTONS_OK, title, message);
    if (xMessageBox != null){
        short nResult = xMessageBox.execute();
    }
}

What's missing is a couple of import statements and standard initializing lines found in every OpenOffice macro template, but this blog post is already too long. And it's better that you ask if you need it rather than get bored and not even get to the end.