Twitter: do you follow me?

This week's hacking task was to implement a "follow all" function for Twitter.

Even for Twitter users, this needs some explanation: the follow functionality now means "enable notifications". However, the command interface in IM/SMS wasn't changed, so the command name remains "follow". For brevity, I will use the word "follow" instead of "enable notifications".

The reason for having this command is that there used to be a function "follow all" in Twitter. It used to instantly turn on notifications for all your friends (users you're following in new terminology). Now there's a user, called "all" and the function doesn't work (ok, maybe that's not the real reason). This put an end to a very useful feature for users who rely often on the Twitter IM integration.

Having a quick look at the Twitter API it seemed pretty straightforward to fetch all users and enable notifications for all of them one by one. It would be fairly slow, but there was no information in the user list whether notifications are enabled for a user or not. This would have eliminated the need to send requests for users, for whom we already have notifications enabled. Ah well...

The first tool I reach in my toolbox is Ruby. I tried using JSON, but had to give up- I simply couldn't handle Unicode issues:

/usr/lib/ruby/1.8/json.rb:288:in `chr': 1090 out of char range (RangeError)

It turned that it was much smoother with REXML, and it really is a superior library for XML processing (Python's are either easy or full-featured, REXML seems to be both).

I initially took the path of using 'open-uri' for fetching the data over http. After all, it handled even http base authentication and abstracted the nitty-gritty details, and so was easy to use.

But it isn't meant to be used for more fine-grained control, and I soon ran into performance problems, which required special treatment. I found that I quickly exhausted the rate limit of the Twitter API- it's only 70 requests per hour, and with one request per user... you get the picture. The web interface wasn't actually subject to such restrictions, so I wanted to check how it's doing it. A slightly different URL, but worked like a charm, and rate limits seemed to be no problem now!

This time, though, the script ran much longer- 80 seconds compared to about 30 before the change. I analyzed the requests and found out that each received a 302 response, forwarding back to the home page. That meant that open-uri was downloading the whole home page for each user!

At that point open-uri had to go and make way for Net::HTTP. It took more lines to rewrite it, but now I had the choice not to follow redirect responses. I only needed to toggle notifications and didn't care what I got back (as long as it's not an error code). In addition, I could use the same Net::HTTP object, meaning that I use the same HTTP keep-alive connection (not sure if open-uri can do this).

And here's the result- dirty, but still quick. You can configure the action to "follow" or "leave" (to disable all notifications). You need to configure the user and password. Putting the configuration options as command-line arguments is left as an exercise to the reader.

#!/usr/bin/env ruby

require 'uri'
require 'net/http'
require 'rexml/document'
include REXML

user = "lazyuser"
pass = "notmypassword"
action = "follow"
PAGE_USERS = 100

Net::HTTP.start("twitter.com") do |http|
    page = 0
    begin
        page += 1
        req = Net::HTTP::Get.new("/statuses/friends.xml?lite=true&page=#{page}")
        req.basic_auth(user, pass)

        doc = Document.new(http.request(req).body)
        ids = doc.elements.to_a("/users/user/id")
        ids.each do |entry|
            req_follow = Net::HTTP::Get.new("/friends/#{action}/" + entry.text)
            req_follow.basic_auth(user, pass)
            http.request(req_follow)
        end
    end while ids.size == PAGE_USERS
end

XML escape

Friday afternoon is a prime time for blitz-tasks and a rich opportunity for your hacking one-liner 5K1LLz skills.

This Friday the finish-up task came from a colleague, who had to leave in half an hour to catch the plane. There's a big several-megabyte XML file and all characters in it had to be escaped, I presume for preparation to be sent over the wire (read http).

Problem is, a certain Windows editor (does it really qualify that definition?) hangs when opening files bigger than 1234KB, and writing a Java program would take a fairly long time compared to the alternatives. Not to mention that many programmers could write the Java program even less memory-efficiently than the joke of an editor that Notepad is (there, I said it). And Java is not very forgiving on memory problems.

But what are the alternatives? As the Perl manual page says, "The three principal virtues of a programmer are Laziness, Impatience, and Hubris." Being a lazy programmer, I tried to see if somebody else had already written a utility to do this (there was zero chance that there wasn't one), and if it was available. First I found this eclipse plugin. However, pasting megabytes into a text box didn't make me confident that it would work.

There was also the xmlstarlet package, which would have done a wonderful job, had it been installed on the old servers where the file could be easily transferred. But it wasn't, while it would take too long to copy it to my machine and back just to convert the file. It would also be hard to find an appropriate package for that old Linux version. No, that's not for impatient programmers.

The next thought I had was: why spend effort on trying to install a package when with Ruby I could do this in a one-liner. Of course, I have nothing against Python, but if there's one thing nobody would argue is that it doesn't fare well against Ruby when it comes to writing one-liners. Anyway, Ruby wasn't installed there either (note to self: this must be amended).

The clock was ticking. So Python it is, and instead of an obfuscated one-liner I convinced myself to write many short readable lines. I hadn't done serious XML processing in Python for a while, but a google search away the answer came to me. It was really insultingly simple, but given enough hubris one could turn even this meager piece of code into a rambling blog post:

#!/usr/bin/env python

from xml.sax.saxutils import escape
from sys import stdin

for line in stdin:
        print escape(line)