AppVenture has been updated

All articles have been rewritten and improved. You will be forwarded to the updated article.

Click here to go there directly.

Tue, 2 Feb 2016 #

Hirundo: Comfortably follow Swift Mailing Lists on OSX

If you're interested in Swift, the mailing lists are a fantastic resource. Swift-Evolution discusses the future of the language and includes many interesting messages about the shape that the language may become. Following it also really helps understanding the bigger idea behind many of Swift's most interesting features. Swift-Dev, on the other hand, goes down into the details of how Swift works internally. If that's not your thing, there's Swift-Users which covers questions and discussions about Swift from a user / programmers point of view.

Following those lists is not only useful, but also brings tremendous benefits. However, following them is also kinda hard:

  • The web archive is not easy to navigate
  • When you subscribe to a list you're bound to the UI of a mailing list / your Mail app
  • You can't see any messages posted before you subscribed
  • Many convenience functions (Bookmarks, Muting, etc) are not available

This bugged me. I wanted to be able to follow the discussions there, but I'd rather have a nice UI for them. So over Christmas I started working on a small app to solve this dilemma.

1 Enter Hirundo

Hirundo is the result of this (Written in Swift, of course). The word is latin and means Sparrow or Swift, so I thought it'd be a nice fit and it sounds cool. There's also an accompanying website for the app under stylemac.com/hirundo.

The app is currently in beta as I'm trying to iron out the remaining bugs. I'm already very happy with how it turned out and with the current set of implemented features. Among other things, the app supports:

  • Follow multiple Swift lists
  • Browse and search everything posted to the lists
  • Detailed search facilities over all messages in all lists
  • You can mute threads so that they don't appear in the lists anymore
  • If you are subscribed to a list, Hirundo will open Mail.app with reply headers set
  • Read messages from selected authors, search for authors, or even manage favorite authors
  • Sort messages by various properties, including a Rank showcasing the most discussed threads
  • Messages are threaded, you can fold them or collapse them
  • Bookmark any message or thread into multiple folders and revisit them later
  • Reply to threads or start new threads

Download it here and give it a try.

2 Building it

Writing the app took longer than I initially expected. I sometimes tend to be naive about technical limitations and so I just started working on this app without really assessing the difficulty of implementing it. The biggest issue was that the Swift mailing list data is not available via a REST API or something which else that one could query. There're only two ways of accessing the data:

2.1 Parsing the HTML archives

However this has several drawbacks. It requires a ton of HTTP requests, and it is very slow crawling the server for them. Those HTML pages use very simple, non-semantic HTML which makes it difficult and error prone to parse them correctly. Even worse, it lacks certain data which is required to properly establish the message tree afterwards.

2.2 Parsing the mailing archives

This is what I used. The archives are in the Unix mbox format so there's a way better chance of parsing them properly. They also contain all the information required to build up the message threads afterwards. I decided to use MailCore 2 for the actual parsing of the mails in the mbox so that I'd not need to reimplement another RFC2822 parser. However, a couple of small differences between RFC2822 and the mbox format generated by pipermail1 required additional effort in order to properly format the mbox contents so that MailCore could read them. The messages are then read, parsed, converted, and finally placed in Core Data.

3 Real Time Messages

After playing with the app for a while, I realized that the mailing list archives were not updated in real time. Instead, the compressed archives are only generated once a day. This makes the whole approach obsolete for actual following the discussions. This was also when I realized I should have invested more time up front researching this. Since the HTML was not feasible for import I was at an impasse. The solution, it seemed, was to host the temporary real time messages myself.

The next step (again, this took way longer than expected) was setting up an smtp email server which receives mails to mailing lists that I subscribe it to, and writes them out into the mbox format. A small webserver then offers those files as temporary caches of current mailing list activity. As soon as the official mailing list archives are updated, the cached ones are deleted2. If you're running Little Snitch this is what happens when you see Hirundo download data from hirundo-lists.stylemac.com.

4 Replying to Messages or creating Threads

When you click reply or new Hirundo opens your standard mail client with a mailto: call embedding all required mail headers for a proper reply. However Mail.app does not respect one of the required headers (In-Reply-To) so I had to use the (References) alternative header instead. That is a working solution but not optimal as it makes it more difficult for Pipermail (the mailing list) to properly thread your reply to the correct location. Other mail clients (such as Thunderbird) don't seem to have this problem. A future solution would be to send the mail directly from within Hirundo.

5 Building the UI

Once that finally worked, I started working on the UI of the app. I decided to limit it to 10.11 El Capitan, so that I could play around with all the new and exciting technologies released last WWDC. As it happens, I haven't even reached 1.0 and there's already a list of upcoming refactorings I'd like to perform as some initial design decisions were not as good as I thought they'd be in hindsight.

6 Mac App Development

I wrote a couple of interesting small classes for this project, and I'll spend the next posts on appventure building a simple Mac App based on some of the Hirundo code. There's currently a certain lack of Swift OSX development information, and I'd like to fill this void a bit.

Footnotes:

1
The mailing software used by the Swift Mailing Lists
2
There's a certain chance that, given the right timing, a message may not end up in Hirundo until one day later

If you read this far, you should follow me (@terhechte)
on Twitter