libera ideoj

When I stumbled across Richard Huang’s DRY bundler in capistrano, I got excited thinking I’d learn even more about bundler’s internals and maybe even a few more tricks with using bundler and capistrano together. Unfortunately, all I really got was to use this in my deploy.rb:

require 'bundler/capistrano'

Now that will probably be enough for most people. But if you’re already using bundler 1.0 with capistrano, you probably aren’t most people.

The bundler team has put a lot of work into documenting every little part of the project, so after we’ve required bundler’s capistrato recipe, let’s grab the task explanation:

% cap --explain bundle:install
------------------------------------------------------------
cap bundle:install
------------------------------------------------------------
Install the current Bundler environment. By default, gems will be installed to
the shared/bundle path. Gems in the development and test group will not be
installed. The install command is executed with the --deployment and --quiet
flags. You can override any of these defaults by setting the variables shown
below.

  set :bundle_gemfile,      "Gemfile"
  set :bundle_dir,          fetch(:shared_path)+"/bundle"
  set :bundle_flags,       "--deployment --quiet"
  set :bundle_without,      [:development, :test]

You’re probably going to want to start playing around with bundler 1.0 on your dev machine before you deploy it on production. Since bundler 0.9 doesn’t support all the useful flags in 1.0, so we need to empty a couple of these default settings to make it work:

set :bundle_dir, ""
set :bundle_flags, ""

Now you can have 1.0 on your dev machines and test that this deploys properly.

If you’re like me, you forgot that need a few gems in your development group on your staging machine. So just set the groups you can live without:

set :bundle_without, [:test]

Of course, our production environment doesn’t actually need anything in the development group, and we use separate production and staging cap tasks to load settings for our different environments. So we just modify those to have the right bundler groups:

task :staging do
  set :bundle_without, [:test]
  # other staging specific settings, like
  # set :rails_env, 'staging'
end

task :production do
  # this is the default, but left here for reference
  set :bundle_without, [:development, :test]
  # other production specific settings, like
  # set :rails_env, 'production'
end

Once you’ve got everything tailored and working, you can update bundler on your boxes and restart the app servers. You could leave the bundle_dir and bundle_flags settings the way they are, but I strongly recommend using the --deployment flag to run in deployment mode, the --quiet flag to stop printing so much information, and a bundle directory in your shared_path instead of deployment mode’s default of vendor/bundle.

You’ll also want to start using bundle package to include all the gems you need into vendor/cache. Check that into your in your version control to make the deployment process even better, you won’t even have to talk to rubygems when you run bundle install during a deploy.

If you like more technical documentation, bundler goes into even more depth in its manual pages for bundle-install(1) and bundle-package(1). They explain in depth how deployment mode works differently, how conservative updating works, how gems are cached and quite a bit more. If you find yourself wanting even more control than I’ve explained here, that’s where I would look.

David Recordon just let us know about a little strawman proposal he’s calling OpenID Connect. It’s not exactly perfect, but it’s a good jumping off point for the ideas that are shaping the next version of OpenID. And I love that he’s just throwing some ideas out there. It’s really in the original spirit of Request For Comments. There’s even code in there!

Most noticibly, the proposal guts most of the work in OpenID, replacing its discovery and security bits with LRDD and OAuth 2 respectively. LRDD is a rewrite of the discovery process used in OpenID 2 intended to be more modular and simpler. OAuth 2 is also a rewrite, but not of existing OpenID parts. Instead, it takes the OAuth concept and makes it easier for to write clients, among many other changes. But with OAuth 2, you can just use curl to get at APIs instead of having to dig into HTTP headers. Having OAuth 2 underneath OpenID should make it much easier to write clients that work with OpenIDs.

There are some other changes as well. OpenID currently lets you delegate your OpenID from any web address to any OpenID server. This is pretty much only used by early adopters who want vanity OpenIDs. That’s how I’ve got my OpenID set up today. But really it provides almost no benefit with the new disco process. And most people are using an insecure web address to delegate from, making them vulnerable to well known weaknesses in OpenID. The consesus is that all future OpenID versions will have to mandate TLS to keep things safe. If you’re enough of a power user to limit your vanity web address to HTTPS, you’ll have no trouble setting up your own OpenID Provider as well. So losing delegation is a non-issue.

But the OpenID Connect proposal has one change that I can’t talk myself into to supporting. It introduces a very simple User Information API that provides the basic personal information every site needs for registering an account. The problem is that we already have the Simple Registration Extension and Attribute Exchange. Instead of reusing the format of either, it introduces another new one.

Instead of inventing another new format for the next version of OpenID, I’d much rather it have a common way to embed existing formats like hCard, vCard, FOAF and PoCo. Like YADIS so long ago, we’ve got a bunch of competing tools for the same job. So let’s build a way for them to compete instead of one more extremely specialized solution. LID and DIX may no longer be with us, but OpenID incorporated their best ideas. Maybe we could call it Yet Another Personal Information Schema System? Nah.

I figure X/JRD gets us most of the way. Webfinger demos show that we can already do this stuff by just pointing around. But what’s missing is a way to inline the data so clients don’t have to keep making requests to get standard data. I imagine it could be as simple as:

{"user_id":"https://graph.fb.me/24400320",
 "url":"http://fb.me/davidrecordon",
 "link": [
   {"rel":"http://portablecontacts.net/spec/1.0#me",
    "href":"http://poco.fb.me/davidrecordon",
    "entity":
      {"entry":
        {"id": "692",
         "displayName": "David Recordon",
         "name":
           {"familyName": "David",
            "givenName": "Recordon" },
         "emails":
           [{"value": "recordond@gmail.com",
             "type": "home",
             "primary": "true" }],
          "photos":
            [{"value": "http://pics.fb.me/davidrecordon",
              "type": "home",
              "primary": "true" }]}}},
   {"rel":"describedby",
    "type": "application/rdf+n3",
    "href":"http://foaf.fb.me/daveman692",
    "entity": "You get the idea"}]}

Specifically, this would just be an OAuth API that provides X/JRD data to authorized clients. The only extension is to add an entity element to a resource descriptor link that includes the copy of the linked resource.

Two things complicate a solution like this. First, encoding is hard. There are four main kinds of data that people will want to embed, JSON, XML, binary data, and everything else. XML and JSON are special because they could concievably be included inline in XRD and JRD respectively. And really, that’s the ideal because no one likes having to pull out more than one parser just to deal with some data. I’m not entirely sure how to handle this, but formats that have both XML and JSON would be very nice. So Portable Contacts gets extra points here.

Binary and other data still make trouble though. What’s the best way to signal that you’re using Base64? Is it enough to just use the type of the link? Only some implementation experience will really tell. Singpolyma has done some fancy things with data: URIs, so that may be a starting point for binary.

The other issue is that web resources are more than just data. Good HTTP servers provide all sorts of metadata about when the resource was updated, how long to cache, &c. The simplest solution would be to just include the entire HTTP response as a string, but that’s certainly more trouble than it’s worth. CloudKit handles this by tacking on etag and last_modified values. Seems like a sound blueprint to me.

If OpenID adopts this sort of solution, it’s not just good for OpenID users. Everything else that uses XRD would benefit. If you want to have standard data publicly available, you could have that on your public XRD, while more sensitive things would be available to clients you trust. And maybe you could finally start doing more with your web identity than just log in.

This week I’ve had a bunch of problems I haven’t finished solving. So instead of writing a handful of walkthroughs, I thought I’d write about all these issues.

For years now I’ve wanted to set up my own chat server so you can message me at joseph@josephholsten.com instead of through Google, Facebook or any of the other IM services. Recently I also found a great looking framework for accessing your XMPP server from a website called Speeqe, and I’d like to give that a try as well. Since I just got a hand-me-down desktop machine, I thought I’d also try my hand at OpenSolaris. It has a futuristic package management system called IPS, but unfortunately there doesn’t seem to be any XMPP server available through the official Sun package repositories or any of the community managed repos. So while IPS is the the most modern package system for OpenSolaris and I’d like to use it exclusively, that doesn’t seem like an option.

The options I haven’t tried are to build from the official source, use old SVR4 packages, or use a weird source based system called a consolidation. Sun maintains one consolidation called Sun Freeware which contains the trusty ejabberd. But unlike the source based package management systems I’ve used, you don’t get the option of just installing the package you want plus dependencies. So at some point I’m going to have to build a ridiculous number of projects just to get the one I want. I’ll give this vaguely official way a shot first before I try anything else.

I’m also trying to get back up to speed with the Open Stack standards I love playing with. So I put some time into updating my webfinger toolkit to work with what’s live at google right now. I’ve got it working, but I discovered a little nit about how XML namespaces handle unprefixed attributes that surprise. I’m investigating further, but it worries me for the compatibility of XRD parsing implementations. But once I get that clarified, I’ll be starting on a prototype of a XRD provisioning service.

While I was getting that webfinger project working, it occurred to me that I should be testing my code against Ruby 1.9 by now so I can post my projects in Is It Ruby 1.9?. So I gave multiruby a try, then played with rvm, but couldn’t convince 1.9 to install rubygems. Ruby says it’s got issues finding the _rb_Digest_MD5_Finish symbol in digest/md5.bundle, but it may as well be speaking Hungarian for all that means to me. After giving up, I installed the macports ruby19 package and it works flawlessly. I did manage to run nm on the bundle in both the broken and the working installs but the symbol tables look the same. One lead I haven’t followed up on is that rvm is using ruby-1.9.1-p378 while macports uses ruby-1.9.1-p376. At some point I’ll be comparing the build process used by rvm to the macport to see if I can divine anything else.

The EFF is starting up a new patent-busting project aimed at VoloMedia’s podcasting patent. Even if you’re a fan of software patents, it’s obvious bad patents screw up the whole intellectual property system. So I thought I’d check out if VoloMedia’s patent is legit, or if those cypherpunks at the EFF are onto something. Turns out someone beat VoloMedia to the punch by at least half a year.

VoloMedia applied for their patent in November 2003 but wikipedia’s history of podcasting mentions quite a few pieces of podcasting prior art. But it’s not good enough to say podcasting existed before the patent, therefore the patent is invalid. The real task is to figure out which prior art knocks out which of VoloMedia’s claims, and see if there’s anything left standing. So what are the actual claims? In their patent submission they’re laying claim to software that can:

1. Subscribe to and automatically download from a feed of episodes
2. Indicate new episodes are in the feed
3. Sync the episodes to a portable device
4. Sync to the device based on settings
5. Sync to the device manually
6. Share the episodes over a local network
7. Limit automatic download based on feed priority
8. Sync less than all of the feed’s episodes to a portable device
9. Sync when the feed removes episodes

That describes every podcatcher available these days. But we’ve got to find out prior art published before the patent was submitted. So key is finding out what the state of podcasting was before November 19, 2003.

There are a whole bunch of standards docs (and conversations about them) from the people implementing this sort of tool. Searching around for the history of the RSS enclosure tag will find a bunch of people talking about downloading media through aggregators. But you don’t often get a big picture in these conversations, hardly anything focused on how the client actually goes about handling feeds.

But I did stumble accross Mark Pilgrim writing in February 2003 about some features in a brand new feed aggregator that sound a lot like the first claim in VoloMedia’s patent:

Ignoring for the moment all the things it doesn’t do yet (which all sound quite cool), it has one particularly disturbing feature: extracting full HTML content from linked RSS items. The feature is off by default, but once turned on (one checkbox during installation), every time it finds a new RSS item in your feed, it will automatically download the linked HTML page (as specified in the RSS item’s link element), along with all relevant stylesheets, Javascript files, and images.

But wait, there’s more! Deep in the comments, Kevin Burton mentions that one use case for sync is accessing that data on a PDA. Bam! Claim 3. In the same comment, he makes a big deal of the fact that this syncing only occurs based on user preferences, taking out claim 4. He later mentions the standard (and for patent purposes, obvious) way to indicate that a feed has been updated using ETags. Yes, that’s claim 2. And that second comment also mentions sharing NewsMonster downloads on the local network using the same protocol Apple uses in iTunes today, stardardized as ZeroConf. Claim 6, down.

Only syncing less than all the episodes so they fit on the device seems novel, if a bit obvious. The rest of the claims (manual sync, prioritized download, and syncing when a feed removes old content) were common in feed aggregators of the time. But considering NewsMonster’s coming soon features of the time, I’d bet that NewsMonster supported every single feature claimed by VoloMedia’s patent. I’d love to get my hands on an old beta 1 just to check.

For what it’s worth, podcasting is a terrificly successful and useful idea. It took plenty of great ideas to invent it and make it successful. And like it or not, those people who created it deserve the rights to those ideas. If Winer kept his RSS work to himself, it would still be as useful as it is today. But he gave it away and now everyone profits from his ideas.

The thing is, VoloMedia didn’t invent podcasting and they don’t deserve exclusive rights to it. So we’ve got to get rid of this patent, or we’re punishing all the podcasters, fans and real inventors that made podcasting awesome. If you’ve got any more information to help, get ahold of the EFF or let me know.

A coworker of mine wants to stub out our access to external services so our tests work faster and better. Trouble is, he seemed to be pointing at our end-to-end testing when he said he wanted to stub things. And that’s exactly where you should stub as little as possible.

But you ask, aren’t all the cool kids are using mocking frameworks? Yes, they are. And so should you. Especially in your unit tests. You should learn the difference between a stub and a mock. And you should remember that really complicated mock expectations mean you’re probably doing something wrong. And that the the harder it is to mock out side effect behavior, the harder it is to make concurrent and thread safe. But you should totally be using test doubles as much as you possibly can in your unit tests, and using the stupidest ones that could possibly work. If you’ve managed to build an app with good test coverage without test doubles, and didn’t suffer in the process, contact me immediately. I’ve got to see that in action.

But theres a maxim that you shouldn’t mock code you don’t own. Instead, it’s generally a good idea to stub out interfaces. And not just to externally developed services, but to externally developed libraries too. Totally makes unit testing more sane.

That’s because unit testing is about creating a suite of assertions that say “it’s not this part that’s broken”. If (and when) that test fails, you can be sure that something in the test is broken. But if that test does alot, then you haven’t got a clue which of the things the test does didn’t get done right. Which makes it about as useful as a bug report from my sister the graphic designer. Pleasant, better than nothing, but sightly twitch-inducing.

But you don’t want the external stuff doubled out in integration tests. Actually, I hate calling it integration testing. ‘Integration’ is what you do when you need to know the volume of a space. End-to-end testing has less sylables and actually means something. But I personally call it smoke testing as in home inspection. Because you really need to push stuff through every pipe to make sure it isn’t squirting out somewhere.

If you’re doing smoke testing, you shouldn’t be doubling out your services. And you really should be smoke testing. Cucumber makes this stuff dead simple. Especially compared to calling DOM events through the COM interface of Internet Explorer from inside an NUnit test. Cucumber tests are even obviously useful, unlike unit testing. They’re useful because they can check that your app actually does stuff. Just like you would do a million times if you didn’t know how to automate tests.

But back before you knew how to automate tests, didn’t you have the program set up to actually talk to everything it would actually talk to? So you’d know if someone updated the database, or reorganized the file system, or the network was down. At the end of the day, that stuff has to get delt with or the app won’t actually work. Which means it won’t make anyone happy. Which means you won’t get paid. Glum face.

So automating smoke tests means you know if you deserve to get paid. That makes it totally the most important testing you could do.

So why is my friend, who is no fool, trying to screw with the most important testing we have so that it wouldn’t actually tell us if we can get paid?

Because he isn’t, of course. He’s got a hankering for a different kind of test, sometimes called an acceptance test or an integration test or a functional test, depending on who you’re talking to today. I like calling them functional tests, but yeah, that’s at least as bad as ‘integration’. But the point of a functional test is to make sure that your code provides the functionality you expect. If the network is down, that’s not your code’s fault. Where unit tests exist to say, “it’s not this part that’s broken,” functional tests exits to say “it’s not my stuff that’s broken.”

Why would you need three different kinds of tests? If you’ve smoke tests to tell you if you can get paid, and unit tests to tell you if you’ve got bug regressions, what’s the point of functional tests? To winnow decision trees. When you get a bug report, the entire reproduction scenario is the functional test. The fencepost error that causes the scenario to fail is the unit test. As a developer with a good testing suite, functional tests can let you inject whatever you like into your app without having to fire up the debugger.

If your fuctional tests are working, then you know it’s not your problem. Well, it’s almost certainly your problem, but that problem is something more like sitting on the phone all day explaining the bug to your service provider’s tech support instead of twiddling with edge cases in the deepest recesses of your code.

So please, write smoke tests. Write other tests if it suits your fancy, and please write tests for any bugs you find to avoid regression. But don’t cripple your existing smoke tests in the name of TDD. Not unless you don’t care to get paid.