libera ideoj

David Recordon just let us know about a little strawman proposal he’s calling OpenID Connect. It’s not exactly perfect, but it’s a good jumping off point for the ideas that are shaping the next version of OpenID. And I love that he’s just throwing some ideas out there. It’s really in the original spirit of Request For Comments. There’s even code in there!

Most noticibly, the proposal guts most of the work in OpenID, replacing its discovery and security bits with LRDD and OAuth 2 respectively. LRDD is a rewrite of the discovery process used in OpenID 2 intended to be more modular and simpler. OAuth 2 is also a rewrite, but not of existing OpenID parts. Instead, it takes the OAuth concept and makes it easier for to write clients, among many other changes. But with OAuth 2, you can just use curl to get at APIs instead of having to dig into HTTP headers. Having OAuth 2 underneath OpenID should make it much easier to write clients that work with OpenIDs.

There are some other changes as well. OpenID currently lets you delegate your OpenID from any web address to any OpenID server. This is pretty much only used by early adopters who want vanity OpenIDs. That’s how I’ve got my OpenID set up today. But really it provides almost no benefit with the new disco process. And most people are using an insecure web address to delegate from, making them vulnerable to well known weaknesses in OpenID. The consesus is that all future OpenID versions will have to mandate TLS to keep things safe. If you’re enough of a power user to limit your vanity web address to HTTPS, you’ll have no trouble setting up your own OpenID Provider as well. So losing delegation is a non-issue.

But the OpenID Connect proposal has one change that I can’t talk myself into to supporting. It introduces a very simple User Information API that provides the basic personal information every site needs for registering an account. The problem is that we already have the Simple Registration Extension and Attribute Exchange. Instead of reusing the format of either, it introduces another new one.

Instead of inventing another new format for the next version of OpenID, I’d much rather it have a common way to embed existing formats like hCard, vCard, FOAF and PoCo. Like YADIS so long ago, we’ve got a bunch of competing tools for the same job. So let’s build a way for them to compete instead of one more extremely specialized solution. LID and DIX may no longer be with us, but OpenID incorporated their best ideas. Maybe we could call it Yet Another Personal Information Schema System? Nah.

I figure X/JRD gets us most of the way. Webfinger demos show that we can already do this stuff by just pointing around. But what’s missing is a way to inline the data so clients don’t have to keep making requests to get standard data. I imagine it could be as simple as:

{"user_id":"https://graph.fb.me/24400320",
 "url":"http://fb.me/davidrecordon",
 "link": [
   {"rel":"http://portablecontacts.net/spec/1.0#me",
    "href":"http://poco.fb.me/davidrecordon",
    "entity":
      {"entry":
        {"id": "692",
         "displayName": "David Recordon",
         "name":
           {"familyName": "David",
            "givenName": "Recordon" },
         "emails":
           [{"value": "recordond@gmail.com",
             "type": "home",
             "primary": "true" }],
          "photos":
            [{"value": "http://pics.fb.me/davidrecordon",
              "type": "home",
              "primary": "true" }]}}},
   {"rel":"describedby",
    "type": "application/rdf+n3",
    "href":"http://foaf.fb.me/daveman692",
    "entity": "You get the idea"}]}

Specifically, this would just be an OAuth API that provides X/JRD data to authorized clients. The only extension is to add an entity element to a resource descriptor link that includes the copy of the linked resource.

Two things complicate a solution like this. First, encoding is hard. There are four main kinds of data that people will want to embed, JSON, XML, binary data, and everything else. XML and JSON are special because they could concievably be included inline in XRD and JRD respectively. And really, that’s the ideal because no one likes having to pull out more than one parser just to deal with some data. I’m not entirely sure how to handle this, but formats that have both XML and JSON would be very nice. So Portable Contacts gets extra points here.

Binary and other data still make trouble though. What’s the best way to signal that you’re using Base64? Is it enough to just use the type of the link? Only some implementation experience will really tell. Singpolyma has done some fancy things with data: URIs, so that may be a starting point for binary.

The other issue is that web resources are more than just data. Good HTTP servers provide all sorts of metadata about when the resource was updated, how long to cache, &c. The simplest solution would be to just include the entire HTTP response as a string, but that’s certainly more trouble than it’s worth. CloudKit handles this by tacking on etag and last_modified values. Seems like a sound blueprint to me.

If OpenID adopts this sort of solution, it’s not just good for OpenID users. Everything else that uses XRD would benefit. If you want to have standard data publicly available, you could have that on your public XRD, while more sensitive things would be available to clients you trust. And maybe you could finally start doing more with your web identity than just log in.

This week I’ve had a bunch of problems I haven’t finished solving. So instead of writing a handful of walkthroughs, I thought I’d write about all these issues.

For years now I’ve wanted to set up my own chat server so you can message me at joseph@josephholsten.com instead of through Google, Facebook or any of the other IM services. Recently I also found a great looking framework for accessing your XMPP server from a website called Speeqe, and I’d like to give that a try as well. Since I just got a hand-me-down desktop machine, I thought I’d also try my hand at OpenSolaris. It has a futuristic package management system called IPS, but unfortunately there doesn’t seem to be any XMPP server available through the official Sun package repositories or any of the community managed repos. So while IPS is the the most modern package system for OpenSolaris and I’d like to use it exclusively, that doesn’t seem like an option.

The options I haven’t tried are to build from the official source, use old SVR4 packages, or use a weird source based system called a consolidation. Sun maintains one consolidation called Sun Freeware which contains the trusty ejabberd. But unlike the source based package management systems I’ve used, you don’t get the option of just installing the package you want plus dependencies. So at some point I’m going to have to build a ridiculous number of projects just to get the one I want. I’ll give this vaguely official way a shot first before I try anything else.

I’m also trying to get back up to speed with the Open Stack standards I love playing with. So I put some time into updating my webfinger toolkit to work with what’s live at google right now. I’ve got it working, but I discovered a little nit about how XML namespaces handle unprefixed attributes that surprise. I’m investigating further, but it worries me for the compatibility of XRD parsing implementations. But once I get that clarified, I’ll be starting on a prototype of a XRD provisioning service.

While I was getting that webfinger project working, it occurred to me that I should be testing my code against Ruby 1.9 by now so I can post my projects in Is It Ruby 1.9?. So I gave multiruby a try, then played with rvm, but couldn’t convince 1.9 to install rubygems. Ruby says it’s got issues finding the _rb_Digest_MD5_Finish symbol in digest/md5.bundle, but it may as well be speaking Hungarian for all that means to me. After giving up, I installed the macports ruby19 package and it works flawlessly. I did manage to run nm on the bundle in both the broken and the working installs but the symbol tables look the same. One lead I haven’t followed up on is that rvm is using ruby-1.9.1-p378 while macports uses ruby-1.9.1-p376. At some point I’ll be comparing the build process used by rvm to the macport to see if I can divine anything else.

The EFF is starting up a new patent-busting project aimed at VoloMedia’s podcasting patent. Even if you’re a fan of software patents, it’s obvious bad patents screw up the whole intellectual property system. So I thought I’d check out if VoloMedia’s patent is legit, or if those cypherpunks at the EFF are onto something. Turns out someone beat VoloMedia to the punch by at least half a year.

VoloMedia applied for their patent in November 2003 but wikipedia’s history of podcasting mentions quite a few pieces of podcasting prior art. But it’s not good enough to say podcasting existed before the patent, therefore the patent is invalid. The real task is to figure out which prior art knocks out which of VoloMedia’s claims, and see if there’s anything left standing. So what are the actual claims? In their patent submission they’re laying claim to software that can:

1. Subscribe to and automatically download from a feed of episodes
2. Indicate new episodes are in the feed
3. Sync the episodes to a portable device
4. Sync to the device based on settings
5. Sync to the device manually
6. Share the episodes over a local network
7. Limit automatic download based on feed priority
8. Sync less than all of the feed’s episodes to a portable device
9. Sync when the feed removes episodes

That describes every podcatcher available these days. But we’ve got to find out prior art published before the patent was submitted. So key is finding out what the state of podcasting was before November 19, 2003.

There are a whole bunch of standards docs (and conversations about them) from the people implementing this sort of tool. Searching around for the history of the RSS enclosure tag will find a bunch of people talking about downloading media through aggregators. But you don’t often get a big picture in these conversations, hardly anything focused on how the client actually goes about handling feeds.

But I did stumble accross Mark Pilgrim writing in February 2003 about some features in a brand new feed aggregator that sound a lot like the first claim in VoloMedia’s patent:

Ignoring for the moment all the things it doesn’t do yet (which all sound quite cool), it has one particularly disturbing feature: extracting full HTML content from linked RSS items. The feature is off by default, but once turned on (one checkbox during installation), every time it finds a new RSS item in your feed, it will automatically download the linked HTML page (as specified in the RSS item’s link element), along with all relevant stylesheets, Javascript files, and images.

But wait, there’s more! Deep in the comments, Kevin Burton mentions that one use case for sync is accessing that data on a PDA. Bam! Claim 3. In the same comment, he makes a big deal of the fact that this syncing only occurs based on user preferences, taking out claim 4. He later mentions the standard (and for patent purposes, obvious) way to indicate that a feed has been updated using ETags. Yes, that’s claim 2. And that second comment also mentions sharing NewsMonster downloads on the local network using the same protocol Apple uses in iTunes today, stardardized as ZeroConf. Claim 6, down.

Only syncing less than all the episodes so they fit on the device seems novel, if a bit obvious. The rest of the claims (manual sync, prioritized download, and syncing when a feed removes old content) were common in feed aggregators of the time. But considering NewsMonster’s coming soon features of the time, I’d bet that NewsMonster supported every single feature claimed by VoloMedia’s patent. I’d love to get my hands on an old beta 1 just to check.

For what it’s worth, podcasting is a terrificly successful and useful idea. It took plenty of great ideas to invent it and make it successful. And like it or not, those people who created it deserve the rights to those ideas. If Winer kept his RSS work to himself, it would still be as useful as it is today. But he gave it away and now everyone profits from his ideas.

The thing is, VoloMedia didn’t invent podcasting and they don’t deserve exclusive rights to it. So we’ve got to get rid of this patent, or we’re punishing all the podcasters, fans and real inventors that made podcasting awesome. If you’ve got any more information to help, get ahold of the EFF or let me know.

A coworker of mine wants to stub out our access to external services so our tests work faster and better. Trouble is, he seemed to be pointing at our end-to-end testing when he said he wanted to stub things. And that’s exactly where you should stub as little as possible.

But you ask, aren’t all the cool kids are using mocking frameworks? Yes, they are. And so should you. Especially in your unit tests. You should learn the difference between a stub and a mock. And you should remember that really complicated mock expectations mean you’re probably doing something wrong. And that the the harder it is to mock out side effect behavior, the harder it is to make concurrent and thread safe. But you should totally be using test doubles as much as you possibly can in your unit tests, and using the stupidest ones that could possibly work. If you’ve managed to build an app with good test coverage without test doubles, and didn’t suffer in the process, contact me immediately. I’ve got to see that in action.

But theres a maxim that you shouldn’t mock code you don’t own. Instead, it’s generally a good idea to stub out interfaces. And not just to externally developed services, but to externally developed libraries too. Totally makes unit testing more sane.

That’s because unit testing is about creating a suite of assertions that say “it’s not this part that’s broken”. If (and when) that test fails, you can be sure that something in the test is broken. But if that test does alot, then you haven’t got a clue which of the things the test does didn’t get done right. Which makes it about as useful as a bug report from my sister the graphic designer. Pleasant, better than nothing, but sightly twitch-inducing.

But you don’t want the external stuff doubled out in integration tests. Actually, I hate calling it integration testing. ‘Integration’ is what you do when you need to know the volume of a space. End-to-end testing has less sylables and actually means something. But I personally call it smoke testing as in home inspection. Because you really need to push stuff through every pipe to make sure it isn’t squirting out somewhere.

If you’re doing smoke testing, you shouldn’t be doubling out your services. And you really should be smoke testing. Cucumber makes this stuff dead simple. Especially compared to calling DOM events through the COM interface of Internet Explorer from inside an NUnit test. Cucumber tests are even obviously useful, unlike unit testing. They’re useful because they can check that your app actually does stuff. Just like you would do a million times if you didn’t know how to automate tests.

But back before you knew how to automate tests, didn’t you have the program set up to actually talk to everything it would actually talk to? So you’d know if someone updated the database, or reorganized the file system, or the network was down. At the end of the day, that stuff has to get delt with or the app won’t actually work. Which means it won’t make anyone happy. Which means you won’t get paid. Glum face.

So automating smoke tests means you know if you deserve to get paid. That makes it totally the most important testing you could do.

So why is my friend, who is no fool, trying to screw with the most important testing we have so that it wouldn’t actually tell us if we can get paid?

Because he isn’t, of course. He’s got a hankering for a different kind of test, sometimes called an acceptance test or an integration test or a functional test, depending on who you’re talking to today. I like calling them functional tests, but yeah, that’s at least as bad as ‘integration’. But the point of a functional test is to make sure that your code provides the functionality you expect. If the network is down, that’s not your code’s fault. Where unit tests exist to say, “it’s not this part that’s broken,” functional tests exits to say “it’s not my stuff that’s broken.”

Why would you need three different kinds of tests? If you’ve smoke tests to tell you if you can get paid, and unit tests to tell you if you’ve got bug regressions, what’s the point of functional tests? To winnow decision trees. When you get a bug report, the entire reproduction scenario is the functional test. The fencepost error that causes the scenario to fail is the unit test. As a developer with a good testing suite, functional tests can let you inject whatever you like into your app without having to fire up the debugger.

If your fuctional tests are working, then you know it’s not your problem. Well, it’s almost certainly your problem, but that problem is something more like sitting on the phone all day explaining the bug to your service provider’s tech support instead of twiddling with edge cases in the deepest recesses of your code.

So please, write smoke tests. Write other tests if it suits your fancy, and please write tests for any bugs you find to avoid regression. But don’t cripple your existing smoke tests in the name of TDD. Not unless you don’t care to get paid.

Chris speculated some about the way payment processors like PayPal can play nice in the digital identity arena. I don’t think anyone’s surprised that the big issues in identity today are heath data and payments. I do think that Chris has dropped the ball a bit by taking such a big picture view of the problem. He’s a designer and his real forte is envisioning how tech should look, and then pushing the right people to make it happen.

He needn’t bother too much for this, though, since the VRM Project has already done a great job at imagining how payment should look. With their r-button project, they imagine a common interface for payers and providers to exchange cash and services. They even have a nice icon to show that someone supports the r-button. Like the ever present feed icon, I think it would work nicely in the address bar.

The VRM Project even has a side project of the r-button called PayChoice, that is designing a way for users to pay exactly what they want, how they want. Admittedly, most businesses aren’t keen on selling their stuff for just a penny. But if you haven’t heard, there are Real World Famous People, like Nine Inch Nails and Radiohead, who are already doing business like this. If all it took was a WordPress plugin to install, you can be sure plenty of bands would do it too. But there’s a wider range of CreativeCommons smoking, GPL snorting, free-content loving creators who could use something a bit easier to use than a PayPal donate button.

Unfortunately, VRM Project people generally aren’t cypherpunks, crypto-system designers, or programmers at all. So while we’re got a pretty interface that would be great to use, the wires aren’t connected to anything. There’s no protocol, no implementation, nothing. So the next step is to figure out what what wires we need and what parts we can jack from the rusted Jeep in the backyard.

First of all, there’s got to be security. Just sticking your credit card number in an XML file and posting it on your site is not going to work. I’m not positive, but I’d bet some combination of OAuth and OpenID Attribute Exchange ought to be enough to keep private info safe. OAuth tends to use a segmented system so your payment info would only work with a single provider. And when that cute startup gets acquired by Zombie Hitler’s MultiNationalAcme Inc., you could revoke access to your cash with a click of a button.

Behind that basic security, you’ve got to have some server handling credits and debits. Secure payment systems get created all the time by crypto fans, but there’s a few modern examples that might fit the bill.

The first sort is a pretty close to how things work today. Typified by PayPay, you and your service have accounts in the same bank. You or the service request a transfer, both of you authorize it, and you both get proof that the transfer happened. OpenSocial Virtual Currency is a protocol designed to do just that. It’s strangely tied to the social network where all this would happen, but it seems like that social network could be a marketplace or something else very unlike today’s social networks. I’d be surprised if it weren’t easy to pull this protocol out of it’s social network shell and make it happen anywhere.

This sort of bank-centric has a lot going for it. The protocol is super lightweight, and most of hard security stuff happens behind bank walls. In fact, it’s pretty much the exact same thing as PayPal does now, with a little more automation. Just getting a common protocol to handle the things PayPal and Google Checkout do would make it much easier to support competitors and push innovation into the market. Unfortunately with this kind of protocol, a bank will have a hard time competing with big players. Since both you and the service have to have accounts at the bank, big banks will get more users. And without a protocol that tackles non-repudiation, you have to seriously trust the bank handling the transaction.

More powerful protocols can take care of that. They make sure that every transaction is safe and could let complete strangers exchange things without trusting each other. Protocols like ripple let you use whatever bank you trust and ensure the money gets to the service’s bank safely. In between, the protocol anonymizes the transaction so the service can’t see where you got your money and you can’t see where they spend it. It’s quite a bit like onion routing or using a darknet. Obviously, I’m much more keen on using a protocol that gives me more control over my information and my money.

None of these projects is exactly an Enterprise Turn Key Solution, but that’s okay. Back when OpenID started, there were a large handful of projects that did the same thing. But enter yadis and they could reuse components, fight on an even playing field, and slowly add the features of the others until OpenID eventually had everything in all the others. I didn’t think OpenID would win back then, but won because it got better faster than anything else. We need to get the same thing happening with payment processing. Getting them all to support OpenIDs and publish XRD services would be the first step to snowball into a real solution.