OS X did rather well just because of the nature of their hermetically sealed ecosystem.  Sure, there were casualties (the entire MacFUSE ecosystem comes to mind), but overall it wasn’t bad.  Linux has had its rough spots for close to eight years now.  More on that in a second.  Finally, Windows still struggles with driver compatibility and weird glitches in 64-bit mode.  It’s amazing it has been this difficult to make the switch, but it does give a good perspective on how the inertia of an open community can act as an anchor at times.  The entire Apple ecosystem is a dictatorship, and therefore is very efficient at making big changes.

The problem both Linux and Windows (and yes, even OS X) have had is that it is very difficult to force entities that won’t give up their source code to make a change.  The best example of this is Adobe and their ubiquitous Flash Player.  It’s the most polarizing technology on the internet right now, with Apple shunning it almost entirely, and Google embedding it directly into the browser (“keep your enemies closer…”?).  In many ways it both enables and hobbles the modern web; it has been a transformative force while violating all the original ideals the web was based on.

So, here I am, once again making the leap to 64-bit thinking “C’mon man, it’s 2011, they must have worked out all the glitches by now…”.  I am typing this on my shiny new Lenovo X201 running Ubuntu 11.04 Beta 1 in 64-bit mode, only to find out that Flash still doesn’t work.  At all.  So, I manually download Adobe Flash “Square” and push the binary-blob to the right directory so all my browsers will find it.  And it works…sort of.  The new Amazon Cloud Player and my Subsonic setup keep producing this weird metallic buzz in all the music, and it makes my services useless.

So, I spent about an hour searching for what the problem could be.  After a whole lot of testing and searching, I determined it must be the 64-bit Linux incarnation of Adobe Flash.  It turns out that it is, and no one can force them to fix their broken use of memcpy with something else because their project is still closed source.  If it were an open-source project, the changes would be made in under a day and a new version would be pushed out shortly thereafter. As it is, we have to wait for the dictator to do the right thing.

So, I suppose that means that dictatorships can either be the most agile or the most stagnant, and open communities are somewhere in the middle.  In any case, I think it is more fun to be part of a community than a subject to a dictator.

A Working Definition of “Net Neutrality”

It is important to outline exactly where we stand on net neutrality.  First, a working definition: net neutrality is intended to ensure that internet service providers do not prioritize (or throttle) traffic based on the owner of the traffic or the specific application of that traffic.  In general, this is intended to allow actions like optimizing VoIP traffic to minimize latency (so your internet calls don’t have a high delay because someone next door is torrenting the latest Ubuntu ISO).  On the other hand, it is intended to prevent the prioritization of one VoIP service (or client) over another: Vonage, TMobile @ Home and Skype should all receive the same priority because they are all VoIP applications. 

As a good friend of mine said when we discussed it: if you own a website, net neutrality dictates that you shouldn’t be able to go to your customers’ ISPs and buy better access to them.

Do We Even Have Net Neutrality?

Many people believe neutrality is a good idea, including Google.  But there seems to be a misunderstanding on where we currently stand with respect to net neutrality.  Much of the coverage seems to indicate that we somehow already have network neutrality.  We don’t.  We never have.  It is well known that various ISPs reduce bandwidth to particular sites like YouTube because of disproportionate traffic consumption coming from those sites.  Bittorrent has been a whipping boy as well; Comcast actually won in court after throttling consumers’ bandwidth if they used Bittorrent.  So, this concept that seems to floating around that we all live in a net neutral world is false.  NPR’s coverage of the “deal”, as they insist on calling it, is but one example of this misunderstanding:

In a move that could change the landscape of the Internet, Google and Verizon are reportedly close to a deal that would allow Internet service providers to prioritize traffic from certain websites.

Not only does the article call it a “deal” (it is not at all clear that there is any deal) but the phrase “that would allow Internet service providers to prioritize traffic from certain websites” makes it sound like they currently can’t.  But NPR is not the only reputable news source to get it wrong.  In the article from the New York Times covering Comcast’s win in court, they said:

A federal appeals court ruled on Tuesday that regulators had limited power over Web traffic under current law. The decision will allow Internet service companies to block or slow specific sites and charge video sites like YouTube to deliver their content faster to users.

Note the phrasing: “The decision will allow Internet service companies to block or slow specific sites”.  Embedded in that phrasing is the notion that prior to this decision, companies could not do this.  But, there wouldn’t have ever been a court case if that were true.  The reason Comcast was brought to court was because they engaged in this activity.  The court essentially clarified the current situation on net neutrality: there isn’t any.

I don’t know how to make this any clearer, so I’ll just say it: we do not currently live in a world in which net neutrality is a reality.  Net neutrality is an idea that a lot of people agree with (like world peace), but it isn’t real, either in the wired networks that run to our homes and businesses or the wireless networks that we use on our phones.  The only news coverage I’ve been able to find that actually acknowledges this reality is the article on TorrentFreak discussing this exact issue:

Seriously, we don’t understand where all the hatred towards Google comes from. The proposal is not going to destroy Net Neutrality, simply because Net Neutrality doesn’t exist yet.

My sentiments exactly.

But, that doesn’t stop the vitriol from spewing forth from publications like eWeek, The Huffington Post, The Last Watchdog, and even the very respectable Ars Technica, who dubbed the proposal a “flip-flop”.

A Step Towards a Common Understanding

The ISPs’ Position

So, it is in this context that we come to the Google-Verizon proposal.  After a decade of stagnation on the net neutrality front (or even a regression, if you count Comcast’s win) both ISPs and websites want some certainty in a highly uncertain landscape.  On one hand, many internet service providers also engage in other businesses, and may wish to provide “private” services alongside the broadband internet service.  This usually includes legacy information dissemination mechanisms, most notably television service, but also other services like VoIP.

Let’s take an example using VoIP, which would be a bundled service in a package like AT&T’s UVerse.  Back in the old days when VoIP was newer, I remember having trouble with Vonage; if I were downloading a large file (let’s say I was executing a large apt-get install command and my Linux computer) and my friend called, I would answer the phone and they would immediately complain: “Hey, Rick, I can’t hear you.  What’s going on?”  This was a real problem, and while it’s gotten better (as bandwidth has increased), there is an idea that providers should be able to identify latency-sensitive types of traffic and prioritize them accordingly.  Hopefully that is something we can agree on, because it has nothing to do with net neutrality, because it isn’t about service providers buying better access to customers.

Here’s where it gets interesting.  If you do decide to buy AT&T UVerse, should AT&T be able to bundle in phone service and make guarantees that the phone service won’t have choppy audio, even when you saturate your entire bandwidth with a torrent you’re downloading?  What about TV?  Should AT&T be able to guarantee that your TV channels still come in clearly and without horrendous compression artififacts even if someone in your home is watch Netflix for Hulu or downloading Bioshock off of Steam?  As I read the Google-Verizon proposal, this is what they mean when they talk about service providers being able to provide “differentiated services”.  Everyone should be asking “Differentiated from what, exactly?”  I think the answer is differentiated from “normal” or “public” internet.  The internet you’re using to read this post.

But let’s explore what happens if we don’t let providers do this.  Consider the same example.  Currently, cable providers already carve out a chunk of the data coming over the wire for channel information, but no one really cares, because it operates independently of the piece carved out for the “public” internet.  The provider is sending prioritized data, but since people don’t think of it as being on their internet connection, they don’t seem to mind.  Now consider the case in which cable providers wanted to switch to an IP-based distribution mechanism, opening up the rest of that coax cable for internet use, but prioritizing their own video streams so that if someone was watching TV, they would be guaranteed a good picture.  Even though this scenario would be a win for all players, it would violate net neutrality in its purest form.  Enforcing pure net neutrality in that case would hinder progress.  So, Google and Verizon introduced the concept of these “differentiated services” that I imagine would include things like VoIP and TV service.

But that’s not the whole story.

The Web Sites’ Position

On the other side of the debate is the web sites that are engaging in significant research and development to introduce new, competitive offerings that challenge traditional business models.  Netflix, Youtube and Hulu come to mind.  These websites have to deliver their services over a wire that is owned by the very companies they are competing with.  In the absence of network neutrality, that poses a very risky proposition.  Since businesses like these have enormous up-front investments, there is a big risk that as soon as they make those investments and become relevant, their competitors like Verizon, Comcast and other digital media providers will drop the hammer on them.  So, in the interest of maintaining innovation on the web, it is vital that these companies have some kind of assurance that their competitors won’t unfairly deprive them of their right to compete on the basis of their products’ and services’ merit.  That assurance is network neutrality.  The need for some kind of regulation in this regard is quite clear.  But why?

The one remaining piece of the puzzle lies in the fact that like so many utilities (water, power, telephone), wired internet providers have a virtual monopoly.  The right way to handle all this would normally be through the free market; if an ISP ruined your experience watching Hulu so they could push you their own video streams, you would simply switch to another that didn’t do that.  But that’s not possible, since providers have a stranglehold on the geographic location in which you live.  Unless you’re willing to pick up and move so you can get better internet access, the lack of competition is a real problem.  That’s the motivation for government regulation.

Wrapup

So where does that leave us?

Well, given that we don’t have net neutrality in either wired or wireless forms yet, and given that we all agree that it would be a good thing to have, I have to say that while I’m a bit disappointed by portions of the current proposals (both those coming from the FCC and those drafted by Google-Verizon), after years of making no progress, it is good to finally get some movement on this.  Getting formal, public agreement that public internet should operate in a neutral way is a big step: a step we haven’t seen before.

I think it is clear that we will need wireless net neutrality as well, although I agree that we’re still figuring out exactly what wireless networks will look like.  The market is still changing very fast in the wireless space, with very distruptive changes (like flat rate, capped or metered data plans) still being figured out.  As phones become more like mini-laptops and laptops replace laptops, the importance of wireless is undisputed.  How exactly to manage the complexity of a such a dynamic market is another question which Google and Verizon have rightfully (in my humble opinion) decided to postpone resolution of.

The majority of this post was written before all the Facebook privacy concerns of May 2010 occurred, but those events make the whole issue much more relevant.I commented on a story about a user who loved AIM shortly after Buzz was launched, and, as I have in the past, I got reactions of confusion from some of my followers.

Computers and Connections Are Unreliable

Solution: Federated Services

The Trend Away From Federation

What We’ve Lost, And How We Can Get It Back

The Three Critical Questions

via Rick’s Posterous

The Extensibility of Firefox

Mozilla recently hosted a Summer Design Challenge for 2009 that focused on “Reinventing Tabs in the Browser”.  Firefox is a fabulous platform to host such a challenge on, as it is the most extensible of all the browsers currently available.  Let me digress into why I say this quickly before I get into my main point (which is about tabs!)

Mozilla itself has gone to great length to expose the browser internals by making the browser open source, which allowed for modification. Next, they provided a framework for developers to write add-ons for Firefox, which provided a robust architecture for extension. The primary distinction between modification and extension is that two modifications to a piece of software may well be mutually exclusive, while two extensions are designed to work side-by-side.  These two aspects of Firefox’s feature set have been well known since its release, and the add-on (extension) community has been very active since Firefox’s release.

But, in recent months, Mozilla has released two extensions that make the process of extension itself even easier.  One is called Ubiquity, which provides a command-based interface to the browser and web, and allows for it to be extended by writing Javascript in its configuration interface (in a browser window with the address “about:ubiquity”).  This type of extension caught on quickly with fans of command-based systems like Launchy and QuickSilver, (and later, Google’s Quick Search Box, authored by the developer responsible for Quicksilver, also only available for Mac) but was otherwise relegated to a niche of the Firefox user base.  One of its primary features was the novel mechanism by which Ubiquity could be modified: simply visiting its configuration interface (inside the browser itself), and typing into a text box.  That was it.  No saving, reloading or compiling.  As soon as the code was typed, it was “live”, and immediately available within the Ubiquity launcher.  The distinction between “development” and “use” was almost non-existent.

This feature was so compelling that it deserved its own extension, independent of the concept of a command-based launcher embedded inside Firefox.  Whereas Ubiquity provided an API centered around the user interface of Ubiquity, it offered little to modify aspects of Firefox outside of the Ubiquity interface.  To fill this more general need, JetPack was born, also from Mozilla Labs.  JetPack had all the quick-development goodness of Ubiquity, but without the extra baggage of a command-based interface.  JetPack was designed to allow extensions for Firefox to be written quickly and simply, and distributed on the web with equal ease.  Though still young (JetPack has only been released for a couple of months), it provides a polished interface for development.  Its internal browser-based editor for coding is based on Mozilla’s own Bespin, and provides all the immediacy of Ubiquity development with extra goodies like line numbering and syntax highlighting thrown in.

So, hopefully at this point you understand why I assert that Firefox is the world’s most customizable browser.  My concluding thought on this topic is that this is all part of Mozilla’s effort to work towards a browser that is designed to be restarted only rarely, is extensible while it runs using a built-in interface, and allows users to change their working environment as easily as developers, because no special tools are required.  This vision is not entirely new; if you are familiar with Lisp-based environments, the REPL, or GNU Emacs, all this will sound very familiar.  So, it would seem Mozilla is learning well from successful projects in the past.

Why Tabs?  Why Not “Buffers”?

Which brings me to my main point (sorry it took so long!)  As Mozilla hosted the design challenge to reinvent tabs in the browser, many teams with a wide variety of approaches sought to provide meaningful improvements to the existing interface.  Every team did research with users by watching users use Firefox, and/or interviewing users about their browsing techniques.  Every team noticed one glaring deficiency in the existing tab interface: it doesn’t scale.  Any user of Firefox that has tried to manage 50 tabs has had trouble.  Even the interface has trouble: it tries to show many of them, and finally gives up, reverting to a scrolling tab bar.  So, when asked to improve the existing interface, every team tried to improve the tab interface’s scalability.

What I found so surprising as I reviewed the entries was that, in a community that has learned so much from Lisp and Emacs, most entries completely ignored the notion that a keyboard-based navigation system scales much better than a mouse-based system.  The trade-off, of course, is learning curve and prior knowledge.  We can attribute the meteoric rise in keyboard launchers across all operating systems to “Start Menu Clutter” – the simple problem that, when you install a few hundred programs, each wanting its own start menu item, quick launcher, and desktop shortcut, you rapidly drown in a mosaic of brightly colored icons, all alike.  But, if you know what you want (“Firefox”, “Word”, “Emacs”, or “iTunes”), you can use a keyboard launcher, type a few letters, and avoid the clutter entirely.  Such systems prohibit “browsing” of available programs, but for most users most of the time, the desired program is known in advance, rather than discovered by searching the start menu.

One entry provided effective keyboard navigation (it was called “Collapsible Tab Groups“) and it provided good scalability.  In particular, it provided two pieces of functionality that should be present in more software today, one of which comes straight from Emacs.

First, (this is the non-Emacs feature), it puts the UI elements on the side of the screen, rather that the top or bottom.  Why is this so good?  Most screens are widescreen.  Heck, even the 1600×1200 screens are wider than they are tall, but even those are going out of style to favor wider aspects (I’m typing this on a 1680×1050 display).  So, given the fact that displays are wide, but most material we read is tall (think word documents, most PDFs, and almost every web site), do developers insist on sticking every interface bar on the top or bottom of the screen, further constraining the already-limited vertical dimension?  Well, collapsible tabs groups gets it right, and puts the tabs on the side.

Second (this is the Emacs feature), it provides “search as you type” for navigation.  What does this mean?  It’s the kind of behavior you get in desktop keyboard launchers.  If you type “irefox” or even “ire” into Launchy or QuickSilver, you will get “Firefox” as the launch option because that substring matches the “ire” in “firefox”.  This is the feature Emacs uses to manage buffers with ido-mode.  You open the interface to change buffers (files, roughly) in Emacs and it presents you with a small interface that lists just a couple of the (possibly tens or hundreds of) buffers you can switch to.  As you start typing, each character is added to the search criteria as you type, and the list is refined to reflect the relevant buffers.

This approach works perfectly in a browser.  The problem with Mozilla’s challenge is that they asked people to reinvent tabs; the realization is that we don’t need tabs.  “Tabs” implies a user interface, which should be customizable.  What we really want is a model for navigation, and a set of pluggable user interface elements that attach to that model.  One user interface might be tabs (for users that keep 4 or 5 websites open at a time), but another could be a search-as-you-type mode that doesn’t use any screen real-estate when it is not in use.  There has been a widespread conflation of view (the tab interface) and model (the idea that we have multiple web sites open) in every browser.

The obvious answer is that I should get in gear and write a keyboard-based tab switching interface using JetPack (Ubiquity has one, but it is painfully slow to use for a variety of reasons).  I don’t know…maybe I just convinced myself that I should.

When I program, I sometimes am fortunate enough to see beauty emerge from what I do. I love programming because of this, but, enthusiastic as I am about it, I find that it is very hard to convey the essence of what I see to others, which often means that they cannot understand what I am so enthusiastic about!

So, I would like to discuss one of my favorite idioms in programming, but before I do so, I would like to discuss something you might be familiar with: organizing information. In the context of information organization, I think my example makes sense, and the beauty arises, and I don’t have to discuss programming at all. This is ideal, because the beauty in programming has nothing to do with programming at all: it has to do with patterns of thought. But I’ve already delayed too long; off we go!

Organizing Information: Hierarchies

In our struggle to store and retrieve information, we have developed systems to help us organize it all. Probably the most prevalent idea in these systems is the notion of a hierarchy. We see this everywhere, from filing cabinets, to file systems on a computer, to organization of library books, to notes taken in an outline form, to the system for classifying life on Earth, and even in this very essay. The idea is simple: establish a set of top-level categories. Each of these can contain either more categories, or items. Thus, any given item has a location in the hierarchy that is defined by all of its parent categories.

This idea of classifying information in a hierarchy is so pervasive that it comes completely naturally to our way of thinking. It has three main strengths:

1. It allows us to locate information using a broad idea, and narrowing that idea until we finally locate the information we seek
2. It allows us to easily add new information in the structure, because we have a pretty good idea of where things should go
3. It allows us to reorganize the ideas, because we can take sub-categories and change their parents as necessary

But it has one major weakness: if a piece of information has multiple facets, it can really only exist in one place in the hierarchy, so users of the system are forced to choose. There is a rich history of developing add-on aspects to hierarchical systems to fix this weakness. In computer file systems used by Linux, Solaris, BSD or Apple’s OS X, for example, a user can establish links in the file hierarchy to allow a file to exist in multiple locations at once. This would be a bit like putting a yellow sticky note in a filing cabinet in the “Hobbies” section that says “Go look at the Automobiles section for info on my hobby sports car.” This works fairly well, but an unsuspecting user might remove the file on the sports car after it got totaled, only to find that little sticky later. Meanwhile, if your brother looked under “Hobbies” to find info on the sports car, he’d be confused when he couldn’t find it where the note indicated. So, it makes maintenance of the system a bit harder, and can confuse users.

Organizing Information: Tags

Nevertheless, hierarchical systems have such great advantages that they are still the most prevalent approach to solving the problem of information organization. Recently, however, another notion has come along: tagging. Tagging is a simple system: each item of data can be assigned a tag, or keyword, that describes some broader category to which it belongs. Each item can have an unlimited number of tags. Going back to our example, the information on the sports car could be tagged “Automobile” and “Hobby”.

This mechanism for organizing information is used in several well-known environments: Google uses it in their bookmarks implementation, Facebook uses it to organize photos and notes, Flickr uses it for photo organization, and many news sites (like Digg and Slashdot) tag news stories. The advantages are obvious: tags allow a fragment of information to exist in multiple places in the hierarchy. But wait, where is the hierarchy now?

The first question many newcomers to tagging ask is how to really organize things. As soon as you try to establish a tag hierarchy, you start to notice the problem. Suppose you had a tag “Cars” and then another for “Sports Cars”. If you wanted to add some information about the discontinuation of the Dodge Viper into your system, what tags would you add? Both “Cars” and “Sports Cars”? Or maybe just “Sports Cars”, since that implies “Cars”? But then, when someone looked for all information on “Cars”, the info on the Viper wouldn’t show up, which doesn’t seem right. Then again, is this system going to require that we add every possible tag when we add information? That seems cumbersome.

So, in exchange for solving a problem, tagging introduces another. We can now freely associate bits of information with tags, but the tags have no hierarchy. When we look back at the strengths of a hierarchical system, we see that we can still locate information using a broad idea. If we look at all information tagged “Cars”, we can then see that many of those items are also tagged “Sports Cars”, so we develop a sort of emergent hierarchy. This concept is often referred to as a “tag cloud“. So, at least tagging systems allow to locate information in much the same way as we could using a hierarchy, and they have the added bonus of allowing us to file data in multiple places.

We’ve said that hierarchical systems are good at adding data. Unfortunately, adding information to a tagged system can be difficult. For each item we want to add to the system, we have to identify all relevant tags when we insert the information. This means not only knowing all the relevant tags in the system, but also anticipating whether any new tags are needed. This could be even more work that creating the data we wanted to insert in the first place!

Finally, we’ve said that hierarchies are good because they allow us to reorganize information. This can also be difficult when we use tags. In addition to collecting cars, we also start collecting motorcycles. So, we’d like to introduce the notion of “Vehicles” to our system, and have it apply to all of our cars, along with the new motorcycles we’ll be adding. In a hierarchy, you’d simply create the “Vehicles” category and put “Cars” into it. With tags, we have to go back and add the tag “Vehicles” to all items tagged with “Cars”. Laborious, to say the least!

Are tags really that bad, or have we missed something? Do we need to build some tools to automate all the cumbersome aspects of using tags? Or is there some underlying truth here that we have failed to recognize?

“Simplicity does not precede complexity, but follows it.” -Alan Perlis

In many tag systems used on the internet, the problems I’ve outlined above remain. The main reason is that it is difficult to formulate elegant systems to address these problems. The primary job of a computer programmer (whether they know it or not!), is to take problems like the one I described above and try to find the underlying model that should be used. You might get an itch in the back of your mind that something is wrong with what I described: tags seems like they should work, but for some reason are difficult to use.

Your first instinct may be to write a program that will automate the more cumbersome aspects of using tags. For example, in the case of adding a “Vehicles” tag, you could just write a program that would grab all the items tagged “Cars” and add “Vehicles” to them. Seems simple enough, and saves work. But it isn’t beautiful. Over time, you’d write a lot of tools to help you perform tasks like that, and the system would become complex. We wanted a simple system when we started using tags, not something that would be hard to understand and use. The key question that is always on my mind is: How can we retain simplicity?

In the systems we’ve been discussing, we have defined two elements. In the hierarchical system, we defined categories and items. In the tagged system, we defined items and tags. What if we tried to define the system using only one element? What if, in the tagged system, we defined tags as items, and items as tags? How would this change its utility?

Such a change would mean that items could be tagged (as before), but also that tags could be tagged, since they are also items. Like many subtle changes, it is hard to see how this small modification would be useful. In fact, it allows us to enjoy the best aspects of both a hierarchical and a tagged approach to organization, and it means the entire system is only composed of one element: a data item. What we’ve done is change how these items relate to one another, and that enables new patterns of thought and organization.

Going back to the example of the sports car that is both a “Car” and a “Hobby”, we can see that this system would support both through the normal use of tagging. But what about our example of “Cars” and “Sports Cars”? Well, in our new system, as soon as we create the data on the Dodge Viper being discontinued, we can simply tag it with “Sports Cars”. Why don’t we have to worry about tagging it “Cars” as well? Well, as soon as we create the tag “Sports Cars”, we tag it with “Cars”. So, now we have embedded the knowledge that “Sports Cars” and “Cars” are related in the tags themselves, rather than in each item using those tags.

Using this idiom, it is easy to see that hierarchies naturally emerge from the structure, but can be more complex than simple trees, due to the nature of tagging. Now, tags can relate themselves to multiple other tags at different points in the tag hierarchy, as can items, because items are tags. Likewise, if you want to provide some notes on a tag, you simply add data to it, because the tag is a data item. Reorganization of the hierarchy is easy – if you want to introduce the notion of “Vehicles”, simply tag “Cars” and “Motorcycles” with “Vehicles”, and you’re done. All items with those tags (or tags tagged with those tags, etc.) will now be tied to the notion of “Vehicles”.

This idea, though simple, may be confusing at first.  Here’s another way to think about it: instead of tagging items with tags, we tag items with other items.  So, if we’re in a filing system, we tag files with other files.  If we organize photos, we tag photos with other photos.  In reality, we have reduced our system of organization to its very simplest form by allowing ourselves only one action: we can establish connections among various members of a homogeneous collection of data.  Now, each member of that collection will ultimately have both data, and connections to other pieces of data.

If this sounds familiar, that’s because it is: it is the exact methodology for organizing the internet.  Web pages sometimes contain data, and they sometimes contain references to other webpages.  Most webpages have both, even in walled gardens like Wikipedia.

Conclusion

By reducing the number of elements in our system, we increased its power, but retained its simplicity.  Our new system has all the benefits of the hierarchical system, as well as supporting all the benefits of tagging.

While I decided to write this and relate the ideas to programming, I got the idea of “data as tags” and “tags as data” from TiddlyWiki, which I used to compose this, as well. If you want to see the system I described in action, visit the TiddlyWiki homepage and download a copy and play with the tagging system: it is an elegant and powerful approach to managing lots of information.

In the past few months, however, I have rediscovered TiddlyWiki, which, despite its name, is a very serious piece of software.  Though written only in Javascript and HTML (to run in a browser), it is a serious software engineering effort, and has a large community doing active development of many different variants.  The purpose of this post is to provide a look at what TiddlyWiki has to offer, so as to assist potential new users (you!) in determining if it might be useful to organize information in their lives.

TiddlyWiki is a Wiki

This means in is accessed in a web browser that you already have on your computer.  It works best in Firefox, but can be used in Safari, Internet Explorer, and Opera as well.  TiddlyWiki works an whatever operating sytem you have.

This means it has “pages”, as seen in other wikis (like Wikipedia), except instead of entire webpages, TiddlyWiki uses Tiddlers: small (or not-so-small) chunks of formatted text about a particular topic.

This also means it uses hypertext, so you get both internal (links between tiddlers) and external (links to other websites) linking, as well as formatted text: bold, underline, italic, monospace fonts, numbered and bulleted lists.  The linking feature is critical, because it allows you to link tiddlers to each other in the same way the internet links webpages.  You can use this mechanism to organize your data as you see fit – whether that be in a hierarchy (like the files on your computer) or in some web that more closely resembles Wikipedia.  You are in complete control.

TiddlyWiki is Just a File

This means it resides on your hard drive or USB stick just like any other file.  You can copy it, give it to others, post it online, or carry it around with you just like any other file.  To open it, you just double cilck on it and it will open in your browser.

TiddlyWiki Supports Metadata

All I mean by this is that it allows you to tag tiddlers (pages) with their topic.  If you’re a programmer, you might tag tiddlers with the language they relate to.  If you’re a journalist, you can tag a tiddler with the article it relates to, or the source it came from.  Tiddlers can have as many tags as you like.

TiddlyWiki Can Search

As you grow the content in your TiddlyWiki (by adding new Tiddlers), you may start to worry that while it is really easy to add information to the TiddlyWiki, retrieving it may be difficult.  Where is that darn tiddler that contained the dentist’s address and office hours?  Even if you can’t remember what you called it (was it under “Dentist” or “Medical”?) so long as you used the word “dentist” anywhere in the tiddler, you can simply type “dentist” into the global search box, and, as you type, TiddlyWiki will search every word you have stored in it and show you all the tiddlers that contain that keyword.  If all you can remember is that you had the words “office hours” in the tiddler, just type that and you’ll track your data down in no time.

You can try this on the TiddlyWiki homepage (which is itself a TiddlyWiki!).  Upon visiting the page, you have no idea if there is any information on whether TiddlyWiki works on the browser available for the Nintendo Wii.  Simply type in the keyword “wii” in the search box and you can find out if there is any information on the Wii anywhere in the TiddlyWiki.

TiddlyWiki is Organized

In addition to full text search of all your data, TiddlyWiki can also list all the tags you’ve ever created and allow you to view lists of all tiddlers that match those tags.  You can then open the tiddlers you want from that list, or open all tiddlers that match a given tag.  At home, I work on schoolwork for my current class (csc576 – Connection Oriented Neworks), so to bring up my notes for that class, I just go to the tag I made called “csc576″ and request all tiddlers matching that tag be opened.  Similarly, at work, if I’m working on a project relating to JBoss, I can open all my notes on JBoss by selecting the tag I created called “jboss”.

In addition to tagging, TiddlyWiki offers a series of tiddler lists that are organized in various useful ways.  In addition to the tag list view (mentioned above), TiddlyWiki also offers a timeline of tiddler edits, allowing you to see what has been changing in your TiddlyWiki at a glance.  Can’t remember the name of the tiddler you were editting on Monday?  Just look at the timeline and it will tell you what changed that day.  In addition to the timeline, you may also find that you create tiddlers that are not referenced from anywhere else.  They may contain useful information (the VIN number on your old 1991 Toyota Corolla), but you won’t find them just by “surfing” through the TiddlyWiki.  Well, luckily, TiddlyWiki keeps track of all these “orphaned” tiddlers and can give you a list, so that you won’t lose them.  Conversely, sometimes a tiddler has a link to it, but it doesn’t yet exist.  TiddlyWiki also tracks these cases, and provides a list if needed.

Conclusion

Overall, I have found that TiddlyWiki has provided a sort of “sweet spot” in the field of tools to organize your data.  It is extremely easy to pick up and start working with, but has an extensive feature set that gives advanced users  a lot to work with.  It is open source and extensible, allowing sties that collect extensions for TiddlyWiki to spring up all over the web.  If you’d like to keep your own TiddlyWiki on your local machine or USB stick to carry with you, you can head over to the official site and download it.  If you’d prefer to keep your data online so you don’t have to carry it with you, a site called TiddlySpot specializes in providing free online TiddlyWikis (public or private!).  Just supply a site ID and password and you’re on your way.

1. Twitter was an attractive idea
2. Twitter didn’t really fit with the distributed nature of the web
3. Twitter didn’t scale all that well

an obvious opportunity emerged: build a better mousetrap, err, Twitter. And that is exactly what the Laconica software does. Built by Control Yourself, and hosted on their site at Identica, it provides a credible alternative micro-blogging service. But how comparable is identi.ca to Twitter?

Twitter has some trademark features. First, it allows for user control of content: a user can not only post and delete posts, but can mark his or her entire feed as private, which allows for a sort of walled garden. This makes Twitter very appealing to casual users that really don’t like the idea of the entire web reading every update. Identi.ca, in contrast, provides no mechanism to make feeds private, and similarly has no mechanism to delete posted content. Depending on your point of view, this can be regarded as either a bug or a feature.

One of the early promises of Twitter was the ability to deteremine how you received updates. This is a new idea – Twitter allows you to decide to pull data from the service via RSS, HTTP, or the API (in a desktop client like Twhirl), or have the information pushed to you via XMPP (Jabber), or even messaged via SMS to your mobile. This flexibility is quite novel – it decouples how data is inserted into the system from how it is retreived. This is quite different from email, chat, or even other web services like MySpace or Facebook, in that the granularity of control is quite small – I can elect to receive notifications via SMS for updates from my family, but not from old friends from university. Now that Twitter has all but given up on XMPP because of the scalability problems, identi.ca stands alone in its ability to integrate with chat and SMS to provide this decoupling.

Maybe most importantly, Twitter is itself a walled garden. While it gains many advantages from being centralized (like fast updates) it also suffers from problems associated with centralization: poor scalability, single point of failure and data lock-in. Identi.ca avoids these by adopting OpenID (I have an OpenID through the handy ClickPass service), leveraging the OpenMicroBlogging specification, and allowing instances of Laconi.ca to federate together to form a web of micro-blogging services. For those of you wondering why this is desirable, it is in part because the internet is fundamentally a peer-to-peer, federated system. Email, newsgroups and IRC are all decentralized – these are the classic services that define the web. Recently we have started to move away from decentralized services, but the problems quickly emerge. With AOL Instant Messenger, for example, AOL actively seeks a way to monetize the service, and can make life difficult for third party chat clients that don’t support in-client advertising. As a result XMPP (Jabber) emerged as a viable, open, decentralized chat service, which most internet users have adopted without knowing it by using Google’s chat service integrated into GMail. Just as the problems with Twitter’s centralization emerge, we see an open, decentralized alternative emerge, Laconi.ca. This seems to be the natural evolution of the internet.

Is Identi.ca an perfect replacement for Twitter? No – there are enough differences to attract different crowds, but the similarities are strong. Perhaps as Laconi.ca adds features, we will see it become a more viable complete Twitter replacement. You can be sure I’ll be looking into hosting in instance of Laconi.ca at something like ublog.etherplex.org soon.