Etherplex

Though perputually on the verge of total disorganization in the physical realm, I try to keep my digital world very organized.  Because of this, I habitually try to move more and more of my life into a digital representation, which allows easy backup, searching and reorganization.  Although I have tried many free and proprietary software products to act as a central hub for storing and retrieving random bits of useful information, even the very best (Emacs Org Mode) has failed to gain traction in my life on a long term basis because it is not immediately accessible to me when its underlying framework is not present (Emacs).  Of course, Org Mode files are just text, but much of the file’s appeal is lost in a pure text mode: Emacs is required to really wrangle all the data present in the text file.  I have used other tools, like DevonThink on OS X, Google Notebook, Google Calendar, iCal, various GTD systems, several Emacs-based packages, and Evernote.  All either store my data remotely, aren’t workable on Windows, OS X and GNU/Linux, or require a sizeable framework to be present whereever I want to view/alter my data.  Each of these shortcomings causes me to shy away from really “investing” in the system.

In the past few months, however, I have rediscovered TiddlyWiki, which, despite its name, is a very serious piece of software.  Though written only in Javascript and HTML (to run in a browser), it is a serious software engineering effort, and has a large community doing active development of many different variants.  The purpose of this post is to provide a look at what TiddlyWiki has to offer, so as to assist potential new users (you!) in determining if it might be useful to organize information in their lives.

TiddlyWiki is a Wiki

This means in is accessed in a web browser that you already have on your computer.  It works best in Firefox, but can be used in Safari, Internet Explorer, and Opera as well.  TiddlyWiki works an whatever operating sytem you have.

This means it has “pages”, as seen in other wikis (like Wikipedia), except instead of entire webpages, TiddlyWiki uses Tiddlers: small (or not-so-small) chunks of formatted text about a particular topic.

This also means it uses hypertext, so you get both internal (links between tiddlers) and external (links to other websites) linking, as well as formatted text: bold, underline, italic, monospace fonts, numbered and bulleted lists.  The linking feature is critical, because it allows you to link tiddlers to each other in the same way the internet links webpages.  You can use this mechanism to organize your data as you see fit – whether that be in a hierarchy (like the files on your computer) or in some web that more closely resembles Wikipedia.  You are in complete control.

TiddlyWiki is Just a File

This means it resides on your hard drive or USB stick just like any other file.  You can copy it, give it to others, post it online, or carry it around with you just like any other file.  To open it, you just double cilck on it and it will open in your browser.

TiddlyWiki Supports Metadata

All I mean by this is that it allows you to tag tiddlers (pages) with their topic.  If you’re a programmer, you might tag tiddlers with the language they relate to.  If you’re a journalist, you can tag a tiddler with the article it relates to, or the source it came from.  Tiddlers can have as many tags as you like.

TiddlyWiki Can Search

As you grow the content in your TiddlyWiki (by adding new Tiddlers), you may start to worry that while it is really easy to add information to the TiddlyWiki, retrieving it may be difficult.  Where is that darn tiddler that contained the dentist’s address and office hours?  Even if you can’t remember what you called it (was it under “Dentist” or “Medical”?) so long as you used the word “dentist” anywhere in the tiddler, you can simply type “dentist” into the global search box, and, as you type, TiddlyWiki will search every word you have stored in it and show you all the tiddlers that contain that keyword.  If all you can remember is that you had the words “office hours” in the tiddler, just type that and you’ll track your data down in no time.

You can try this on the TiddlyWiki homepage (which is itself a TiddlyWiki!).  Upon visiting the page, you have no idea if there is any information on whether TiddlyWiki works on the browser available for the Nintendo Wii.  Simply type in the keyword “wii” in the search box and you can find out if there is any information on the Wii anywhere in the TiddlyWiki.

TiddlyWiki is Organized

In addition to full text search of all your data, TiddlyWiki can also list all the tags you’ve ever created and allow you to view lists of all tiddlers that match those tags.  You can then open the tiddlers you want from that list, or open all tiddlers that match a given tag.  At home, I work on schoolwork for my current class (csc576 – Connection Oriented Neworks), so to bring up my notes for that class, I just go to the tag I made called “csc576″ and request all tiddlers matching that tag be opened.  Similarly, at work, if I’m working on a project relating to JBoss, I can open all my notes on JBoss by selecting the tag I created called “jboss”.

In addition to tagging, TiddlyWiki offers a series of tiddler lists that are organized in various useful ways.  In addition to the tag list view (mentioned above), TiddlyWiki also offers a timeline of tiddler edits, allowing you to see what has been changing in your TiddlyWiki at a glance.  Can’t remember the name of the tiddler you were editting on Monday?  Just look at the timeline and it will tell you what changed that day.  In addition to the timeline, you may also find that you create tiddlers that are not referenced from anywhere else.  They may contain useful information (the VIN number on your old 1991 Toyota Corolla), but you won’t find them just by “surfing” through the TiddlyWiki.  Well, luckily, TiddlyWiki keeps track of all these “orphaned” tiddlers and can give you a list, so that you won’t lose them.  Conversely, sometimes a tiddler has a link to it, but it doesn’t yet exist.  TiddlyWiki also tracks these cases, and provides a list if needed.

Conclusion

Overall, I have found that TiddlyWiki has provided a sort of “sweet spot” in the field of tools to organize your data.  It is extremely easy to pick up and start working with, but has an extensive feature set that gives advanced users  a lot to work with.  It is open source and extensible, allowing sties that collect extensions for TiddlyWiki to spring up all over the web.  If you’d like to keep your own TiddlyWiki on your local machine or USB stick to carry with you, you can head over to the official site and download it.  If you’d prefer to keep your data online so you don’t have to carry it with you, a site called TiddlySpot specializes in providing free online TiddlyWikis (public or private!).  Just supply a site ID and password and you’re on your way.

It can be really difficult to tell the difference between a skillful troll and someone who really is that clueless.  For that matter, it can be really difficult to tell the difference between parody and reality, especially when one predicts the other.

That said, I honestly have a really hard time figuring out whether the HeliOS Project has been the victim of a troll or not when I read this.  This is very similar to Poe’s Law, which I myself have fallen victim to on occasion, though instead of being unable to distinguish parodies of fundamentalism and real fundamentalists, I cannot tell whether or not there are people who really think this stuff.  Anyway, I wish the best to HeliOS, it looks like they have their work cut out for themselves.

Blog of helios: Linux – Stop holding our kids back

Just ran across (via Proggit) an interesting blog post about Core War, a game I played a bit with on Linux a few years back in which programs compete for dominance inside a virtual machine.  Very neat idea – you program an agent that will go and fight for you in a virtual world.  Cool.

What I didn’t know was that A.K. Dewdney wrote an article about Core War in Scientific American in 1984.  This is the same A.K. Dewdney that authored one of my very favorite science fiction/philosophy books ever, The Planiverse.  It turns out it is sort of a small world.

I mentioned a couple of posts ago about a technique for scoping state I used when I programmed in Scheme more often than I do now.  As I’ve been picking up Clojure as my hacking language of preference, I was surprised to find that it doesn’t support the same idioms.  In Clojure, any variables referenced when  a function is defined in a local scope are unbound when the function is returned to an outer scope.  This means the code I wrote in Scheme doesn’t have a direct analogue in Clojure.  This was a deliberate decision made by Rich Hickey, Clojure’s designer/author.  As he says:

If locals were variables, i.e. mutable, then closures could close over mutable state, and, given that closures can escape (without some extra prohibition on same), the result would be thread-unsafe. And people would certainly do so, e.g. closure-based pseudo-objects. The result would be a huge hole in Clojure’s approach.

The end result is that when you first pick up Clojure, if you try to close a function definition over a variable, you’ll find it is no longer bound when you leave the enclosing scope, and if you try to close over let-bound value (local), you’ll find that is immutable, and can therefore not be used to store state.  So how do we accomplish the same result as we did in Scheme?

You use Clojure’s shared transactional memory and create a mutable ref that is modified in a transaction to preserve thread-safety:

(let [z (ref 0)] (defn add [] (dosync (ref-set z (inc @z)))))

We still make use of local binding, but we bind z to a ref, which we can then modify inside of a dosync call.  Naturally, an arbitrary number of functions could be defined inside of the let that all shared a reference to z.  Once outside of the let, z is no longer accessible to the casual programmer, but the functions still have full access to it, creating a sort of data hiding/state-preserving closure that protects the “private” variable z.

I don’t use this idiom often, but when I want it, it’s exactly what I want, and avoids the need for more complex constructs to simply have a pair of functions share a variable that isn’t modifiable in other parts of the program.

My Emacs startup time was stretching beyond 10 seconds, which seemed a bit offputting even for a text editor cum operating system. I do have a fair amount of elisp loading at startup, however, and though I have played with byte-code-cache, I determined there must be an easier way to compile all my elisp files in one go, rather than calling byte-compile-file for each file in my elisp directory.  It turns out there is:

C-0 M-x byte-recompile-directory

will recursively traverse a directory stucture and compile all .el files encountered.  My startup time is back to 4 seconds.

In Java, there are four scopes: public, package friendly, protected and private.  These are useful, but rather coarse.  It would be nice to be able to specify a variable that was only visible to a select set of methods.  For example, suppose you wanted to develop a counter that could not be modified without being accessed through methods that modified it.  In Java, this would be accomplished with an inner class that had two methods.  It’s interface would have to be declared publicly, though.  In R5RS Scheme, this would look like:

(let ((z 0)) (list (lambda () (set! z (+ z 1))) (lambda () (set! z (- z 1)))))

This defines two functions that each close over the variable z such that you must call the first to increment z and you must call the second to decrement z.  Because z is not defined at the top level, there is no way to access it and modify it directly — you must go through the functions.  This allows the functions to be a bottleneck to the modification of z.  So, if, for example, you wanted to log every time z was changed, you could add a logging command to each of the functions.  This approach allows arbitrarily fine scoping to be defined.  If you want a variable to be available to only one or two functions, it is trivial to do so using the “let” construct.

I’ll take a look at how you can accomplish similar results in JVM-based functional languages like Scala and Clojure in my next post.

It took me 30 years, but I woke up this Sunday morning and realized that rational numbers can be expressed as the ratio of two integers.

By far the most brain-bendingly elegant definition of all the Fibonacci numbers I have ever encountered:

(def fibs (lazy-cat '(1 2) (map + fibs (drop 1 fibs))))

Found over an the clojure-euler wiki.

Just watched the December 2007 Google Authors session with Randall Munroe, creator of the XKCD webcomic.  Really good talk; he seems like a real geek, which is awesome.  He inspired me to sign up for Project Euler, which I have.  I did the first problem in a couple of minutes, and visited the forum to find out what kind of code others were writing.  I continue to love lisp, so I think I’ll be tackling the challenges in Clojure.  The first problem yields to some basic list operations:

(reduce + (filter #(or (= (rem %1 3) 0) (= (rem %1 5) 0)) (range 1000)))

We’ll see how well Lisp holds up for the later problems.

Clojure is a superb new-ish lisp for the JVM that is quickly becoming my preferred hacking playground.  As with any new language, as soon as you write anything worthwhile, you want to figure out how to distribute it.  In Clojure’s case, even though Clojure is functional and is dynamically compiled, the process of creatng an executable jar isn’t too bad.  I outline it here for reference:

1. Create the application directory structure.  For this example, I’m using org.etherplex.example.JarExample.  I’m going to make a project directory called simply “project”.  I’m assuming Linux/OS X here, Windows users need to adjust this or install Cygwin.

   mkdir -p project/org/etherplex/example

2. Create your application in Clojure, creating a main method that is named properly.  For this example, I used a file called JarExample.clj.  It resides in the project/org/etherplex/example folder.

   (in-ns 'org.etherplex.example)

   (clojure/refer 'clojure)

   (def JarExample-main (fn [& args] (println "This is an example application!")))

   If we want to have our “main” class be called JarExample, then in clojure, we need to define a function called JarExample-main.  In general, Clojure will look for methods of a class in the same package as that class, and will use the form TheClass-methodName to locate the proper class.

3. Now we need to create a stubbed .class file that will dispatch method calls to our Clojure file.  The easiest way to do this is to use the REPL.  I start the REPL in the directory that is the parent of my project directory.  In this case, that’s my home directory.
   

   cd ~

   java -jar path/to/clojure.jar

   Now that we are in a Clojure REPL, we can create our class stub:

   (gen-and-save-class "project" "org.etherplex.example.JarExample" :main true)

   First, we specify the path that contains our files (“project”) and then the fully qualified class we want to create.  We also specify a flag that indicates that we want Clojure to create a main method that dispatches out to the Clojure file we created in Step 2.

4. Unpack clojure.jar so you can include its contents in your jar file.  We need to make sure our jar has all the dependencies in it so it is easy to distribute.

   cd ~

   unzip path/to/clojure.jar

   mv clojure project

5. Now we need to create a manifest file for the jar so Java know to execute JarExample.class.  This is the contents of a file called manifest.txt that I put in my project directory.

   Main-Class: org.etherplex.example.JarExample

   Class-Path: .

   It is vital that this file have a trailing newline!

6. Finally, we can create our jar file:
   

   cd ~/project

   jar cmf manifest.txt myjar.jar clojure org

   And execute it with:
   
   

   java -jar myjar.jar

   This will create a jar file that contains clojure and your code, and will interpret/compile your .clj file on the fly and run it.  Good luck!