dis·am·big·u·ous·i·ty

I don’t want to worry much about my digital stuff. One of the reasons it’s all digital in the first place is for ease-of-use. Unfortunately, if I wasn’t making sure those files would be safe in case of disaster, theft, or just an accident, I would end up regretting it. On top of that, I can’t access the stuff I need sometimes, even within my own home on the several different computers. Mind you, I’m a software engineer – I can set-up network shares like I have been doing for years, but I’m tired of all that. This is especially true when I’m dealing with my Windows and Linux machines (and probably a Mac too soon).

What I want is a permanent home in the cloud, and I want to hook into that cloud with about one click. I want that cloud to be highly available (that’s IT jargon for “always there, no matter what”), but I don’t really want to pay anything (after-all, I do already have a bunch of machines laying around that cost money). I want every file to have at least two copies and for those copies to reside in more than one location. I want access to all my files everywhere, from every machine. I don’t want to have to even think of “backing stuff up” – that should be done for me.

I think the solution here hinges on a few different things:

1. The cloud is all of your filesystem space wherever you have it.
2. Thin client software that hooks into the native filesystem of the machine being used and runs a control in a “task tray” or maybe a widget. Possibly utilizing something similar to libfuse in order to make remote files look local. The control would probably end-up with a few different features, but most important would be conflict notification and resolution.
3. Automatic version control using something like mercurial (I would have said subversion, but my requirements are for a distributed model as opposed to centralized) that constantly sends your updates to the cloud (even at the frequency of auto-save in certain cases, but usually just at save time).
4. A peer-to-peer buddy system for your back-ups. In other words, if you allocate that extra hard drive space to the cloud, you will get that much space out on the cloud (somebody else’s space) to back-up your files. I’m not sure the donated space would be able to actually be one-to-one because the files would have to be replicated to more than one buddy for reliability reasons, so maybe you would get half or one-third the space that you donate. If need be, you could hook into a paid service (either for the P2P-style sharing or for traditional centralized file sharing).
5. Specialized “syncing” for services such as flickr, gDocs, youtube, facebook, etc. Some easy, possibly filesystem-based way, to indicate which of my things I want to “publish” to various services. I wonder if dataportability has anything to offer here?

That fourth point there needs some work. There are serious reliability issues and the efficiency of P2P isn’t all that wonderful when you are talking about files that are only in a few places. The fifth point is also icing on the cake, so I could live without it for now.

I guess the question is, can libfuse(?) + mercurial + xmpp + some python rig something like this up? By the way open source is awesome.

Does anybody know of some apps out there that already do this? I did try out that PowerFolders app, but it was a bit clunky and I could never get it working, although it had several of the right ideas. When is google or amazon going to come out with this app for me? With how large and important my digital stuff has become (especially family photos and videos), I think this is worth around $10/a month to me (err… maybe less).

By the way, somebody is domain squatting on mycloud.com right now, and I’m sure writing this blog post will make the domain even more expensive, but oh well!

Ok, now I’m feeling a bit obsessed with this idea. It would make my life tremendously easier if I didn’t have to think about managing my different files. I just want them to always be available on all of my mahcines, but without having to change the way I work at all.

Last night, I touched on how to make Windows a little less ugly such that it will work well with the standard *nix directory layout. I also mentioned some resources that go in to detail about how they manage their CVS/SVN syncing issues. But then I started to realize those solutions have one fatal flaw – they still require you to “just cvs up at the end of the day.” I don’t even want to have to worry about that, because I won’t remember to do it.

Now I feel compelled to lay out my requirements for what would be the perfect home directory solution. Maybe I’ll call it “Distributed Home Dir” or something more creative, but along those lines:

• Synchronization of most files should happen automatically, unless there is a conflict
• Certain “large” files (usually binary) should not be automatically sync’ed
• Somewhat intelligent defaults should exist to determine which files need to be synched up and which ones don’t
• Options should exist for whether or not to have “full synching” ( constanly tracking changes a la Google Docs and keeping a record of the history), “just copy” (no history), “manual history” (manually choose to ‘commit’ changes), “ignore,” and probably others.
• Bad practices should cause warnings along the lines of: “Saving history for these large binary files will take up lots of space, maybe you should just save the latest copy “
• Files that are not automatically sync’ed should be able to be obtained on demand (assuming connection is available)
• Most syncing occurs in the background at very frequent intervals (on par with gMail or Google Docs)
• Each sync’ing node should be able to be a source as well
• Each sync’ing source should be able to proxy another node as well
• Multiple available nodes might afford a bump in bandwidth a la BitTorrent
• “Installing” should be drop-dead simple, and maybe even pre-configured if you are able to access the install from one of your existing nodes
• “Thin” nodes such as a flash drive or PDA should integrate well
• Alerts might occur when you are starting to cause a conflict, at which time you can choose to branch (layman’s terms: “work on a second copy and merge later or keep a separate copy for this environment”)
• Some “branches” will have to be intentional however, so as not to overwrite intentionally different types of files. For example, the %HOME%/Documents/My\ Google\ Gadgets folder you find in Windows.
• “System tray” capabilities would be nice
• Integration with other “file systems” (see the MacFuse Google Tech Talk for more info)
• Solving the “Shared” problem would be nice (i.e. when we download pics from our family camera, they go in a shared home dir . . how do we get access to those resources on another machine)
• Backed by established version control systems, most likely a distributed one like mercurial.
• Encryption should be mandatory

That is probably more than enough to get things running. Hopefully I won’t have to write much code (otherwise this may never happen).

I’ve really over-complicated my life at this point. I’ve turned my quaint little home into a freakin’ disorganized computer lab (in case my wife ever reads these posts, I should thank her for putting up with me). At the moment, I’m trying to get a handle on all the files all over the place.

I’ve gone down this road a million times before and never ended up tackling the problem sufficiently. However, I’m feeling inspired by a blog post I just found illustrating HOWTO rid yourself of the abhorrent “Documents and Settings” and “My Documents, My Music, My This and My That. ” I always thought my aversion to the Windows way of things was just because I was a command-line junkie, but I now realize their mess was part of the roadblock to organizing the various files on all these systems.

I’ve been hoping to use a source control solution like subversion (of course, after listening to Linus Torvalds at a Google Tech Talk, I’m thinking this is the perfect use case for a “distributed source control” solution). However, I’ve never wanted to go after bits and pieces at a time – I want the whole dang home directory synched – BOOM. I’ve never had the desire to play around with where to put various crap. Up until now, the best solution I could come up with was from one article I read which had a slick solution of checking in little set-up scripts for various environments. Still though, those scripts sound kind of ugly when Windows is all high maintenance.

Then comes the article: Derek Slager: Emulating Vista’s User Directory Structure on XP In a nutshell, instead of relying on the various scripts for the various environments to piece together your desired directory structure, just change the screwed up environment itself! If I can pull this off, then the change will also be able to be applied to the Home Directories of the wife and kids, which will be the proverbial grand slam of overly complicated home computing.

So, the plan is the following:

1. Implement the changes from the referenced blog on all my XP instances
2. Implement source control (some resources: http://toykeeper.net/tutorials/svnhome, http://tratt.net/laurie/tech_articles/articles/home_directory_synchronization, http://linuxjournal.com/article/5976)
3. Make the conflict resolution process drop-dead simple for even my wife (who is actually pretty savy at computing, but hasn’t seen a diff in her life)

Honestly though, this will only solve some of my problems. I would still much rather prefer being able to just “mount” certain network directories to certain places (specifically in dealing with our shared pictures, as the size of them is too big to “distribute” to all machines).

Alas, let’s see where this adventure takes us.

I came across this product called gumstix the other day while investigating Bluetooth on Linux. Apparently they’ve got cards to add Bluetooth, WiFi, GPS, etc. to these tiny “computers.” Make these devices part of “home grid” with a web-based control center, add a wireless PDA-esque device to control the whole thing, and you’ve got some nice potential.

If you can find a way to add audio or video output, you’ve got something on par with a squezebox. Or, if nothing else, these devices could just be hubs for a given “zone” in the house, connecting to local Bluetooth devices, interfacing with X10, etc, etc.

More on the PDA devices:

• RemoteCentral’s CEDIA write-up
• Nokia’s Internet Tablet
• The MP3 Toaster
• miniITX
• UMPC’s
• Sharp Zaurus