Dominic Sayers

Social Data Flow minor update

I'll probably write some more about Social Data Flow shortly, but for now I just wanted to talk about extending the canonical data flow with bespoke updates flowing through my own site. In a perfect world, my syndicator (which happens to be ping.fm) would take care of all my syndication needs. It would take my micro-blogs and status updates and send them to all known social sites where I have an account. Unfortunately while ping.fm appears to have the best coverage, it's not complete. Posterous, for example, is a significant omission (any site where Stephen Fry is active has to be considered a significant site, or anyway it soon will be). As it happens, it's very easy for me to update Posterous with my ping.fm outpourings since I've already implemented a ping.fm Custom URL for my own site - it updates the dynamic sections in the left-hand column. All I had to do was arrange for the PHP page that receives the ping.fm updates to then mail the same update to Posterous. Job done. Not sure why ping.fm haven't done it themselves, except that pressure of other feature request probably prevents them doing everything they want to straight away. More to come on my struggle to get an acceptable flow of location data in place.

Social data flow in action

OK, I've seen a few things online recently that amused me and I'd like to share them. I don't want to add them to delicious.com because that's what I use for links I need to find again someday. These are things I've seen and enjoyed but they are gone. For me they are yesterday's newspaper. But you may not have seen them and I think you should. So I've decided to add them to Tumblr. Easy. Tumblr is a great tool for doing this, it makes it easy to add videos, links, pictures, anything. I am very happy with this choice. But this has now repurposed Tumblr in my social data flow from an output device to a content provider. Before, it just used to receive my status updates and micro blogs. Now it is the place where I link to things that have amused me. Here's how I needed to reconfigure my social data flow: 1. Add Tumblr to my aggregators. Now my Tumblr posts will appear in my All Activity feed. 2. Remove Tumblr from my syndication list. Now Tumblr will not receive my status updates or micro blogs. This is important because otherwise these updates would appear twice in my All Activity feed. 3. Er... 4. That's it. Simple.

Revised social data flow

In an attempt to remove the latency problem from my social data flow, I've rearchitected it a bit. Unfortunately the reworked flow is not quite as elegant as the original but then it had more sophisticated design goals so that's not terribly surprising. Here's the revised flow with notes below:

(Click on the image for a bigger version)

Notes:

1. Now I can allow ping.fm to update Twitter and identi.ca immediately so I have removed the latency problem from the previous configuration. Once again I can use Twitter in a conversational way instead of it being a mere receptacle for RSS updates every half hour.
2. You'll note also that Twitterfeed can now update identi.ca and other Laconica-based micro-blogging sites. Thanks, Mario. This means I can promote identi.ca to be a peer with Twitter and receive all my updates including real-time micro-blogs (tweets).
3. The three subscription points are the same: Writing, Status Updates and All Activity. The URIs for those resources have not changed.
4. I have distinguished between blog posts such as this and the notification of those posts. To subscribe to the full post stream use the Writing feed. If you subscribe to the All Activity feed you will be notified about posts but only see the title not the whole article. Unfortunately if you subscribe to both you will see both the blog post and a separate notification of its existence - I can't do anything about this yet.

So there's an emerging gap between those select sites that I use to do stuff and those sites that merely receive notification of that activity. My strategy will be to keep to a minimum the sites I use for doing things, and maximise the number of sites that are notified about it. This way I can keep all my pictures, links, writing etc. in one place but all my disparate multiple friend communities will get equal notification of new activity in one of those upstream sites. I think the new slightly less elegant social data flow is functionally better and I'll try to stick with it for a while to see how it performs in practice.

Initial thoughts on my new social data flow

Firstly, apologies to anybody who couldn't see the social data flow diagram in my earlier post - I've now hosted it on my own server so it shouldn't disappear quite so often.

(Click on the image for a bigger version)

The new data flow, using ping.fm to syndicate my status updates and micro-blogs and FriendFeed to aggregate my other activity and consolidate everything into a coherent set of feeds, is working quite well. I would like to be able to have more control over the format of what both ping.fm and FriendFeed do for each target system, but the defaults are adequate for now. Here are the issues I'm experiencing: 1. Using Twitterfeed to take my FriendFeed RSS and pipe it into Twitter introduces a degree of latency that's a bit inappropriate for Twitter. Twitter should be near real-time and conversational (whilst not being an instant messaging system), but Twitterfeed only reads my FriendFeed every half an hour. If I get ping.fm to update Twitter directly then I'm going to get some updates twice, which is exactly what I'm trying to avoid. 2. Some of the connected systems (identi.ca for example) are Twitter-like, and I'd like to treat them the same way as Twitter: as an output device for all my activity. But Twitterfeed is Twitter-specific and I haven't found an equivalent for identi.ca etc. yet. In any case I don't want a different Social ETL tool for each output system; I want Twitterfeed (or something) to do this for a number of target systems. 3. There's no class solution for location-based systems yet. Obviously I'd like to be able to update my location and travel plans in one place and have it propagate to TripIt, Dopplr, BrightKite etc. I think FireEagle may be trying to do this but it's short of two things at the moment: co-operation from the other systems and an invitation for me. If anybody has access to FireEagle I would appreciate a way in. 4. Now I need to do the other thing - deduplicate my profligate friends' updates. When Myrto uploads a picture to Flickr, for instance, I get notified about it four times. I can rationalise this a bit but I can't ignore her FriendFeed or Twitter because I would miss some of her updates, so I'm condemned to hearing about her new pictures multiple times unless (a) she adopts a version of my social data flow or (b) I find a solution to deduplicating the same update coming through multiple channels. This duplicate filter is necessary until everybody adopts my architecture, i.e. for ever. Does anybody know of a solution out there? This all seems quite difficult to manage and well beyond anybody who is either busy or technically challenged. There's no real way of developing a packaged solution until all the social systems adopt a single sign-on technology like OpenID, and many of them have sound technical reasons for not doing so (OpenIDs can be created by anybody anywhere so it's a bit like expecting system owners to trust a digital certificate that has no trusted root certification authority). Anyway, that's the state of play at the moment. More to come I'm sure.

Music by year of release

JP wrote about choosing his Desert Island Discs and said how hard it would be. I can save him the trouble - his eight tunes would be these: JP's Desert Island Discs

1	Blind Faith – Can't Find My Way Home full track
2	America – Don't Cross The River full track
3=	Cheb Mami – Desert Rose
3=	Gordon Lightfoot – Did She Mention My Name
3=	Crosby, Stills & Nash – Suite: Judy Blue Eyes
6=	The Who – Behind Blue Eyes full track
6=	Leonard Cohen – Famous Blue Raincoat full track
8	John Martyn – May You Never full track

I copied this list from his last.fm profile. They are the tracks he has listened to most often. Ipso facto, they are the tunes he should take to his island exile. JP's approach was a bit more labour intensive. As a high-bandwidth consumer he has too many albums to choose from. He tried to narrow it down by various criteria, ending up by selecting arbitrarily from only material released in 1971. For him, this appears to be a sweet spot year. I wondered where my own sweet spot was, so I looked at all the tunes I have in my Media Player library (sorry I don't like iTunes, particularly not on a PC, and I don't have an iPod because my phone plays tunes quite adequately thanks). Here is how they break down by year of release:

Some of my tunes are from compilations released a long time after the tracks were originally recorded, so this chart is skewed quite a lot towards recent years. Still, it's interesting (to me) what sort of picture it paints of my musical tastes. I would have guessed the 90s would be a sweet spot but it seems that a lot of my stuff is more recent than that. There wasn't much tabla-based cheesy house music around in the 90s I suppose. And if you're interested in my Desert Island Discs, last.fm says they are these: Dominic's Desert Island Discs

1	Paul Weller – Wild Wood full track
2=	The Bangles – Walk Like an Egyptian full track
2=	BT – Blue Skies [Deep Dish Blue Phunk Mix]
3	Opera IX – Bela Lugosi's Dead
4=	The Modern Lovers – Egyptian Reggae
4=	Tom Waits – Rain Dogs
4=	Pink Floyd – Shine on You Crazy Diamond, Pts. 1-5
4=	Tom Waits – Whats He Building

If I had compiled the list manually it would have been completely different, I think because I would have been asking what the tunes said about me instead of what I actually liked listening to. In other words, I would have been worried about how cool the tracks were not how much I secretly liked them. What are your last.fm Desert Island Discs? And what is your sweet spot year?

The next layer, and the 64-bit tipping point.

2008 is a hard deadline for the adoption of 64-bit computing, according to Eric S. Raymond (author of The Cathedral and the Bazaar). He made this claim in 2006 based on historical trends of memory use by personal computers. As far as I know, there is no reason why this could not be come to pass - next year you will be buying a 64-bit processor and more than 4GB of RAM in your personal computer. It's never been easy to predict in advance what triggers the need for this extra horsepower, but in the past it has been the addition of layers of software underneath whatever programming language that software developers were using. If you're writing in assembler for a particular processor then you can squeeze a lot of functionality and performance into quite a small space, but your application won't run on anybody else's processor and won't be able to use a wide range of peripherals. When you start to abstract those incompatibilities away, that's what starts to slow down your application and make it consume lots of memory. What's the next layer of abstraction that will push us over into the 64-bit world? Joel Spolsky thinks he has identified it: in his Strategy Letter VI he writes about a hypothetical application platform that will let you forget about the incompatibilities between different browsers' interpretation of Cascading Style Sheets and their different object models. Moreover it will let you forget about different ways of publishing your online status and journal (Facebook or Twitter? Who cares? I publish to all of them). For me, it's one of those eye-opening articles that makes the world seem like a slightly different place. GMail is today's Wordperfect? Obvious when you think about it.

And while you’re not paying attention, everybody starts writing NewSDK apps, and they’re really good, and suddenly businesses ONLY want NewSDK apps, and all those old-school Plain Ajax apps look pathetic and won’t cut and paste and mash and sync and play drums nicely with one another. And Gmail becomes a legacy. The WordPerfect of Email. And you’ll tell your children how excited you were to get 2GB to store email, and they’ll laugh at you. Their nail polish has more than 2GB.

Somebody must be working on this application platform, surely? Any ideas who? And when can I play with it?

Dick Hardt: no longer homesick

[Edited to add: the marketing company have now taken down the promotional site leaving simply a banner ad for Dylan's album. Boo! The intended joke was a mapping of Dick Hardt's famous talk about identity onto the Bob Dylan video that inspired it. It was quite a good joke and it's a shame that the marketing company have taken the site down.] You may have seen the viral campaign promoting Bob Dylan's new album. Here, for your amusement, is the obvious mashup: (for the uninitiated here is the "original")

Managing my reading lists

Steve Patrick, who clearly has my education as a personal mission, has recommended another couple of books (for which I thank him). I've added them to the lists. By the way, the idea of storing del.icio.us tags in del.icio.us itself is a fabulous one which I can see all sorts of uses for. I am very pleased with my list of lists. Now, Steve's comment raises a couple of points of order. Firstly, one of the books is a work of fiction. You (o erudite reader) might think that some of the other books on my lists are works of fiction too but I haven't read them so I can't say that yet. This has led me to open a new category, fiction I should read. Hopefully it won't get too long too soon. Secondly, Sean Park suggested that one of the books on my list wasn't very good so I gave it a "deprecated" category - not removing it from the list altogether but perhaps putting it to the end of the queue. Steve Patrick disagrees with this assessment and thinks it should be restored to its rightful place. How do I deal with this? I've decided to use the Charmed principle. Steve cannot remove a spell that somebody else has cast on him. Sean either has to remove it himself or another witch needs to cast a reverse spell. Them's the rules. If you want Traders, Guns and Money restored to its place on the list then let me know. Footnote: I don't think I could manage my reading lists this well using LibraryThing, Google Books or Facebook's Visual Bookshelf. Using del.icio.us is really quick and easy and does everything I need. The book-specific tools want me to do things in a particular way but del.icio.us is general and universal.

Listening to last.fm recommendations

If you have a last.fm account, one thing you can do is listen to tracks that it recommends for you based on your listening habits. Last time I tried this it gave me a farrago of lowest-common denominator stuff that meant nothing to me. Today I tried it again and I'm loving what it is piping into my ears. This is what it recommends for me. What has happened to change this into something useful? There are a number of possibilities: 1. last.fm has more information about me I've now got my favourite tracks in a playlist and I've been playing them a lot. That's a couple of hundred eclectic tunes that represent what I like hearing. last.fm learns from this. 2. last.fm has more information The number of users of last.fm is growing and it has information about a wider selection of tracks and artists, and more information about which tracks people play a lot. 3. The matching algorithm has improved Not necessarily, but certainly possible. The improvement in the service could be explained by the first two factors alone, but I'm sure they tweek the algorithm too. My listening tastes vary from the popular to the quite obscure (like everybody else's probably), but I've been impressed with the variety of tunes last.fm has chosen for me. Artists I would not have considered because I thought I knew I didn't like them (Moody Blues for example), artists I had never even heard of (Sun Electric anybody?), tracks I had never heard (a remix of Tubular Bells with an extended drunken 3am monologue from Vivian Stanshall). Some of the selections cause me to press the Ban button straight away but presumably this contributes information to the algorithm to improve the service for me and everybody else. Try it.

Hiatus

Whenever I am chided by people for not updating this blog I turn to What everyone should know about blog depression.