Dominic Sayers

Revised social data flow

In an attempt to remove the latency problem from my social data flow, I've rearchitected it a bit. Unfortunately the reworked flow is not quite as elegant as the original but then it had more sophisticated design goals so that's not terribly surprising. Here's the revised flow with notes below:

(Click on the image for a bigger version)

Notes:

1. Now I can allow ping.fm to update Twitter and identi.ca immediately so I have removed the latency problem from the previous configuration. Once again I can use Twitter in a conversational way instead of it being a mere receptacle for RSS updates every half hour.
2. You'll note also that Twitterfeed can now update identi.ca and other Laconica-based micro-blogging sites. Thanks, Mario. This means I can promote identi.ca to be a peer with Twitter and receive all my updates including real-time micro-blogs (tweets).
3. The three subscription points are the same: Writing, Status Updates and All Activity. The URIs for those resources have not changed.
4. I have distinguished between blog posts such as this and the notification of those posts. To subscribe to the full post stream use the Writing feed. If you subscribe to the All Activity feed you will be notified about posts but only see the title not the whole article. Unfortunately if you subscribe to both you will see both the blog post and a separate notification of its existence - I can't do anything about this yet.

So there's an emerging gap between those select sites that I use to do stuff and those sites that merely receive notification of that activity. My strategy will be to keep to a minimum the sites I use for doing things, and maximise the number of sites that are notified about it. This way I can keep all my pictures, links, writing etc. in one place but all my disparate multiple friend communities will get equal notification of new activity in one of those upstream sites. I think the new slightly less elegant social data flow is functionally better and I'll try to stick with it for a while to see how it performs in practice.

Initial thoughts on my new social data flow

Firstly, apologies to anybody who couldn't see the social data flow diagram in my earlier post - I've now hosted it on my own server so it shouldn't disappear quite so often.

(Click on the image for a bigger version)

The new data flow, using ping.fm to syndicate my status updates and micro-blogs and FriendFeed to aggregate my other activity and consolidate everything into a coherent set of feeds, is working quite well. I would like to be able to have more control over the format of what both ping.fm and FriendFeed do for each target system, but the defaults are adequate for now. Here are the issues I'm experiencing: 1. Using Twitterfeed to take my FriendFeed RSS and pipe it into Twitter introduces a degree of latency that's a bit inappropriate for Twitter. Twitter should be near real-time and conversational (whilst not being an instant messaging system), but Twitterfeed only reads my FriendFeed every half an hour. If I get ping.fm to update Twitter directly then I'm going to get some updates twice, which is exactly what I'm trying to avoid. 2. Some of the connected systems (identi.ca for example) are Twitter-like, and I'd like to treat them the same way as Twitter: as an output device for all my activity. But Twitterfeed is Twitter-specific and I haven't found an equivalent for identi.ca etc. yet. In any case I don't want a different Social ETL tool for each output system; I want Twitterfeed (or something) to do this for a number of target systems. 3. There's no class solution for location-based systems yet. Obviously I'd like to be able to update my location and travel plans in one place and have it propagate to TripIt, Dopplr, BrightKite etc. I think FireEagle may be trying to do this but it's short of two things at the moment: co-operation from the other systems and an invitation for me. If anybody has access to FireEagle I would appreciate a way in. 4. Now I need to do the other thing - deduplicate my profligate friends' updates. When Myrto uploads a picture to Flickr, for instance, I get notified about it four times. I can rationalise this a bit but I can't ignore her FriendFeed or Twitter because I would miss some of her updates, so I'm condemned to hearing about her new pictures multiple times unless (a) she adopts a version of my social data flow or (b) I find a solution to deduplicating the same update coming through multiple channels. This duplicate filter is necessary until everybody adopts my architecture, i.e. for ever. Does anybody know of a solution out there? This all seems quite difficult to manage and well beyond anybody who is either busy or technically challenged. There's no real way of developing a packaged solution until all the social systems adopt a single sign-on technology like OpenID, and many of them have sound technical reasons for not doing so (OpenIDs can be created by anybody anywhere so it's a bit like expecting system owners to trust a digital certificate that has no trusted root certification authority). Anyway, that's the state of play at the moment. More to come I'm sure.

Managing my reading lists

Steve Patrick, who clearly has my education as a personal mission, has recommended another couple of books (for which I thank him). I've added them to the lists. By the way, the idea of storing del.icio.us tags in del.icio.us itself is a fabulous one which I can see all sorts of uses for. I am very pleased with my list of lists. Now, Steve's comment raises a couple of points of order. Firstly, one of the books is a work of fiction. You (o erudite reader) might think that some of the other books on my lists are works of fiction too but I haven't read them so I can't say that yet. This has led me to open a new category, fiction I should read. Hopefully it won't get too long too soon. Secondly, Sean Park suggested that one of the books on my list wasn't very good so I gave it a "deprecated" category - not removing it from the list altogether but perhaps putting it to the end of the queue. Steve Patrick disagrees with this assessment and thinks it should be restored to its rightful place. How do I deal with this? I've decided to use the Charmed principle. Steve cannot remove a spell that somebody else has cast on him. Sean either has to remove it himself or another witch needs to cast a reverse spell. Them's the rules. If you want Traders, Guns and Money restored to its place on the list then let me know. Footnote: I don't think I could manage my reading lists this well using LibraryThing, Google Books or Facebook's Visual Bookshelf. Using del.icio.us is really quick and easy and does everything I need. The book-specific tools want me to do things in a particular way but del.icio.us is general and universal.

Listening to last.fm recommendations

If you have a last.fm account, one thing you can do is listen to tracks that it recommends for you based on your listening habits. Last time I tried this it gave me a farrago of lowest-common denominator stuff that meant nothing to me. Today I tried it again and I'm loving what it is piping into my ears. This is what it recommends for me. What has happened to change this into something useful? There are a number of possibilities: 1. last.fm has more information about me I've now got my favourite tracks in a playlist and I've been playing them a lot. That's a couple of hundred eclectic tunes that represent what I like hearing. last.fm learns from this. 2. last.fm has more information The number of users of last.fm is growing and it has information about a wider selection of tracks and artists, and more information about which tracks people play a lot. 3. The matching algorithm has improved Not necessarily, but certainly possible. The improvement in the service could be explained by the first two factors alone, but I'm sure they tweek the algorithm too. My listening tastes vary from the popular to the quite obscure (like everybody else's probably), but I've been impressed with the variety of tunes last.fm has chosen for me. Artists I would not have considered because I thought I knew I didn't like them (Moody Blues for example), artists I had never even heard of (Sun Electric anybody?), tracks I had never heard (a remix of Tubular Bells with an extended drunken 3am monologue from Vivian Stanshall). Some of the selections cause me to press the Ban button straight away but presumably this contributes information to the algorithm to improve the service for me and everybody else. Try it.

Outboard memory

I started taking photographs of my whiteboard a couple of years ago. I use the whiteboard as a thinking tool. Start with what you think you know then, pen in one hand and eraser in the other, you go through a hundred iterations in about fifteen minutes. With ideas, it's deliver early, deliver often.

I became reluctant to clean the whiteboard, realising that I had not fully exploited the ideas that were being generated. But in a shared office this is anti-social at the very least. When is it OK to erase somebody else's thought stream? When it's been there a day, a week? It's a tricky social problem. The answer for me was to photograph the whiteboard when the burst of creative energy had run its course. I put the photographs in dated folders on my computer. Ideally this would have been a taggable repository like Flickr, but my computer at work is a pre-Vista PC so the tagging option isn't easily achievable.

Now my screensaver is simply a slideshow of pictures from my computer, including (almost coincidentally) the whiteboard pictures. The unexpected side-effect is that I am exposed every day to my thoughts from the last two years. For me this a great thing. I don't have one of those minds that can remember lists. In other words, my mind is almost stateless. The whiteboard photographs act like session cookies, enabling me to remember where I am on a much wider range of topics than I could without them. It seems like the external artifact is an intrinsic part of the mental process. The memories of those thoughts are there but I cannot unlock them without the visual key. I've put some of the photos here so I can refer to them in discussions and possibly blog posts.

Famous old joke

I've been looking for a copy of this image for a while. Click the image for a slightly bigger version if you can't read the text. I first remember seeing it when I was a junior developer back in the Thatcher era. I'm pretty sure it was long in the tooth then. Two things about it: 1. We still get the joke today. Worse, it is still true for some projects today. Why haven't we fixed this yet? 2. Does anybody know where this originated? Who wrote/drew it and where was it first published? 3. How can search I the internet to find copies of this image? OK, that's three things.

Handbags at dawn

Hee hee. Armani attacks Savile Row Hugh retaliates Thomas Mahon has already spoken, possibly stirring Giorgio to respond some 18 months later: Don’t fool yourself that just because it's a Hugo Boss or Armani it’s a better fit than than the Marks & Spencer. I am, of course, wearing an old-fashioned Prince of Wales check suit as I write this. And I urge you to do the same until Signore Armani realises the futility of his campaign to get us all wearing shiny tight suits.

Social OPML

Remember how you used to save your bookmarks in the browser (Favorites if you use Internet Explorer)? Then when you discovered del.icio.us you realised that you no longer had to maintain your bookmarks in two different browsers on three separate PCs. And you could use the fact that everybody else kept their bookmarks in the same place to make connections between your bookmarks and other people's You don't go back to using local bookmarks once you have seen the world of social bookmarking. Browsers have been around a lot longer than feed aggregators. Maybe that's why we are still happy to keep our feed collections locally. Maybe we simply haven't discovered social feed collections yet. But this would be the answer to something that is annoying me a lot more now than it did a few months ago. As more and more of the information I consume is available in a syndicated format, I add more and more channels to my aggregator. Some important messages now come through my aggregator. I am starting to depend on it for critical notifications. Wherever I go, my aggregator is open on my desktop and it is as necessary to me as my email client and my chat client. So the fact that I have to re-read the same posts that I have just checked when I move to a different PC is starting to bug me. The fact that I have to add a new channel manually to each aggregator on each PC is starting to bug me. I want to keep my feed collection centrally, outside my aggregator. And the Read/Not Read status of the posts too. del.icio.us for feeds. That's what I need. You read it here first. Stop and think about it for a minute before I start talking about implementation problems. 1. There is, of course, a chicken and egg issue. There's no point storing my feed collection centrally unless my aggregator can use the central store. The central store won't get any kind of critical mass until most common aggregators are using it. None does at the moment. 2. The second point is that once you have decided to store your feed collection centrally and also the Read/Not Read status for each post then you are very close to deciding to store the posts themselves centrally. I'm not sure about this although it seems to be a logical conclusion. 3. I don't understand the business model of del.icio.us sufficiently to understand how you could finance a feed.licio.us equivalent. I hope these are mere details :-)  If we build it they will come, as somebody once said. p.s. This post arose from a conversation with Adam Tennet

Corporate feed aggregator revisited

I should clarify something I said in a previous post about the ideal corporate aggregator. I said it could not be browser-based and when I said that I had in mind a very specific architecture that would not work in a corporate environment. Thinking about this some more there are of course ways of implementing a personal aggregator with a browser interface. My objection was to feed caches that simply subscribe on behalf of their users to a set of feeds then serve them using HTML. This cannot work in a corporate environment because the server is either subscribing only to public (non-secured) feeds or supplying a general set of credentials on behalf of the entire user population. You could implement a corporate aggregator with a browser interface if you designed it so that it was in effect a virtual personal aggregator application sitting on a server. Each user would have his or her own credentials held on the server, and for secured feeds they would need their own personal storage of that content. Public content could be shared among all the users and this might cut down on the overall storage requirement across the corporation. How many times do you need to store the same Boing Boing posts? So, I will reiterate my seven requirements for an ideal corporate aggregator: 1. It must be able to supply to a secured feed the personal credentials of an individual user. 2. It must support all the authentication mechanisms used in the corporate environment: digital certificate, Kerberos, NTLM. Any more? 3. It must be able to be managed centrally by the support department. This means managing a user's portfolio of feeds for him. There are three reasons for this: (a) there are users who will not want to manage their own feeds at the outset as the technology is new to them, (b) there will likely be a standard set of feeds that all users must consume (corporate bulletins and the like) and (c) when the aggregator is upgraded or replaced we will need to preserve each user's feed collection during the process. These actions need to be scriptable in some way. 4. The Read/Not Read status of any post in any channel must follow the user around. Corporate users change locations frequently, they work from home, they make VPN connections from internet cafes. Whatever the situation they need to see the same feed collection and the same unread posts. 5. The user interface must be sufficiently rich to make consuming a large number of feeds efficient. This is another challenge to browser-based aggregators which currently do not support single-key reading. However carefully you select your channels there will always be a certain amount of noise in amongst the signal. You need to be able to move rapidly through irrelevant posts to give you more time to read the relevant ones. 6. On the same subject of signal-to-noise ratio, there are channels that frequently have useful content so you need to subscribe to them but it is mixed in with the personal bugbears of the author. Some feed publishers will provide a feed of certain categories of post so you can filter out the noise this way, but this means you miss any interesting new categories the author introduces. What I need is a way of subscribing to all posts except those which mention Unicorns for instance. The aggregator must support filtering. 7. We need to get away from the arrangement of feeds into a folder hierarchy. I want to tag my feeds with arbitrary categories. I want to be able to Mark All As Read in any given category if I am pushed for time. Der. I can't believe nobody has done this yet. Tags not hierarchy! Somebody tell me they are working on this, please.

The user is always right (curse them)

I love this result, and the methodology behind it: The user is always right (curse them) They made two versions of a home page and served a random one of the two to each visitor to the site. Then they measured how many people clicked through the home page to the other services on the site. The simple, clean design actually lost out to the more cluttered version. Looking at the page you can see why: the "cluttered" version tells you what the site is all about and how it works. The "clean" version is a bit mystifying - you don't immediately know why the page is there. The site is HearFromYourMP and you can see the different versions of the home page by clicking the refresh button on your browser. The point is about measuring effectiveness. We cannot improve without data on how we are doing today. Building measurement into the process tells us what we need to improve and how successful we have been in doing it. Over several iterations the effect compounds itself. My learned colleague ConfusedOfCalcutta calls this "ready, fire, aim". The point (for me) being that the first "fire", while it might be wide of the mark, at least gives you a calibration point for the first "aim". After several "fire, aim, fire, aim" cycles you should be hitting the target. I will leave you with a story of his on a similar subject: “Don’t ask them what they want, they don’t necessarily know. Show them a Ford Escort and ask them what’s wrong with it. And keep improving it. Quickly”. The next step is to build some monitoring into the Escort's effectiveness and use that data, not just the user's opinion, when you come to improve it. (from BlackbeltJones/Work)