Recognising specific products in images

Wednesday 21st January, 2009

Yesterday Andrew Stromberg pointed me to the excellent IPhone app by image-matching outfit Snaptell.

Snaptell’s application takes an input image (of an album, DVD, or book) supplied by the user and identifies that product, linking to 3rd party services. This is equivalent to the impressive TinEye Music but with a broader scope. As Andrew points out, the app performs very well at recognising these products.

Algorithmically the main problems faced by someone designing a system to do this are occlusions (e.g. someone covering a DVD cover with their thumb as they hold it) and transformations (e.g. skewed camera angle, or a product that’s rotated in the frame)

There are a number of techniques to solve these problems, (e.g. the SIFT and SURF algorithms) most of which involve using repeatable methods to find key points or patterns within images, and then encoding those features in such a way that is invariant to rotation (i.e. will still match when upside-down) and an acceptable level of distortion. At query-time the search algorithm can then find the images with the most relevant clusters of matching keypoints.

It seems like Snaptell have mastered a version of these techniques. When I tested the app’s behaviour (using my copy of Lucene in Action) I chose an awkward camera angle and obscured around a third of the cover with my hand and it still worked perfectly. Well done Snaptell.

Welcome to the image link revolution

Monday 19th January, 2009

The hyperlink revolution allowed text documents to be joined together. This created usable relationships between data that have enabled one of the biggest technological shifts of the recent age… large scale adoption of the internet. Try to imagine Wikipedia or Google without hyperlinks and you’ll see how critical this technique is to the web.

We’re on the verge of another revolution, this time in computer vision.

Imagine a world were the phone in your pocket could be used to find or create links in the physical world. You could get reviews for a restaurant you were standing outside without even knowing its name, or where you were. You could listen to snippets of an album before you bought it, or find out where nearby has the same item for less. You could read about the history of an otherwise unmarked and anonymous building, get visual directions, or use your camera phone as a window into a virtual game in the real world.

A team at the university of Ljubljana (the J is pronounced like a Y for anyone unfamiliar) have released a compelling video demonstrating their implementation of visual linking. They use techniques that I assume are derived from SIFT to match known buildings in an unconstrained walk through a neighbourhood. These image segments are then converted into links to enable contextually relevant information.

When you combine this with other other techniques, such as the contour-based work being done by Jamie Shotton of MSR and you start to see how that future will appear. Bring in the mass adoption of GPS handsets driven by the Iphone amongst others and it’s pretty clear there’s going to be a change in the way people create and access information.

The only questions are who, and when.

Another two handy IPhone apps – Shazam and Stanza

Monday 5th January, 2009

I’ve got a couple more IPhone apps to add to my previous list of apps.


This is the IPhone version of the popular phone service. Just open up the app, tap on “tag music” and it’ll have a crack at identifying whatever music you’re listening to. Seems to work well so far, although I haven’t tested it with anything obscure or in noisy areas.

Conceptually this is pretty close to what we’re working on here at Pixsta, but with audio rather than pictures.


A free E-Book reader.  You can get access to lots of public domain content, as well as select paid content from some publishers. Very handy for reading those stubborn classics on a cramped tube train. Currently reading a Bertrand Russell book that I otherwise would never get round to.

Great Iphone apps

Wednesday 17th December, 2008

Okay, so yesterday I had a bit of a rant about the Iphone’s wooden syncing policies amongst other things. It’s not all bad though, far from it.

Here are some of the great apps I’ve come across so far:


This the Iphone version of Microsoft’s Seadreagon engine, a mechanism for navigating very large images, including nested images to an infinite (or at least unstated) depth. It’s ultimately pointless but geekily fun.


This shows you the current status of the tube lines in London. Fatalists might suggest that this is pointless, but I’m an optimist, which means that I hope one day this will enable me to stay in the pub rather than waiting on a jammed platform on a line with severe delays.

MyRail Lite

This shows you the status of the overground train lines in the UK. It also hooks in to Google Maps to show you where stations are, find your closest ones, and give you directions to them.

Chess With Friends

Does as it says on the tin. Play by mail chess for the touch-screen generation.


The Iphone version of the lo-fi classic Crayon Physics. I’m still waiting for Crayon Physics Deluxe.

So close, yet so far

Tuesday 16th December, 2008

At the weekend I got myself a shiny new Iphone 3G. It was so close to living up to the hype, but has sadly fallen short. Below is a quick list of reasons why I’m disappointed by my Iphone.

Music playback FAIL

Sometimes playing a track does nothing. No sound. No visible activity. When I press the button that takes you back to the previous screen it registers the action but freezes for a while before actually doing as requested. It’s intermittent but it’s happened to me at least five or six times since the weekend.

That’s not really what you want from the second generation of what’s alleged to be an MP3 player.

Itunes FAIL

To start with I’m not a fan of being forced to hook my phone into Itunes in order to get it to work, but I’ve already had an annoying experience because of it. My girlfriend logged me out of my Itunes account and logged into hers, and when I synced my phone with it Itunes silently transferred my phone over onto her Itunes account. When I later tried to install an app on it I was challenged for her password… unable to do anything about it until I got home.

Apple, if you’re going to enforce a strict one to one policy between a computer and a phone then you have to explain more clearly and simply what the rules are, and warn people when they go near the boundary of your proscribed behaviour.

Syncing FAIL

I keep my music library at work, because that’s where I listen to music. I put a few tracks on the Iphone at home to try it out, but other than that I’ve mostly been installing apps.

Now I’ve brought my cable into work, fully expecting Apple to play their annoying little permissions game and wipe those tracks off when I synced with my work computer… but no. Apparently it also wipes the apps I’ve downloaded. I don’t know whether I’m more shocked that it wipes the free apps, or that it wipes the apps that I’ve actually paid for.

When I go to re-install the wiped apps, the app store tells me I can re-download them for free, so why didn’t it check I had permission to use them before it wiped them? Thanks Apple. Very thoughtful of you.

Apple, you’ve managed to make an amazing product and then snatch mediocrity from the jaws of victory with your disappointingly wooden approach to synchronisation and multiple devices. Nicely done!