Recognising specific products in images

Wednesday 21st January, 2009

Yesterday Andrew Stromberg pointed me to the excellent IPhone app by image-matching outfit Snaptell.

Snaptell’s application takes an input image (of an album, DVD, or book) supplied by the user and identifies that product, linking to 3rd party services. This is equivalent to the impressive TinEye Music but with a broader scope. As Andrew points out, the app performs very well at recognising these products.

Algorithmically the main problems faced by someone designing a system to do this are occlusions (e.g. someone covering a DVD cover with their thumb as they hold it) and transformations (e.g. skewed camera angle, or a product that’s rotated in the frame)

There are a number of techniques to solve these problems, (e.g. the SIFT and SURF algorithms) most of which involve using repeatable methods to find key points or patterns within images, and then encoding those features in such a way that is invariant to rotation (i.e. will still match when upside-down) and an acceptable level of distortion. At query-time the search algorithm can then find the images with the most relevant clusters of matching keypoints.

It seems like Snaptell have mastered a version of these techniques. When I tested the app’s behaviour (using my copy of Lucene in Action) I chose an awkward camera angle and obscured around a third of the cover with my hand and it still worked perfectly. Well done Snaptell.


Twitter API Fail

Thursday 8th January, 2009

I was going to start this post with an “I told you so”, but have realised that my post about how Twitter’s 3rd party API stupidly requires your user password is still in my drafts folder from November. Oops.

Anyway, the punchline is that apparently one of Twitter’s employees gave away a password that can be used on their administration interface by giving it to a 3rd party service. This naturally has caused a bit of a stir.

I hope this acts as a lesson to both users and service providers… handing out your password to anyone, even someone who looks legit like Facebook, is a BAD idea. If you want an example of a better way to do work, services like FriendFeed enable 3rd party integration using a syndication key (that you can revoke if you wish) that gives them limited access. Another option would be OAuth, an emerging open standard for authorisation.