The University of Adelaide's Australian Centre for Visual Technologies in collaboration with the Oxford Brookes Computer Vision Group have released a demo video and paper describing their VideoTrace software.
This is rather like Sketchup Photomatch on steroids! They are currently looking for commercial partners to turn this into a product. It looks very much like the sort of thing Realviz might be interested in (in which case it will be rather expensive), or hopefully even Google.
I'd like to see a demo of it modeling an urban street scene, which I'd imagine would give pretty good results.