The system consisted of a barcode reader, a text recognition system with OCR, and a cover image matcher that could identify books covers, DVDs, CDs and games.
A real-time barcode detector runs on live video stream allowing the barcode to be tracked even while the camera is in motion.
The text reco system allowed users to pick a few words quickly from a photo with just a few taps.
When a word is tapped, the word is put directly into the search box and zooms the photo to the center of the screen automatically thereby avoiding the fat finger problem for subsequent selections.
Bing Camera
A clever camera that can recognize cover art, text and barcodes simultaneously.
Camera as search
Added the ability to use a camera as a search input for Microsoft’s Bing app. The app’s detectors used computer vision algorithms to read barcodes, match cover art, and do optical character recognition (OCR). In collaboration with engineering team, scoped the project, ensured it matched the rest of the Bing app product architecture, and established new user experiences.
Multiple detectors
Conventional cameras at the time had separate apps or modes for detecting different content types (bar codes, cover art or OCR). After some experimentation, our team worked out that the app could run multiple detectors at once so that users did not have to make that choice. Instead, the camera automatically indicated all types of content it could see. As an example, the detectors could spot, highlight and decode barcodes in the frame while also matching cover art.
Text selection from images
A second challenge was establishing the user experience that would enable users to capture text from an image, convert those pixels into text characters, select some words, and put them into a search box. Understanding the constraints of the problem and the underlying technology was critical. Created After Effects motion and interaction studies to explore and define the new user experience. Worked closely with program managers and engineers to spec and execute the design.
History
As a historical note, these features were launched as part of the Bing Mobile app for IOS in November 2011. Apple added a similar set of capabilities to IOS 15 almost 10 years later in September 2021.