How to use mobile phones sensors to reorder day to day products

The camera’s untapped potential

Pondering how sensors could be used to crack our brief led us to think about cameras.

They are, after all, the most familiar digital sensor. With one or more on every mobile phone sold today, it’s hard to imagine the world without the ever-present snapping, Instagramming and filming that these connected cameras have enabled. Their inclusion in smartphones is also a major factor in the dramatic reduction of their price, whilst constantly pushing quality higher and higher.

But digital cameras, whether inside your smartphone or otherwise, can be so much more. Emerging computer vision techniques, coupled with advances in machine learning, are making the light sensor in mobile phones one of the most versatile tools in the sensor box.

Computer vision techniques can already help us rapidly detect text, colours, shape and motion in a field of view. With new machine learning APIs, it will be possible to make these sensors more intelligent over time, improving their ability to pick out text, or even particular brands, and to rapidly identify what’s passed in front of the lens.

With these thoughts in the back of our mind, we set out to explore and test the ways in which this most familiar of sensors could help us crack our brief.

The concept

It was an idea that was simple on paper, but difficult to execute: if we could train a light sensor to automatically detect which products pass by it, then we could get a log of all of the packaging that a household recycled.

The data captured by the mobile phone sensor would provide a goldmine of valuable information for retailers and manufacturers. A customer indicating that a product has been exhausted could not only prompt a quick and easy reordering system, driving sales, but it could also provide a way to really fine-tune packaging volumes and sizes, reducing waste and cost in the manufacturing process. Similarly, for new product variants, it’s a great opportunity to capture consumer feedback about what they liked, or disliked, about the product.

For consumers, instead of branded push buttons all over the house, the process of reordering everyday goods becomes ‘invisible’ and discreet. To tackle the challenge with recycling, integrating the experience with a loyalty scheme would allow retailers to positively reward customers for recycling by awarding them points, and by tracking the amount of total space saved from landfills.

Saving the earth, saving money and saving time reordering everyday goods seems like a much more well-rounded proposition than just the push button reordering of a single, specific product.

The prototype

Given the time constraints, we didn’t focus on making something pretty. Instead, our attention centred around training the light sensor in mobile phone cameras to pick out packaging that ‘fell’ in front of it, logging the products as it did so. This, in turn, triggered a push notification relating to what had been thrown into our makeshift recycling bin.

Working with our fellow hackathon team members, who joined us from Superdrug, we also explored and demonstrated how this could be integrated into a mobile ordering flow for the retailer.

Technical investigations in this area are an ongoing part of our R&D efforts as we look to refine this proposition and make it more accurate. It is clear that we are just beginning to uncover the opportunities that can come with harnessing these connected light sensors.

Our hackathon team came back to Kin + Carta HQ with even more ideas about the potential of using light sensors in mobile phone cameras more in our everyday, connected world - so stay tuned for more updates in the future.