Company Updates

Preview: Envisioning the Future of Grocery Shopping with GPT-4V

AI has long been at the heart of everything we do, and we’re always looking to partner with AI industry leaders and leverage the latest AI advancements to deepen the relationship people have with food and grocery shopping. Our early access and adoption of the new vision-enabled GPT-4 Turbo with Vision (GPT-4V) is the latest such example. Today, we’re sharing a preview of what we’re cooking up next: upgrading our existing AI-powered Ask Instacart search feature with this new vision capability. Soon, in addition to being able to ask food-related questions in natural language, Instacart users will be able to use Ask Instacart to convert handwritten recipes and shopping lists directly into digital, shoppable item lists in the Instacart app. The vision-enabled Ask Instacart will decipher ingredients and quantities in a single step so that our users don’t have to do the mental translation and manually look for one item at a time.

 

Here’s how it will work:

  • 📸 Snap: Simply open the photo feature in your Instacart app and snap a photo of your list– perhaps it’s a family recipe for Thanksgiving apple pie lovingly handwritten by Grandma, or a scribbled shopping list of groceries you’ve jotted down throughout the week.  Don’t worry if the handwriting isn’t the cleanest; GPT-4V’s language capability takes into account context from other legible words to help clarify the meaning of any item that isn’t clear.
  • ☑️ Select: Thanks to GPT-4V, the Instacart app will analyze the image, recognize the text, and convert it into a comprehensive, shoppable list of items.  From there, you can select specific ingredients you need, or de-select any ingredients you might already have at home– like cinnamon, in the case of Grandma’s apple pie.
  • 🥧 Savor: With your new digital shopping list, you can instantly place an order and get all of the ingredients or grocery supplies you need delivered straight to your door in as fast as an hour– leaving you more time to savor the fruits of your efforts. 

We hope to launch this feature in the Instacart app to a small number of customers in the next few weeks and more widely in the coming months. This feature is the latest in a set of generative AI-enabled shopping experiences we’re building to help people more easily answer the eternal, “What’s for dinner?” question. From the Instacart plugin for ChatGPT to Ask Instacart, which lets people ask open-ended food questions in natural language– like, “What are dairy-free snacks for kids?” or even, “What should I make for dinner if I have kale and cheddar in my fridge?”–we believe that AI can help us create personalized, inspirational, value-driven shopping experiences that enrich peoples’ relationship with food and the retailers and brands they love. Now with our early adoption of GPT-4V, this new vision-enabled feature promises to be a big help to anyone following a recipe or looking to save some time tackling their grocery shopping list.

JJ Zhuang

Author

JJ Zhuang is Chief Architect at Instacart, where he drives key technology and architecture decisions across all Instacart product pillars and ensures engineering investments are aligned with the company’s long term business strategy. Prior to Instacart, Zhuang was a Distinguished Engineer at Microsoft and led Office365 app technology. He was also a co-founder and CTO of Acompli where he co-created the app that Microsoft acquired and adopted as the modern mobile Outlook. He also served in Chief Architect roles at VMWare and Yahoo! Zhuang graduated from Shanghai Jiao Tong University with a Bachelors degree in Mechanical Engineering.

Most Recent in Company Updates

Instacart’s Cornucopia of Thanksgiving and Winter Holiday Insights

Company Updates

Instacart’s Cornucopia of Thanksgiving and Winter Holiday Insights

As the air turns crisp and the anticipation of holiday gatherings begin, it's time to rediscover the foods and festivities that define the season! According to a recent Instacart survey of over 2,000 U.S. adults…...

Oct 24, 2024
Instacart Expands Presence in Canada with Addition of Sobeys, FreshCo, Farm Boy, and Longo’s on the Instacart App

Company Updates

Instacart Expands Presence in Canada with Addition of Sobeys, FreshCo, Farm Boy, and Longo’s on the Instacart App

Customers in Ontario can now find their favorite Empire Company Limited banner stores, including Sobeys, FreshCo, Farm Boy, and Longo’s, on the Instacart App for same-day delivery in as fast as an hour. Rolling out…...

Oct 24, 2024
Joining Our Table: Meet Chief Technology Officer Anirban Kundu

Company Updates

Joining Our Table: Meet Chief Technology Officer Anirban Kundu

We are thrilled to announce Anirban Kundu will be joining us as our new Chief Technology Officer! Anirban joins us from Uber, where he was VP and Head of Engineering for Uber Delivery. At Uber,…...

Oct 16, 2024