Data Engineering and Infrastructure at Instacart with Engineering Manager Abhi Kalakuntla

“It sounds cliché, but data drives Instacart’s entire business—our logistics and shopping processes, as well as our ordering experience, are all based on data,” explains Abhi Kalakuntla, Engineering Manager on Instacart’s growing Data Infrastructure team. Thousands of algorithms inform the data models that make Instacart run—if those algorithms don’t work or our data models are inaccurate, the entire business would be impacted.

Not only does this mean that Abhi and his team have to worry about things failing, it means they take it a step further and assume the algorithms might go wrong and plan accordingly by building robust systems that can handle anything. “Our data infrastructure must be resilient, not only in the case of failure but to deliver a better, constantly improving experience to our customers and partners,” he says. “For instance, when the pandemic hit, the volume of Instacart orders skyrocketed—this meant that the data team had to adjust our models to get faster input from stores, track products and orders better, and make sure Instacart’s capacity could handle the increase of sheer data volume.” 

Because of the critical nature of data to Intacart’s business model, nearly every team at the company relies on Data Engineering, from customers to shoppers and teams across the company, including ads, machine learning, product, finance, catalog, and logistics. 

For example, item availability prediction is powered by data engineering. Instacart maintains a catalog of all of the possible items that partner retailers may typically stock in their stores, but actual, real-time inventory can vary widely from store to store. As a result, at any given time, we might not know the exact quantity of a product that’s available today for a shopper to pick out—but we have AI models to help with that. As shoppers pick up items, the AI model updates and stores that an item has been picked; with more real-time events come more accurate predictions of what is and isn’t in stock. While this process used to take 5-6 hours when Abhi joined the team, they have since worked to get the model to update in under 30 minutes throughout the day. 

Other flows that rely on data include delivery optimization for logistics and routing of deliveries, optimizing picking times for shoppers in-store to decrease the time they spend per order, and item recommendations for customers. What’s more, with constant growth and experimentation with new features, data engineering drives which experiments are conducted and how they’re evaluated. 

The impact of Data Engineering at Instacart

As Abhi sees it, the impact of Data Engineering at Instacart is twofold: the internal effect it has on the team and culture across Instacart and the external effect it has on the product and customers.

Internal impact of Data Engineering

• Speed: It’s faster and easier for team members across the company to find the data they need with solid data engineering foundations.
• Visibility and alignment: Teams are more aligned and have more clarity on how the company is doing with data insights. 

“For example, if a Product Manager needs a data set or metric and data science takes two weeks to get them that information, that’s two weeks of wasted time,” he explains. “With an efficient system, a PM can get the results of their query within a day, which enables them to work quickly. Waiting on data to move on something slows everyone down. When we move faster and data sets are more accessible to those who need them, we see compounding effects across the organization.” 

External impact of Data Engineering

• Improved experiences: Teams are able to make predictions for and optimizations to Instacart’s product much faster and on a larger scale. This improves the experience for customers, ads partners, and everyone on the shopping and logistics side. 

Data Engineering Teams at Instacart 

There are three different teams across Data Engineering, each focused on a different area of the function: 

1. Batch processing: this team runs big queries on large data sets and manages the processing around them
2. Realtime processing: this team focuses on sub-10-second processing to optimize many systems across Instacart 
3. Scheduling and orchestration: this team is focused on building, providing, and maintaining the tools for others to build their pipelines and data assets

The power of simplicity in Data Engineering at Instacart

“Instacart isn’t the only company handling large swaths of data, but I do believe our unique approach to data engineering sets us apart,” says Abhi. “One of our core principles is to weigh simplicity and impact and find solutions that are self-serve and simple by design. It’s how our Data Infrastructure team stayed so lean for so long—our core system design was functional, simple, and impactful.”

One project Abhi is proud of is how he and his team simplified the scheduling process at Instacart. The job scheduler runs about 30,000 jobs per day with 7,500 unique jobs per day on one single server and one database. The reason it works at such a scale without any resources is that they’ve simplified how jobs run and allocated set resources for each job through groups. “Basically, we created a glorified chron with a couple of dependency managements built in,” Abhi explains. 

For simple flows, they put an SLA on when the job should finish and how frequently it should run. For complex flows with more jobs, they run in order of a pipeline and branch out, thereby introducing complexity because there’s interdependence between jobs and the team could end up managing a lot of state. So the team decided not to manage any dependencies, except at the beginning of the day. “If something has to run that day, everything downstream can run that day whenever it wants to and as frequently as it wants to,” Abhi says. 

This method of running scheduling does two things. It forces the system and each job to be very lean, so thousands of jobs can run without complex dependency management. It also forces people to not think about the scheduler when they’re writing their jobs—instead, they think about how their job works with the data because there’s no guarantee that it’ll run at a certain time. As a result, jobs have to be resilient. 

“This system and the outcomes it forced gave the team’s data sets more integrity because people build in resilience to the system while also decreasing the computational costs of the scheduler,” says Abhi. To go alongside this system, they also built a complementary system to optimize landing times by predicting when something should start for other dependencies to go. For example, if Job A needs to run at 8 am and it’s an upstream dependency of Job B that needs to run at 9 am, but we know that job A takes 4 hours to run, the system will shift Job A to 4 am so that Job B will run at 9 am. This helps with maintaining and means there’s no hand holding for when jobs should run. If you set the number of jobs, it’ll run—the rest is freeform. 

“When the pandemic happened, nothing went wrong with the scheduler because it works,” Abhi says. “It’s an example of how powerful simplicity can be.”

Our data engineering tech stack

• Apache Flink
• Kafka
• Spark
• Airflow
• DBT
• Clickhouse
• Delta Lake
• Snowflake
• Postgres
• Trino
• DynamoDB

Abhi’s team works mostly with Kafka as its main data layer that powers real-time systems. He’s constantly thinking about the developer experience across Instacart: how do they interact with his team and is there tooling available for them to write an app or run data analysis quickly? “We want to provide the right tools to power the ecosystem so that minimal manual intervention is required and other engineers feel empowered by data—not like we’re a bottleneck,” he says.

The future of Data Engineering at Instacart

Looking to the future, there are no shortage of interesting DE challenges to solve at Instacart around speed, effectiveness, and connection. 

Speed: While the team used to crunch data a couple of times a day at set intervals, there’s a need for more real-time data now, so Data Engineering will focus on speeding up their systems. “We’re moving to a fast system where latency is important to the core business—ads are real-time, our machine learning models are improving, and our business model needs data that can inform the buyer and shopper experience,” Abhi explains. “That’s an exciting challenge and a huge opportunity for impact.”

Effectiveness: The batch processing team’s near-term focus is finding better implementations on crunching data as it grows, as well as doing so cheaply, effectively, and with a rapidly growing team. 

Connection: The scheduling and orchestration team is focused on lineage and how Instacart’s data is connected how to best connect real-time data with data at rest, and improving pipeline efficiency. 

“There are so many challenges to solve and improvements we want to make,” says Abhi. “Data is playing an increasingly important role at Instacart and our team is striving to optimize for impact. It’s not a question of what is there to work on, but what to work on in order to make the greatest difference for the organization.” 

Ultimately, the future of Data Engineering at Instacart comes down to the fact that it moves the business forward: “We’re here to make data at Instacart better so our business can continue to grow—it’s a simple engine, but an incredibly powerful one,” says Abhi. “I’m looking to build solutions for the next 5, 10, and 15 years, not just to get new systems out the door. I believe in the fine line between shipping things quickly and shipping the right things with speed—and I always strive to optimize for the latter.”

Since joining Instacart in 2018, Abhi and the entire Instacart team have seen the impact data has on our product and company—and it’s only growing. We have a lot to build and the entire Data Engineering team has the chance to make an outsized impact on our product, customers, and business—it’s an exciting time. 

Interested in joining Abhi and his team? Check out our open positions.

Instacart

Author

Instacart is the leading grocery technology company in North America, partnering with more than 1,400 national, regional, and local retail banners to deliver from more than 80,000 stores across more than 14,000 cities in North America. To read more Instacart posts, you can browse the company blog or search by keyword using the search bar at the top of the page.