There’s a rapid transition and expansion happening within the data ecosystem today.
Thanks to flexible pricing models and scalability, every company can now own and control their data.
That’s given rise to the modern data ecosystem. And the new and growing ecosystem is fundamentally changing the modern data team—its responsibilities and how it exists within organizations.
Indicative CEO Jeremy Levy hosted a webinar with Hightouch founder Tejas Manohar and David Krevitt, Head of Acquisition for dbt about how the modern data stack is shifting data teams.
You can watch a recording of the full webinar right here, or keep reading below the video for a written summary.
What Is the Modern Data Stack?
In the data world, we’re moving toward a new infrastructure that enables companies to own and control their own data in the cloud. There’s consensus around that being the way forward for modern data tools, with cloud data warehouses (CDWs) at the center and a whole ecosystem of tools (like Hightouch, dbt, and Indicative) springing up around them.
That marks a sea of change for the data industry. Whenever an area of technology goes through a wave of change like we’re seeing, tons of new tools and technologies pop up.
While that’s ultimately great for innovation, it can make it hard for consumers to figure out which tools they need and how all the pieces of their stack should fit together.
The modern data stack serves as a guide to help everyday companies conceptualize how all these solutions actually fit together—how they can work together as a toolset to solve problems.
What’s Driving the Rise of the Modern Data Ecosystem?
Data warehouses aren’t new—they’ve been around for years. So what’s changed to make cloud data warehouses the standard for nearly every company?
As Tejas notes in the webinar, the change comes down to two things:
- Accessibility
- Maturity
The CDW came into mainstream popularity because it’s much more accessible. Despite early efforts by Amazon RedShift, it wasn’t until BigQuery and Snowflake came into the market that data warehouses got a lot easier to administer.
They were finally accessible for the average company, and that helped standardize the data setup for a majority of the market. Once you have a standard configuration that exists across a large number of teams, it’s really straightforward to integrate with—and build around—it.
So the maturity and standardization of the CDW unlocked a whole world of new possibilities, use cases, and features. It’s gotten much easier to work with data warehouses, and the skills required to own and control your data have been drastically reduced.
That’s what spurred the modern data ecosystem.
Even legacy functions are changing. Ingestion, transformation, business intelligence (BI)… they’ve all started to change.
The Modern Data Team
All that change and innovation in the data ecosystem is inevitably changing the way data teams work within organizations.
As CDWs have gotten faster and eaten up more data—and as that data has gotten cleaner—there’s more incentive for people across the org to learn SQL and dig their hands into the data.
That’s led to two things:
- Easier to use data solutions
- More technical people throughout the org
From sales to customer success, people across the org are getting more technical. They’re writing SQL and administering their own data solutions, without hands-on help from the data team.
The barriers to entry are lower. Data is more accessible and democratized than ever.
So what’s the role of the modern data team?
According to Jeremy, Tejas, and David, today’s data teams are able to focus on more holistic approaches to organizational problems. Their role now centers around:
- Enabling self-service data access
- Implementing tests and quality control
- Ensuring business users can access and trust their data
Analytics Engineers, David presents as an example, serve as the “glue,” ensuring knowledge (in the form of data sets) is disseminated evenly throughout an org to all the people who need it.
Will Orgs Even Need a Data Team?
In short, yes.
According to Tejas, about 80% of Hightouch buyers are still situated within the data team, and a centralized core data engineering team (who manages tooling, infrastructure, and automation) is still the norm for most orgs.
With use cases for data and downstream tools growing in number, there’s actually more need for a centralized data team—because so many teams are affected by changes to the data stack.
The Role of Data Engineers
The goal for vendors like Indicative, Hightouch, and dbt is to reduce the number of people companies need in data engineering and freeing those people up to get closer to business problems.
David expects to see more data engineers who are embedded within departments, too. These engineers will spend more time meeting with the people who ultimately consume data within the org. That way, they’ll be able to develop a better feel for what those end users need to get out of their data.
Because, as Tejas points out, SaaS tools can’t solve for every use case across every company. For example, your average SaaS can’t build an integration with every company’s internal CRM.
Instead, the data ecosystem can help solve 90% of use cases—freeing up data engineers to focus on customization and solving those last 10% of problems that are unique to their company.
The Modern Data Team’s Role in Democratizing Data
Somewhat counterintuitively, Jeremy has seen some pushback cropping up—that companies don’t want to democratize data access because it opens up the potential for messiness.
All three agreed that, while the endpoint of democratization is ultimately great, it may be a messy road.
As David pointed out, “The flipside of democratization is bottlenecks, and no one enjoys those.”
“The modern data stack is materializing,” Tejas said, “but it’s also still being iterated on.”
Ultimately, it’s the tools within the data ecosystem that will have to uplevel their solutions to solve for maintaining data clarity and quality as access expands.
“At the end of the day,” David said, “we need people. We still need data practitioners to uphold a high standard as that democratization is taking place.”
What Lies Ahead for the Modern Data Stack?
Both Tejas and David are looking forward to developments in speed as the next frontier for the modern data ecosystem.
David sees real-time analytics warehouses, in combination with tools like Hightouch, that allow you to move data out to operational tools with very low latency.
He envisions the data ecosystem to revolutionize data in the way Zapier changed the game for automation: total democratization in real-time.
Tejas agreed, noting that innovation around the data warehouse tends to happen in waves—waves triggered by new technology from the data warehouses themselves. He sees the next wave being spurred by advancements in the speed with which data moves into the warehouse.
“When we can have that through a technology that most customers have adopted,” Tejas said, “that will create a huge wave of new features, new players, new technology, new use cases for the data warehouse and for the data team.”
Where to Go Next?
Looking to get started on democratizing your data? Look no further, and start a free 30-day trial with Indicative – no credit card required.
Alternatively, if you aren’t ready to create an account – we offer public access to the PetBox data set. PetBox is a fictitious company that sells subscription boxes of commonly needed pet supplies. Jump into this public demo and grab a quick feel for Indicative while analyzing all the data PetBox has to offer.
Indicative Team
