Caterpillar uses Snowflake technology to create a consistent view of its data
Manufacturing giant Caterpillar is using a digital platform that includes Snowflake technology to make it easier and quicker for the business to create data-led services for customers.
Brandon Hootman, Director of Digital Data for the company, which designs, engineers and sells machinery and engines to customers around the globe via a dealer network, says success for his team is all about exploiting the huge amounts of data the company holds from this network in order to help customers be more successful:
The aim is that from the moment someone becomes a Caterpillar customer, whether it’s their first asset or their hundredth, that they are able to do the work that they need to do with that product more efficiently and more effectively, with more uptime at the highest level of quality. And our applications are focused on enabling that.
Hootman’s team is busy creating a range of apps, such as in fleet management, that allow customers to monitor and manage their assets in fine detail. From availability through utilization and service scheduling to maintenance, the objective is to create a joined-up approach to data-led services. Hootman says his company’s cloud-based digital platform plays a crucial role in producing insight for customers:
A lot of the work that we we've done with data from a processing standpoint is ensuring that we get the basics right to be able to support consistency of data. It’s about ensuring that – as you move across our digital properties – the experience is consistent and we're showing the same data to be able to support it.
There are some areas where the company builds systems in-house. Data pipelines, for example, are seen as a core element of the company’s long-term business proposition. In other areas, the company draws on external expertise – and when it comes to the underlying data lake, Caterpillar works with Snowflake. Hootman explains:
We've invested a lot within digital in terms of cloud-native software engineers that can really help us from the position of core cloud services and building and maintaining that platform. There's a lot of complexity and eventing that happens on the back end. Snowflake is an important part of what we're doing because, as we get data into a usable state, that's when we make information available in our data lake.
Long-term planning
Caterpillar started working with Snowflake in 2017. After proving its worth in some initial use cases, the cloud-native nature of the technology – plus its evolving collaborative capabilities – meant it proved a good fit with the firm’s long-term requirements, according to Hootman:
In the early days, the appeal was that it was a well thought-out, cloud-based data warehouse. The platform has really evolved since then. One of the differentiating capabilities that we really saw as benefiting us was the logical data sharing. As we're building independent data lakes, being able to logically share that data back and forth without having to fully do an unmanaged replication is key.
The company was keen to avoid some of the common pitfalls of extract, transform and load (ETL), where new business cases can lead to a replication of existing data sets. The firm’s enterprise data strategy is focused on developing consistent standards and principles. The Snowflake data lake forms a crucial element in helping the company’s staff to share data seamlessly. Hootman explains:
If you have unmanaged replication of data, you just you lose control of what's happening with it, how it's being used, and the insights that are being inferred from data. Snowflake has kept us from feeling like one team has to go and solve every data problem. We have now empowered these other functions, so they can go bring the data, build their data objects, put the data in a data lake, and then seamlessly and logically share that data back and forth.
Hootman advises other digital leaders to ensure their teams bring the right workloads to data. While Snowflake technology forms a key part of the company’s digital platform, Caterpillar was also keen to take a polyglot approach, where it can use the best technology for the right use case, whether that’s based around events or online analytical processing (OLAP):
Anyone in a data management role will tell you that getting data into these platforms is relatively easy, and then you just have to be really conscious of the runtime and workload you put on it. One of the reasons that we've got a polyglot platform is that we're not taking the wrong workloads to different technologies and we can really optimise that platform depending on what the type of workload is. So, if the workload is more event-based, we've got a message bus to be able to support that. If it truly is more of an OLAP-based workload, we're able to take that to a technology like Snowflake and really, really leverage it.
The cloud-based nature of Snowflake will be crucial moving forward, he adds. As well as helping Caterpillar to create a consistent view of its information assets, the technology will help the company to run data-intensive processes that support the development of innovative services for the company’s customers:
As we move into more of our analytics-based use cases, Snowflake’s playing a role in supporting that. So, as we look at more things like determining when we may have assets that need service, determining what the optimal service will be on those assets, and being able to have a unified environment where our data scientists can work, it has really helped us to bridge that gap.
The end result of his team’s data-integration efforts, says Hootman, is a digital platform that provides a consistent approach to data at the back end and that allows the business to develop new applications for the firm’s customers at the front end:
We're really excited because we're asking a lot of these digital properties that are much more of a digital ecosystem. We've been really clear that digital is a core enabler for what we do with our customers. We want our customers to be more successful with Caterpillar products than any other products in the industry. We believe that the digital offerings in our ecosystem complement that approach very well.
Gloss