VIDEO | Pinterest Head of Data Engineering: Your Choice of Technology Depends on the Scale of Your Data

(US and Canada) Dave Burgess, Pinterest Head of Data Engineering, speaks with Robert Lutton, Vice President at Sandhill Consultants and Editorial Board Vice Chair of CDO Magazine, in a video interview about the best practices for modernizing data architecture and the technologies required.

Sharing the best practices for modernizing data architecture at Pinterest, Burgess says the process began with listening to internal customers to find their top pain points. Next, working groups were formed to figure out the next-generation system. The groups included representatives from internal customers such as security, site reliability, and engineering.

Once the current and future requirements were clear, the organization assessed the best software available. The findings were presented to the senior leadership to get the buy-in and move forward with putting the system into production and migrating.

Burgess shares some advice for fellow data leaders regarding the technologies needed to modernize data architecture, saying they must use a cloud service provider to have the infrastructure agility required for computing and storing data. He stresses that the key technologies they choose will depend on the scale of their data. They should also consider the criticality of their service and have organizational expertise and bandwidth.

Continuing, Burgess reveals Pinterest’s technologies. They are:

• Kafka for distributed data collection.
• AWS's S3 to store all data.
• Spark, Spark SQL, and Presto for batch processing.
• Flink for real-time stream data processing.
• Superset on the analytics side for results and dashboards.
• Pinterest’s open sourced Querybook for creating, running, and sharing queries.
• Druid for real-time analytics.
• MySQL to integrate with the online data systems.
• Memcached for caching.

CDO Magazine appreciates Dave Burgess for sharing his insights and data success stories with our global community.

See more from Dave Burgess

Under the Covers, Pinterest Is Just a Massive Machine Learning Engine