When the database can't keep up

04:30 PM - 05:25 PM on July 17, 2016, Room CR7

Adam DePrince

Audience level:

intermediate

Description

An overview of Peloton's custom leaderboard database.

Abstract

When postgres couldn't keep up, redis ran out of room we found answers in an unusual place. Going "back to the basics" and building a very simple database in python using zerorpc, and operating directly on the file system using plain posix calls provided a surprisingly robust and scalable solution for us. In this presentation we'll review the use case of Peloton's leaderboard server, the specific operations we require and how a combination of simple reads, appends and memoization allowed us to achieve surprisingly high levels of performance for our application.