Reddit’s database has two tables (2012)

Steve Huffman talks about Reddit’s approach to data storage in a High Scalability post from 2010. I was surprised to learn that they only have two tables in their database.

Lesson: Don’t worry about the schema.

[Reddit] used to spend a lot of time worrying about the database, keeping everthing nice and normalized. You shouldn’t have to worry about the database. Schema updates are very slow when you get bigger. Adding a column to 10 million rows takes locks and doesn’t work. They used replication for backup and for scaling. Schema updates and maintaining replication is a pain. They would have to restart replication and could go a day without backups. Deployments are a pain because you have to orchestrate how new software and new database upgrades happen together.

Instead, they keep a Thing Table and a Data Table. Everything in Reddit is a Thing: users, links, comments, subreddits, awards, etc. Things keep common attribute like up/down votes, a type, and creation date. The Data table has three columns: thing id, key, value. There’s a row for every attribute. There’s a row for title, url, author, spam votes, etc. When they add new features they didn’t have to worry about the database anymore. They didn’t have to add new tables for new things or worry about upgrades. Easier for development, deployment, maintenance.

The price is you can’t use cool relational features. There are no joins in the database and you must manually enforce consistency. No joins means it’s really easy to distribute data to different machines. You don’t have to worry about foreign keys are doing joins or how to split the data up. Worked out really well. Worries of using a relational database are a thing of the past.

This fits with a piece I read the other day about how MongoDB has high adoption for small projects because it lets you just start storing things, without worrying about what the schema or indexes need to be. Reddit’s approach lets them easily add more data to existing objects, without the pain of schema updates or database pivots. Of course, your mileage is going to vary, and you should think closely about your data model and what relationships you need.

Update, 10:05AM PDT: It’s worth reading the comments from a current Reddit engineer on this post. Particularly this one:

I’m personally not a fan of using an RDBMS as a key-value store – but take a look at, say, line 60 of the accounts code. Each item in that _defaults dictionary corresponds to an attribute on an account. For pretty much all of those (1) we don’t need to join on it and (2) we don’t want to do database maintenance just to add a new preference toggle. Those points are particularly more important when you’ve got a staff of 2-3 engineers. Sure, reddit has more now – but we’ve also now got a lot of data to migrate if we wanted to change, a lot of code to rewrite, and a lot of more important problems.

The data architecture made sense for Reddit as a small company that had to optimize for engineering man hours. Now they are much bigger and can afford a saner structure. He/she mentions that they are in the process of migrating their Postgres data over to Cassandra, but slowly.

Update, 11:31PM PDT: A former engineer at reddit adds this comment.

There isn’t a “table” for a subreddit. There is a thing/data pair that stores metadata about a subreddit, and there is a thing/data pair for storing links. One of the properties of a link is the subreddit that it is in. Same with the comments. There is one thing/data pair for comments and the subreddit it is in is a property.

Still today I tell people that even if you want to do key/value, postgres is faster than any NoSQL product currently available for doing key/value.

Update, 7:11PM PDT: From Hacker News, it looks like they use two tables for each “thing”, so a thing/data pair for accounts, a thing/data pair for links, etc.

Liked what you read? I am available for hire.

from Hacker News https://ift.tt/PQZTH4k