Beyond the app servers, we had our database. Now, we just talked about how funky a database
setup we had, but it was still Postgres, and we’ll still use the industry-standard
cylinders for representing our databases, and we had a couple of verticals–
I think we stored links on one, votes on another, comments were on one,
and then we had like a miscellaneous one. Did we store subreddits and users on
the same one or something like that? Subreddits were with comments,
users were with links. When we first built the system,
it was all on one machine, and we had different tables for each data type. And so a natural way to grow was
to split apart the data types, because remember, I talked about how
if you don’t do joins, scaling is a lot easier– This is an example of that. You have to think a little bit harder about
what your queries are going to be, but you can start splitting things apart. You know how many comments are
in the comment’s database now? Something in the order of like 250 million.
>>250 million comments. When I left, I don’t remember how many. There was a lot and it wasn’t fitting
on one machine with links. All these databases were replicated.
We used Londiste, I think. Do you guys still use Londiste? Yes we do.
Londiste is a third-party tool made by Skype. Basically what this does is–
I think it is a log shipping right? No, so when you write to the master, it hits
a trigger, which then inserts the same query into a queue and that’s replicated to all the slaves.
The same inserts will run all the slaves. Okay. So, it just actually replays
the exact commands Yes. Obviously, things like random
get precalculated. Okay. So all these machines were replicated
using Londiste, and Londiste is really cool, and it might be a Londis-te, I don’t know. It’s a problem with web technologies is right, you
read about the technology, like I don’t know. Some people I think called this HAProxy,
but it’s the first guy in the office who discovers the piece of technology, he gets this is
how it pronounced for the rest of the company. We replicate all of this machines both for durability, and also if we lost the machine,
we wouldn’t lose our data. In Reddit, if we lost your comment
history, you’d be mad. And for load, which spread our
reads across all these machines. Now, remember we talked about replication lag,
and I learned this lesson the hard way and it was really hard for me to figure out
the first time because I’ve never heard of replication lag, and we had to
really subtle bugs as a result.