Kris (Backup Account)
Back then it was Apache with mod_perl in the Apache workers, later changed to a more fcgi-like architecture, and then moving to nginx.
Back then a set of 4 front-end apache servers were talking to a fixed set of database replicas for reads.
Some backend databases were too lightly loaded so they were shared between cells.
4 frontends spoke to 1 bp, one app, 1/2 av and 1/4 res instance.
Kris (Backup Account)
Because if something works, you do it more.
Somewhere along this timeline we celebrate 86400 bookings a day, an average of 1 booking per second over the day.
A booking is typically a small number of roomnights, and the cost of a roomnight varies a lot from country to country, between 40 Euro in India to 250 Euro in UAE.
Today, during peak hours, it's probably 100s of bookings per second.
Kris (Backup Account)
Not just av, we also isolate tables with a time dimension in the primary key or table name into a second chain, dw, "data warehouse" (it's not, but close enough).
These tables have archive-nature, they are only ever appended to, or they contain aggregates of time intervals.
They are also less critical for the "take customers money" codepath.
Soon other sets of tables are identified and also split off.
Kris (Backup Account)
Great success: We isolate a lot of change-rate from the main database, and we also shrink the data size in av through optimization.
We can keep all the working set of av in memory all of the time, only writes reach the disk.
By adding av replicas, we can scale out reads at will, unlimited.
Kris (Backup Account)
Look at the data.
Find tables with a very high change rate, and talking to the Perl devs, learn that these tables are part of a subsystem called "availability".
The tables store hotel room availability, and each time a hotel puts up a room for sale, or a room is sold, they change.
We take out the availability tables, change the data format to make it all smaller, and numeric and put things into a special database hierarchy, "av".
Kris (Backup Account)
Then build a replication architecture and a plan for downtime-free upgrades, and a plan for smooth high availability primary pairs using shared disks (using DRBD with great success).
Get in some nice MSA arrays from HP, spread load over spindles.
Get more memory, keep working set in memory, get rid of disk reads.
Learn that MySQL has a global InnoDB Buffer Pool lock, engage with my MySQL colleagues and ask for a version of MySQL with per-page locks.
Great success, and with 5.0.36 it is GA.
Kris (Backup Account)
So they sat out to move from Postgres to MySQL and completed that just the yesr before I was asked to look into performance and scaling.
They were running on 32-bit kernels on 64-bit machines, using MySQL 4 with MyISAM, using 1 or 2 HDD per machine.
There were a few easy wins – move to 5.0, move to 64-bit kernel, move to disk arrays, use thin wide stripes.
Use memory tables for GeoIP and other busy, static tables.
Kris (Backup Account)
The pre-travel questionaire was weird.
The customer wanted 2-3x more capacity immediately and a plan for 10x within 2-3 years.
That's a nice scaling job.
They were running on Postgres, initially, and just completed the migration to MySQL – mostly, because Postgres had no scaling story worth telling and was operationally hostile.
They tried Slony, and that was broken more often than not.
Upgrades required downtime, and dumping and reloading the database.
Kris (Backup Account)
Today is August, 2nd of 2024.
This week, 18 years ago, I met a customer first time in Amsterdam.
They were a small travel agency, bookings.nl, occupying two floors of Peper en Zout.
The building is iconic, it scored in the list of the ten ugliest buildings in the Netherlands.
http://www.architectureguide.nl/project/list_projects_of_architect/arc_id/79/prj_id/342
Es hat eine Wikipedia-Seite
https://nl.wikipedia.org/wiki/Weteringschans_26-28
und dort
De gebouwen zijn te zien in de film The Hitman's Bodyguard uit 2017, waar ze ironisch genoeg worden weergegeven als gevangenis.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.