Kris (Backup Account)
Availability data now changes so fast and is so large, that we have to implement "CQRS" (which is java architect speak for materialized views)
The AV Queue workers overwhelm MySQL 5.5 with their writes, we need larger redo logs (limited to 4 GB) and the first SSD machines.
In May 2012, we push fastav (the read-view of av) to MySQL 5.6-alpha in production, on SSD hardware. MySQL implements a number of changes for us, based on our experience.
Boom! That works, we do not die in summer 2012.
Kris (Backup Account)
We move away from DRBD and to filers, but only until GTID is invented.
Replication between data centers becomes a thing, which means replication hierarchies are generally at least 3 deep, introducing intermediate primaries.
All replication delay measurement breaks and we move to heartbeat tables for end-to-end replication delay metering.
The Perl libraries pick up heartbeat information and moderate bulk writes by measured repliction delay automatically.
Kris (Backup Account)
The teams objective was "to be able to move 95% of all servers to a new version of MySQL within 20 work days/1 calendar month, no matter the number of servers."
We did not meet that goal at all at that time, and it took a long time to get there.
Other goals were "downtime free primary switches" (with GTID), which enables and also gets you "downtime free ugrades"
Kris (Backup Account)
And we have that really botched change from 5.1 to 5.5 that drags us back.
And we have trouble finding people, because what we do with MySQL is at a level where it is not easy finding people that understand our way of thinking.
Kris (Backup Account)
2013 things exploded, for a numver of reasons.
For example, MySQL can serve a "memcached"-like query "SELECT * FROM t WHERE id = 17" in 125µs if the data is in memory.
It scales better than memcached, though, can automatically prewarm on start, and unliked memcached runs well on machines with many cores and much memory.
The cost of memcached and MySQL on bare metal are the same.
We drop memcached, gain a LOT of replicas.
Automation is really, really necessary.
Kris (Backup Account)
Parts of database provisioning are already automated.
There is a git that is checked out into every database, collecting all the scripting and tooling that DBAs use.
A lot of things are half-automated.
The DBA team begins to also move to Python, and starts to work on B.admin, automated database operations.
GNU social JP is a social network, courtesy of GNU social JP管理人. It runs on GNU social, version 2.0.2-dev, available under the GNU Affero General Public License.
All GNU social JP content and data are available under the Creative Commons Attribution 3.0 license.