Tuesday, March 1, 2011

Rolling Blogger-outs?

If you're seeing the "Blogger too busy" screen a lot lately and wondering what's going on, here's the deal: There are two web servers talking to one database (there are also three other servers doing peripheral stuff, like transferring files, but they're not a problem right now). Prior to getting a new server, there was one main web server, and it was the bottleneck. We added another web server, which we set up to equally share the load. This dramatically sped up that part of the chain. However, the two web servers in combination are now sending so many requests to the one database, it has become the bottleneck. (This is why, among other things, when you load the home page sometimes lately, it will appear as if you don't have any blogs. If this happens, don't panic. Just try reloading.)
We still have a couple more servers sitting here that we can spread some of the load to, but it's a lot more difficult to distribute database requests than web server requests. We know lots of things we can do to distribute the database, as well as make it much more efficient. And I'm working on those, but it hasn't gone smoothly, and some of them require me to rewrite some code, which is just time-consuming to get right. So, while I'm doing that, when the database gets too overwhelmed (which is more and more frequently), I'm switching off one of the web servers for a while. Half the people will still get in, but half will get the "Blogger too busy" screen. I know this is frustrating, but hopefully it will be less frustrating than it being dog-slow and timing out for everyone. It's sort of like the rolling black-outs here in California. When it happens to you, you know it's only going to be for an hour, and you can hold off on your 'lectricity needs until then. Hopefully, I won't have to do this for more than a couple days. We did all the quick-fixes, but there's a medium-fix I should be able to do today. However, Zoso (the db server) isn't the only one who's overwhelmed these days, so with me luck. :)

No comments:

Post a Comment