We’re too Popular!
David ShorthouseYou may have noticed that the EOL site has been flaky at best since approximately 12 EST this afternoon. Although we are serving the site from a load balanced cluster of several machines, we are experiencing phenomenal loads.
I just churned through the web logs from web machines in this cluster and there were 5.8M hits in the span of 3 hours. Most of these happened within 1 hour. We were down (and continue to experience intermittent access) for a few hours, then flipped the machines back on. Since then, there were an additional 5.7M hits, totaling 11.5M hits since 9AM this morning and it is now 2:45PM here. Wow!
We are working hard to resolve the issue so stay tuned and please have patience! I’ll post updates here as the day progresses.
Update Feb 27 @ 11:45AM EST:
In the first 24 hours (minus the approx. 3-4 hours we were completely down) there were:
18.5M hits, 13.3M page views, and 940GB of data transferred.
February 27th, 2008 at 4:12 am → don't worry guys!! nice work in eol!! Read it ↓
don’t worry guys!! nice work in eol!!
February 27th, 2008 at 5:28 am → I'm looking forward to see!! Read it ↓
I’m looking forward to see!!
February 27th, 2008 at 5:56 am → Great work, guys! Hope to see that very soon. Read it ↓
Great work, guys! Hope to see that very soon.
February 27th, 2008 at 7:43 am → Hi, Good work! Hope to see that very soon. I will post information about Uzbekistan animals Read it ↓
Hi, Good work! Hope to see that very soon. I will post information about Uzbekistan animals
February 27th, 2008 at 7:45 am → Good idea, Mila. I will help you! Read it ↓
Good idea, Mila. I will help you!
February 27th, 2008 at 12:42 pm → Tremendous effort! I can't wait until your site works! This may be preaching to the choir, but ... Read it ↓
Tremendous effort! I can’t wait until your site works! This may be preaching to the choir, but perhaps you guys and gals might have planned for more scaling and content distribution before eol.org was so widely announced? In 2008, is round robin DNS with 4 IP’s the best that WHOI can come up with? Back in the nineties moderately busy sites frequented mostly by consumers with dial-up accounts would have 100+ servers behind Big-IP’s and Cisco Distributed Directors. What about setting up topology-based load balancing with other .edu’s around the country, as well as locally distributed? I assume you are doing analysis as to what *exactly* (http requests or sql queries, etc) is responsible for choking your servers? Or how about Amazon EC2 until you ramp that up?
February 27th, 2008 at 4:32 pm → First of all, we're at the MBL (Marine Biology Laboratory www.mbl.edu), none of us are associated with WHOI, they're down ... Read it ↓
First of all, we’re at the MBL (Marine Biology Laboratory http://www.mbl.edu), none of us are associated with WHOI, they’re down the road.
Second, we’re actively looking into mirroring, but the project is just starting and hasn’t even staffed up fully yet, so doing more than a local cluster simply wasn’t feasible for this release. I’m hoping we’ll have something lined up for this in the next six months.
Third, it’s not the computing hardware that’s causing the problems, what we’re seeing is a series of networking issues that due to their distributed nature is taking us some time to sort out. We’re hoping to be able to have a more stable environment up soon. I also want to make a public thank you to everyone working overtime on this, especially Pam.
And finally, a clarification for all of you who have gotten this far – we’re hearing people saying we’ve had millions of visitors – please be aware that this isn’t the case. We’ve had 18.5 million hits but every page view has a number of hits associated with it due to the aggregation that we do, this can be as many as 20 for some of the more detailed pages. I keep being told it’s nice to be popular.
Thanks for bearing with us as we get things sorted out.
February 28th, 2008 at 1:36 am → 變異狂犬病疫情 Read it ↓
變異狂犬病疫情
February 28th, 2008 at 2:04 am → I can hardly wait!...do you ever think you'll have it tuned for the b'width yer gonna draw CONSTANTLY...it's really going ... Read it ↓
I can hardly wait!…do you ever think you’ll have it tuned for the b’width yer gonna draw CONSTANTLY…it’s really going to be popular; TOO POPULAR
February 28th, 2008 at 7:40 am → Funny: you sound happy. You shouldn't, really. It's a problem and a signal of unprepared and unaware people. Unwared (of ... Read it ↓
Funny: you sound happy. You shouldn’t, really. It’s a problem and a signal of unprepared and unaware people.
Unwared (of what you’re doing)
Focus.
This is not a supermarket!
February 28th, 2008 at 11:50 am → Asterix Obelis: On the one hand, we are overjoyed with the public response. In effect, this launch is a real ... Read it ↓
Asterix Obelis: On the one hand, we are overjoyed with the public response. In effect, this launch is a real test of what loads we would expect. As stated by Jennifer Schopf, we have a complex networking issue going on because our harware appears not to be the problem. We did manage to do some load testing prior to launch but the network loads exceeded our expectations by an order of magnitude.
February 28th, 2008 at 12:21 pm → Good work! But you had to be ready for digg effect :) I hope, you'll launch soon :) Read it ↓
Good work! But you had to be ready for digg effect
I hope, you’ll launch soon
February 28th, 2008 at 11:16 pm → Thank you for such a fantastic site, and excited to see it grow further! Read it ↓
Thank you for such a fantastic site, and excited to see it grow further!
February 29th, 2008 at 12:14 am → Wow !! Internet begins to bear fruit : This is an Excellent work... with a big "E"!!, congratulations to the ... Read it ↓
Wow !! Internet begins to bear fruit : This is an Excellent work… with a big “E”!!, congratulations to the Team B:)
February 29th, 2008 at 3:41 am → Sincere congrats, guys! I'm excited! An order of magnitude more traffic than expected is certainly better than an ... Read it ↓
Sincere congrats, guys! I’m excited! An order of magnitude more traffic than expected is certainly better than an order of magnitude less….no matter the transient down time. Keep The Faith! I know that I will….
Aloha,
Rich
February 29th, 2008 at 4:11 am → Respect. Thank you so much for a fantastic resource and site. Very nice design. I'm looking forward to see more!! Read it ↓
Respect. Thank you so much for a fantastic resource and site. Very nice design.
I’m looking forward to see more!!
March 20th, 2010 at 7:56 am → Ryan’s Ramblings » Blog Archive » Stop picking on the EOL! Read it ↓