July 2008
19 posts
Some developers using the Twitter API may notice stale data returned when requesting JSON formatted responses. This is an issue with older cached data being served and we are actively working on a fix for this. Thanks for your patience!
Database down
We have to take the service into an unscheduled maintenance because of some issues we saw with one of our databases. Thank you for your patience as we work to bring it back up.
Update (9:16a): We’re back after an unplanned 30 minute maintenance. We’re watching the service closely as things return.
Missing replies
A number of folks are unable to view their replies tab. We fixed some of these cases yesterday and Thursday, but there are more who are still without. As a potential workaround, some have said that refreshing a few times will bring the replies back.
We’ll get this fixed up as soon as we can.
Followers and Followings Restored
We have restored 99.6% of the following/followers that temporarily disappeared as a result of a database error. The remaining 0.4% are on their way back. NOTE: Although we have fixed the problem, the numbers and profile pictures in your sidebar will probably not be accurate until tomorrow afternoon because they are cached.
More restoration underway
We’re still putting follower/following relationships back into place. There will be inconsistencies over the next several hours as this restoration takes place. We will provide more updates as they become available.
Update (2:14p): we’re going into maintenance mode to speed up some of the recovery work we’re doing. We’re think the downtime will be brief.
Update (2:34p):...
Updated follower/following counts
We’re still in the process of recovering from the missing follower/following problem that occurred earlier today. Over the next several hours, you may see inaccurate counts or timeline inconsistencies as the correct data is propagated to all parts of the system.
One thing to note: Even after this recovery is complete, your counts may appear lower than previously. In almost all cases, this...
Missing followers
Folks are reporting problems with missing followers - that is, seeing the follower counts drop on their profile pages.
We’re looking into this now.
Update: We’re going to go into maintenance mode to correct this problem.
Update (5:08p): We are out of maintenance mode. The missing followers and relationships are being restored. It may take several hours for your relationships to be...
Service availability
We’re investigating a service outage that began a few moments ago.
Update: Apologies for the late update, this outage was brief in duration. But we’re still investigating site issues this morning.
Whales
A number of users are seeing whales (over capacity messages) this morning. We believe this is because our master database is still warming after a RAM upgrade over the weekend. This may also explain some of the inconsistencies people are seeing on their timelines.
Timeline inconsistencies
We’re investigating various inconsistencies with people’s timelines including significantly stale content. Some people are seeing older updates from folks. A work around is to go to a person’s profile page directly. We’ll keep you updated here.
Profile image issue
We’re experiencing an issue loading profile images and other static content from Amazon S3. We’re working with Amazon to resolve this.
Database machine issue
A monitoring process on one of our database machines spiraled out of control, causing the machine to become unresponsive. We’ve killed the process and are working on restoring services. Thank you for your patience.
Update: and we’re back. Load times may be a bit slow until all services have restarted.
Unplanned outage
We’re working to recover from an unplanned maintenance event this evening. This was caused by a problem with one of our database servers. We apologize for the downtime.
Although this outage was unplanned, we’re going to use this time to upgrade the affected database. We’re hopeful this will help with some of the whale blips we’ve seen a couple times this week.
Update...
Whale blip
We had whale sightings for about 10 minutes this morning due to an issue with our master database. This affected all services. Apologies for the inconvenience.
Continued Progress
As an update to my post earlier this week, just wanted to note that the last few days have continued to be good:
Whale sightings have continued to go down and pageviews up. Also, we’ve raised the API limit considerably (on an experimental basis).
We’re not out of the woods — but it’s been a good week. Thanks to our awesome team for their hard work.
-@ev
Measurable Improvements
Last night’s maintenance was a success — and, in fact, it only took 30 minutes instead of the scheduled two hours. It was another small step, but definitely in the right direction.
As we mentioned yesterday, page response times have significantly improved in the last week, as you can see from this chart:
(Here’s a live version from Pingom, so you can check on us any time.)
...
Maintenance window tonight
Just a reminder that we’ll be having a two hour maintenance window tonight starting at 10p Pacific.
While any downtime is unfortunate, we have been making noticeable improvements to site reliability and performance during these windows. A lot of the most recent work has focused on upgrading the caching layer we use.
As an example of the gains realized through this changes, we had one of...
Thursday update
Our planned maintenance window was a success and we’ve added some capacity to our cluster. Still more upgrades and expansions to do.
We’re hopeful that once we’ve improved the stability of the service we can bring back IM. It remains the highest priority feature we’re working to restore.
Update errors
People are reporting errors when they update. Updates appear to vanish on the web (requires a page reload) and clients like Twitterrific & Twirl aren’t able to update at all. We’re working on a fix and intend to deploy it today.
Update (9:42a): One of our slave databases went down 10 minutes ago. We have to put the site into maintenance mode while we bring it back up. Thanks for...