May 2009
19 posts
Display issues during code changes
Unfortuantely, users are encountering some odd display issues on twitter.com as new code rolls out. This shouldn’t be the case and we’ll work to fix that; the display problem should resolve shortly.
Both user icon and user background uploading is currently disabled but we’re working on the fix.
April 2009
41 posts
Bringing back disabled features (sidebar trends,...
Users in the percentage test of integrated search have had the trends in the sidebar turned off. We’ve also temporarily disabled background image uploading for all users.
We’re working on fixes to both of these issues.
Fixing the elevated error rate
We’ve seen an elevated number of whales and increased slowness of the site. We’ve identified the underlying problem and are working to deploy a fix.
Update: We fixed this.
Trouble deleting direct messages
Some users are not able to delete direct messages. We’ve found the problem and will ship the fix shortly.
Update: We fixed this.
Nearing the end of data cleanup project
We still have some problems with profile images, follower requests for protected users and device settings. These remaining issues are due to the data inconsistencies we posted about last week. The good news is that we expect these remaining issues to be fully resolved within the next 24-48 hours.
Cleanup continues; profile issues and direct...
We’ve finished cleaning up the problems that caused users to see the wrong profile page when logging in as well as seeing misdelivered direct messages.
We’re now working through inconsistent follow requests for protected users, errors in phone settings and missing profile pictures.
Progress update on data inconsistency issue
Having isolated and removed the cause of the data inconsistency problem from yesterday, we continue to clean up the affected accounts.
Thus far we have corrected the problem with misdelivered direct messages. We’re now cleaning up the problems with registered phones, profile images and users getting the wrong homepage when logging in.
We will post more updates as we continue to resolve the...
Tracking down data inconsistencies
A number of users are experiencing problems due to an underlying data inconsistency in our service. This problem manifests itself in a couple ways:
the broken profile image issue noted earlier
problems with phone settings
user name changes not being reflected consistently
inconsistencies when approving follower requests (for protected users)
misdelivery of direct messages
While these...
What's The Deal with OAuth?
More information on OAuth has been posted to our main blog.
Missing user images
Many people are seeing missing user images in their timelines, tweets and other places throughout the site. We’re aware of this critical problem and will be working to fix it.
A note about per day following limits
For some time, we’ve limited the number of accounts that a single person can follow in a day to 1000. A few days ago, we found that this limit was not being consistently applied and we started enforcing it for all users.
While there are technical reasons behind having some limit on following activites, this per-day limit exists to discourage spamminess. Also, it is unlikely that anyone can...
Site is slow
The site is currently slow and you may experience an odd whale. We are working to correct this problem.
Update: the site is back to its normal speed. Some follow actions may be delayed.
Progress on combating ongoing spam attacks on...
We’ve addressed the most recent variant of the Mikeyy worm. Still, we suggest that you avoid viewing the profiles of users posting uncharacteristic or otherwise suspicious tweets.
Defending against ongoing spam attack
We’re aware of the ongoing spam attack happening on Twitter and we’re working to bring it under control.
Delays when following or unfollowing @aplusk
Due to the incredible rate of follows & unfollows of @aplusk, some users may experience a delay before the follow or unfollow takes effect. We’re working to speed this up; sorry for the inconvenience.
Defending against worm attack
We are currently addressing a new variant of the Mikeyy worm. Please avoid viewing the profiles of users posting uncharacteristic or otherwise suspicious tweets.
Duplicate text messages
Some users are currently receiving duplicate tweets over SMS. You might experience an interruption in your SMS delivery while we solve this problem.
Update: We have fixed the problem.
Search results are delayed about 20 minutes
Updates aren’t showing up in Twitter search in real-time. Changes we’re making today will allow search to have more timely information.
We're seeing much better site performance this...
Delivery delay, site latency and errors are much improved this morning due to changes made over the last several days.
We are still tracking a missing update problem (not receiving all tweets) for some users. We’ll be cleaning this up as quickly as we can.
Continuing to improve stability over next several...
More changes we made this afternoon have improved the latency and error rate on the site, but we have more to do. Many of these improvements will take a few days to roll out. We will continue to update here on our progress.
We have resolved a couple critical SMS bugs...
Two SMS bugs have been fixed that affected users on specific carriers (including Vodafone in the UK.) First, messages sent to Twitter were being posted multiple times. Second, Twitter did not respond appropriately to commands like OFF and ON.
Both of these problems are now resolved. Apologies for the confusion.
Responding to slow load times and high error rates
We’re working through some problems with high site latency and elevated error rates. The problem is more constrained that those we were dealing with last week. We will update as we gather more information and work toward a resolution.
Update (11:12a): We’re pushing changes now to address this morning’s problems. This may introduce some inconsistencies when you view your...
Update on worm
We are currently addressing a new manifestation of the worm attack.
No passwords, phone numbers, or other sensitive information were compromised as part of this renewed attack.
More information about this weekend's worm attacks...
Please see the additional information we posted regarding the incidents from this weekend.
Update on StalkDaily.com Worm
Earlier today we were informed of a malicious site that was spreading links to StalkDaily.com on Twitter without user consent via a cross-site scripting vulnerability. We’ve taken steps to remove the offending updates, and to close the holes that allowed this “worm” to spread.
No passwords, phone numbers, or other sensitive information were compromised as part of this attack.
We're working to resolve an issue with some...
Some users will be missing tweets from the past couple hours; they’re not lost, just not visible in your timeline. We’ll be putting things back in order shortly.
We're seeing big improvements in site speed and...
This morning has been our best peak load period of the week with snappy page load times and low timeline delivery delays. Also, the missing or deleted timeilne issue should be resolved for most users. We know there is more to do - we need to take additional steps to keep these problems in check.
We still have some major bugs that need to be fixed. These include:
Image upload problems: we believe...
Updates on site issues
We’ve made good progress on delivering tweets in a more timely fashion; timelines have been up-to-date all day. We are also cleaning up the issue with missing tweets / deleted timeilnes. This should be fully resolved in the next 24 hours (and is fixed for most users currently).
Site latency has been the biggest problem today. The site is much slower than is acceptable and we are still...
Twitter.com is experiencing high latency. We're...
The site is quite slow this morning and we’re continuing to add capacity to address this problem.
Steps we took late last night have significantly improved tweet delivery time (new updates appear in timeilnes faster). However, some users will still be seeing the missing tweets or blank timeilne issue. We are deploying more code today that will isolate this issue.
Update on delivery delays
We’ve been actively combatting the delays in posting tweets to timelines. In some cases, this will mean missing tweets so that we can deliver fresher content more quickly. This is only a temporary solution and we are bringing online more capacity so that we can delivery tweets in timely manner.
We’ve also restored status deletion which was temporarily disabled as a preventative...
Site improvements, updates on delays and missing...
Due to improvements made yesterday, site latency is much improved and the number of errors being thrown (whales, robots) is significantly down.
We still have changes to make and we are currently contending with delays in tweets appearing in timeilnes. Also: a number of users are seeing missing tweets (these tweets are not permanently lost). We are actively working on both of these problems.
Update on delivery delays, errors
We’re actively making improvements to deal with site-wide issues. The number of errors on the site is still quite high which is causing slowness. And we are dealing with delivery delays of outbound tweets to timelines.
The missing user icon problem of yesterday should now be resolved.
Update: Although previously uploaded images are re-appearing, we do still have problems with errors on...
One hour maintenance starting at 5:45p Pacific
We apologize for the lack in advance notice, but we are taking Twitter offline for one hour starting at 5:45p Pacific. This is a reschedule of the work we had planned for yesterday morning and is critical to address the site performance issues affecting the site.
Update (6:24p): And we’re back.
Tracking improvement in error rate, combatting...
We’re seeing improvement in the rate of errors that’s been a problem for us for the past 12 hours. However, there’s still more work to do. Specifically, we’ve got more to do to improve delivery delays on tweets.
Missing user icons (avatars) and direct messages
A number of users are seeing the default user icon in place of an uploaded user image. Also, some are missing direct messages from their sent tab.
We’re working on both problems now.
Update: We’re in the process of fixing these problems and expect the cleanup to be complete in the next 24 hours.
Working through some errors this morning
We’re contending with an elevated number of error pages this morning. Site latency has also been a problem. We’re working through both issues now.
Update: Some users may notice tweets missing in their timelines. These tweets have NOT been lost. We are working to correct the problem.
Planned maintenance on Sunday 10a Pacific
Twitter.com will be down for an hour beginning around 10AM PST on Sunday, April 5, 2009. We will use this maintenance window to upgrade systems and hardware.
Update: We are rescheduling this maintenance window; Twitter will be up all day today.
Recovering from errors this morning
The last several hours have featured a significantly elevated number of errors on the Twitter website. We’ve just completed a deploy that has addressed a number of the underlying issues. There’s more to do but we’re pleased that the error rate (whales, robot pages) has returned to a nominal level.
T-Mobile delivery problems
We are still experiencing a problem sending messages reliably to T-Mobile. Due to a problem with our SMS provider, a significant number of outbound messages are not being delivered to T-Mobile subscribers.
We’re hoping to gain some improvement in the short-term and are working on a more robust solution.
Not finding yourself in People Search?
Some users (especially those who have joined Twitter recently) won’t be able to find themselves in People Search. We’re working through the underlying bug and expect to see Search up-to-date quite soon.
Intermittent problems with image uploading
Many users are running into errors when trying to upload either background or profile images.
We’re working on this problem and will update here with more information.