November 2008
22 posts
Canadian SMS service
Unexpected changes in our billing have forced us into a difficult situation with our Canadian SMS service. We can’t afford to support this service given our current arrangement with our providers (where costs have been doubling for the past several months.) As a result, effective today we are no longer delivering outbound SMS over our Canadian shortcode (21212).
The ability to update...
SMS problems with Sprint users
The provider we use for SMS is making changes on how Sprint traffic is handled. This is resulting in all Sprint users getting a error 2115 when trying to send to 40404.
We will update here as we get more information as to when the problem will be fixed.
Update (12:55p): Our provider has resolved the connectivity issue in the past hour or so, but we are still working with users who reported the...
Facebook app offline
We introduced a bug into the facebook app which we are working to fix.
Update (6p): This has been resolved.
Search Restored
The network issues that caused the search outage have been corrected. Search is again available with dat during the outage missing.
Search downtime
There’s a network outage affecting the data center where search.twitter.com is hosted. We are working with our hosting provider to resolve this as quickly as possible.
Search behind realtime
search.twitter.com is about ten minutes behind realtime. This should be corrected shortly.
Update: this has been corrected
Systems nominal
The outages we experienced today and last night were caused by a sub-optimal configuration of our database servers (a problem that we knew about but did not expect to go critical before it was resolved).
We’re making changes based on the outage to improve performance. New machines will go online as early as this evening. We do not anticipate the need for planned maintenance in order to...
Twitter is down
We’re recovering from an outage and will post here with updates.
Update (9:49a): We’re back up and will be conducting an extensive review of the problem that happened this morning and the one that occurred last night. We apologize for the unplanned downtime.
Update on SMS delivery problems
We uncovered and fixed a major problem with the SMS service that resulted in a large number of messages being dropped. This problem affected users from all carriers.
There may still be problems that remain. We know there’s additional things that we need to do to make the system more reliable. However, fixing this problems should result in a significant improvement in message...
Updates on deletions
Status deletion has been re-enabled.
User restoration has also been re-enabled.
User deletion remains off as it still represents a significant problem for the stability of the service. In the past several months, we’ve had to turn off user deletion many times in order to keep things running smoothly. Each time we turned it back on, we’d hoped we had sufficiently patched the existing...
Image upload errors
There are a couple errors happening with image upload for profile pictures. Some images become resized incorrectly and break formatting when uploaded. Others are getting errors when uploading images (particularly with gifs) even though the files conform to our size restrictions (under 700k).
We are reworking the image upload system to deal with both of these problems (and others).
Site outage
We’re experiencing a site outage and are working to recover.
Update (1:32p): And we’re back - this was a DNS configuration error that impacted the entire site.
We apologize for the unscheduled downtime. We’ve not had an outage of this length since mid-July and will be carefully reviewing what went wrong.
Search temporarily behind
The search.twitter.com feed is behind. This will be resolved shortly.
Update: this has been resolved
User deletion/restoration temporarily off
We’ve temporarily disabled user deletion and restoration to speed the recovery of the data inconsistency issues noted previously.
Data inconsistencies
We had a database replication service fail last night, which has caused a number of problems — missing direct messages, direct message counts off, missing replies, etc. All the data is in the system, it’s just not in the right place.
It’s now in the course of correcting itself but will take an hour or two to catch up. Thanks for your patience.
SMS trouble
We’re investigating a problem, affecting many users, where messages sent via SMS to Twitter are not being received.
There are also people who are not receiving updates from Twitter; we are working on this as well.
Update on SMS delivery issues
We’ve received notification from our SMS provider that a problem has been resolved that would have affected delivery for a large number of mobile users inside the US. A broken connection between our provider and the mobile operators caused a significant number of texts to go undelivered. This connection has since been fixed.
We are still investigating a number of lingering issues. We know...
Verizon delivery problems
Many Verizon users are still experiencing problems with SMS delivery. We are working closely with the providers we rely on to delivery these messages in order to get delivery working smoothly again.
SMS delivery problems
We’re having SMS delivery problems to T-mobile and other operators. We’re tracking down these problems now and believe they primarily exist upstream from us.
Deleting Posts Temporarily Suspended
This function will be back in all its glory tomorrow.
Canadian Short Code problems
We’re working on problems affecting users sending and receiving text messages over the Canadian short code.
People search disabled
We’ve had to disable person search; it was being abused in ways that were detrimental to the overall stability of the service. We hope to bring out a more stable (and better) version but not in the short term.
In the meantime, you can still find folks you know on Twitter by importing your address book (note that we don’t store your username and password and won’t email you or...