Wow. Let’s hope not!

In the Interview I did with Twitter executives Evan Williams and Biz Stone yesterday they said it would be several more months until Twitter’s problems are solved. http://scobleizer.com/2008/05/31/clearing-the-air-with-twitter/ — your questions are interesting, though.

Wow, are you kidding me…only 3 servers hosting their 1 database?! and no High Availability, no Clustering? Whoa!! What about load balancing on the web servers to handle the load? WTF?! Serious re-engineering ahead!! the twitter IT folks really need to roll up their sleeves and get busy!

BTW, I offered my services today to @twitter @EV @Biz today on twitter – no reply. I am seasoned Systems Engineer and Project Manager (Certified in both)… NO reply from twitter… they need all the help they can get and didn’t even reply to my Pro Bono offer. oh well.

Haven’t they received another 15 mil in funding? Where is the money going?

First.. amazing that so much is running on small MySql cluster. We have those setups here and I am glad to know we can scale up before we have to change architecture at CelleCast

Secondly, to get this big and be nursing it along is quite a shock

Interesting

What a bunch of noobie IT folks over at Twitter, not to mention another black eye for the amateurish RoR community.

Wow. Dude, I know you aren’t paid to be nice, but you have a serious asshole streak when it comes to Blaine Cook. Just remember that what goes around comes around.

@Robert Scoble, way to get that interview! This is a messy state to be in for twitter.

Good questions. In regards to the last one; six or seven months may be too long for the users to wait.

Friend Feed! Who thinks Friend Feed is the new twitter?

Susan: they covered this in the interview yesterday. They don’t want outside help. That makes sense. You might, indeed, be someone who could help them, but they don’t know that and increasing team size does bring a lot of inefficiencies that could decrease their speed greatly. They also covered the money issue. They have piles of it. But getting someone hired, and an effective part of the team, especially one that totally needs to rebuild systems, isn’t going to be something that solves these issues in hours. -

Sarah: FriendFeed is not a new Twitter. It’s a new FriendFeed. Listen to yesterday’s Gillmor Gang for a separate interview with FriendFeed’s founders, who took on the idea of cloning Twitter head on.

Wow Susan! Ev and Biz didn’t respond within 5 hours to you, a PM (certified)! Shocking! I can’t imagine anything better they might be doing right now more important than instantly responding to one of the 200,000 people who seem to know how to fix Twitter.

Is it true that a lawyer is trying to grill a free service about their technical architecture when he would not know the first thing to do to try and fix, let alone manage, this infrastructure?

Michael, you make it sound like a Congressional hearing. It’s not like whether Twitter is up or not is a matter of national security. (or is it??)

Why are questions – not answers – posted on a site that’s had a serious bone to pick with Twitter for weeks being treated as true by default?

I haven’t liked Twitter’s unreliability, and they shot themselves in the foot until very recently by keeping everyone in the dark on issues. Still, how about we attack them for confirmed things, not rumours?

Wow – why so personal? I’m sure Blaine & the team worked really hard and did the best they could with the massive, unexpected traffic growth. They built a great business. Scaling is hard. This type of personal attack I’d expect to read on Valleywag not Techcrunch.

I sure hope that is not true. That would be amazingly poor engineering.

Then again, maybe that happens when you don’t have revenue to provide the motivation of “we lose $x every minute the site is down”.

There is just no way that there is a single master and two slaves. That just can’t be.

Holy shit, if Arrington’s investigative reporting is true then RoR developers are bigger douchebags than I ever thought possible.

After all if the servers of 700 signal’s flagship products have to be restarted constantly to work (see: http://www.zedshaw.com/rants/rails_is_a_ghetto.html) this platform is one giant FAIL of a choice.

Seriously, children. Just for l337 boasting rights you’ll choose an immature turd of a platform? You should not be allowed to have any decision making positions. 2 years of failure at the highest profile site implementing RoR and more to come.

Jeez. You’re acting like Twitter is a missle shield or something. It’s a social network. I’m being completely serious when I say that you may have some sort of addiction or disorder if you’re this angry about mild downtime of a social networking tool.

Please get help.

Everyone gets frustrated when a service they use goes down, but I think you’re being pretty harsh, Mike!

Also, to other commenters, don’t assume all of this is fact. Like only having 3 database servers…. I don’t believe that. Just not possible for such a high load, write heavy application to have so few database servers backing it.

Angry Fan: You do realize that (a) that quote was from like 2004 and (b) he was running on FastCGI, don’t you?

OK, we’ve rehashed the architecture, platform, engineering and technical components as they relate to scaling and downtime over and over and over. Am I the only one that sees this as a leadership issue? The IT / technical component can only be poorly implemented if the leaders / founders don’t build the right team and don’t make the right calls in regarding to how they are going to build out the service. If you accept that Blayne was part of the issue, then the founders are surely at fault for keeping him around for as long as they did….. I would like to see an open source version of twitter of sorts evolve and would like to see twitter have actual competition. Where are the twitter competitors? There IS room for competition in this space …….

Geez, why so agro? And why does everybody seem to assume that all of these statements are verified? Let’s see if they answer. Or go watch Scoble’s interview of the Twitter execs?

I agree with @Ben, I don’t see why you are specifically “blaming” Blaine?

I guess it’s always easier to blame the last guy – the guy that’s no longer there to answer his accusers.

I’m hoping Blaine responds to clarify.

Question of my own. Does anyone know if Twitter even has a test system?

[...] Read this post on TechCrunch. Then watch this video. [...]

Actually, if you proxy your database in many gigabytes of of memory cache, with various views into it, then really all you need to do to the database is tell it new information and delete old stuff.

It doesn’t help when the memory store gets corrupt (i.e. as in the past couple months) and then causes the db to almost literally explode.

“undo the damage”? no one will remember it in six months.

I think it is time we all recognized that “TC” stands for “TwitterCrunch” not “TechCrunch”…

People! Twitter is a cool idea that was badly implemented. That’s been well established in many contexts and admitted to on a number of occasions by folk that work at Twitter. It doesn’t do anyone any good to keep repeating the Twitter is screwed up. Heck, if it required Blaine Cook to keep the servers running, then all that does is confirm that it was a badly designed system and Twitter is doing the right thing by moving on from what was there when he was there. Why do you keep trying to embarrass Cook any more than he already is? It’s not useful. Also, why do you care about the number or type of databases they are using? For a messaging system like Twitter, the database should be largely irrelevant and only rarely accessed. By focusing on “the database” you only prove that you don’t understand this class of application. (No better than the people who think “Ruby” is the root of these issues.)

Look. It is well established that a service such as Twitter can be implemented reliably. We’ve even got a number of existence proofs. Of course, the one I’m most familiar with was PubSub.com which handled significantly more message traffic with larger messages and more complex matching to about the same number of distinct endpoints that Twitter serves. (We processed well over 2,000,000 messages per day scrapped from over 40 million blog feeds, where each message was an arbitrarily large blog post, and we routed them in real-time to over 1.5 million delivery queues defined by full text, complex boolean queries. Just like Twitter, we provided RSS/Atom and XMPP interfaces. Unlike Twitter, GrabPerf consistently showed us to be the fastest “search engine” on the web while we were running.) Others have provided examples of systems with similar or greater message volume as well as the same extreme fan-out that Twitter has. Twitter-like systems are not exotic or particularly hard to build… You just need to know what you’re doing.

Solutions to the Twitter problem are well know and commonly in use. It doesn’t do anyone good to keep pounding on these folk and trying to figure out “Whose fault is it?”. They’ve got system that needs to be fixed and everyone knows it. Pounding on them doesn’t do anyone any good.

Can we please convert “TwitterCrunch” back to “TechCrunch” and revisit all this in a few months?

bob wyman

Why beat this dead horse? We all know that their architecture needs work. What you’re doing now is just rubbing it in and slinging childish personal insults. Grow up.

Hey Arrington-

Would you please switch over to politics reporting? You could have prevented the Iraq war. I freaking love the harsh questions. Keep up the good work.

@Scoble – thanks for the feedback on outside help; that makes sense for sure (thinking of the “Mythical Man” book as I type this). They do need a highly skilled team that is really experienced at re-engineering web apps and db environments – so yes, speed of execution will definitely be an important consideration in team selection. I wish them the best. And twitter team wisely assessed the time element. It will easily take 6 months to re-architect, build, test and deploy twitter ver. 2.0

Hey Mike, stop crying like a little girl who can’t send her stupid tweets, grow up.

What’s funny is twitter that is about the simplest type of web app I can imagine, where the tweaks should be retrieved with single table db calls and heavy page caching. I kind of wonder why all the chaos?

I am so sick of hearing about Twitter.

This article would make you think the only problem is that they have an over-burdened database server. If that was it with the money they have they could run Twitter on a System Z to give them breathing room while they build out the team and re-architect the system to leverage multiple instances of commodity hardware.

Michael,

Did it ever occur to you that the reason Twitter is running so hot with only three DB’s or not enough servers, etc. isn’t actually Blaine’s fault? My “official” title at Digg is “Lead Architect” and I have NO AUTHORITY to buy, procure or install new servers. In fact, I don’t even have root on production boxes (many might argue this is a good thing). ;)

–Joe

I agree with Mike Doeff. Let them work on it.

Why does every non-techie thinks that tech is so simple? It is not!!

Alex

all you armchair quarterbacks in here make me want to reach through the tubes of the internet and strangle you with my bare hands!!

you’re so lucky the tubes are clogged right now…

Wow, Micheal is being a real DB today. He blames the guy who brought twitter to the point where he gets serious withdrawal symptoms if it ever hiccups. Just wow.

No matter what language u use Michael (Ruby, Java, PHP, C++), just try writing at least 20,000+ records when Scoble posts. Every time. Along with every other twittard at the same time. If you know anything remotely about databases, you’ll know that this is a difficult proposition.

Fact is, this is one of those problems that have not being easily solved as yet. The problem of sending a message to tens of thousands of people in real time from a centralized database is not easy at all. It’s not just like email, since they have to be stored in the DB as well, and be hit my hundreds of thousands of queries through the API at the same time.

People quip that its a messaging problem. It really is not so much. People surmise that its an RoR problem, it definitely is not. People think Java will make it “magically” work. It certainly will not.

The problem is one of architecting twitter in a way so that when our buddies like Scoble, Arrington, Laporte and the Diggnation guys posts (not to mention a potential Paris Hilton!), it doesn’t have to make those massive writes to the database. Or a way is found to make all those writes across a equally balanced sharded database cluster. Or some other solution.

Anyone who makes something like Twitter is going to have the exact same problem.

hey mike – is it true that you are an asshole?

Can only agree to what Bob says – Twitter is a nice service but if they are not able to get it running in a short period of time, a new service will come up and take it’s place (pownce?). Doing instant messages is nothing new and maybe RoR just sucks and isn’t made for such huge traffic websites or their database design needs to be updated. I am sure there are good consultants out there who can help in this case so please stop crying …

And there are much more important things going on than twitter.

who cares ? Don’t like the way twitter is running ? then don’t use it. Do you think you can do better ? then build it. But pls, give us a break about twitter. it’s really not that important.

@39 ‘Ericson’ Smith: seriously, language is not an issue? you had a legitimate point about it being bad design to use a database as the source for the outgoing messages (FAIL). But, you must either be smoking crack to suggest that a Ruby based application handles the same throughput as a C/C++ compiled one, or simply yet another RoR douche. The speed of the application has a huge impact when you’re trying to do any sort of large scale real-time work, and choosing Ruby and RoR is just plain stupid for this task.

Twitter: Please give Mr. Arrington an exclusive… He is feeling left out.

@angry_fan

Dude, its not that big of an issue. Wake up… A database can only accept so many writes per second. If RoR can overwhelm it, can you imagine how much more your C++ app will overwhelm it?

Suppose your DB can sustain 1000 writes per second (that;s the limit) and RoR is throwing those 1000 per second at it. How does it improves things if your C++ app begins to throw 2000 per second at it? I think you see the problem now.

The problem is somehow making those writes parallel across multiple databases, or some other solution. Not increasing the flow to one or a few DB’s. Don’t be a DB like Mike was today.

A house of cards… Twitter to soon become a victim of its own success. If they have “piles of money”, yet have let this problem perpetuate itself to the point of absurdity, it certainly speaks volumes to the inexperience of leadership, as well as not having a strong enough customer centric approach to the business. It is 2008, there is no time, or excuse, for this degree of patchwork systems management. If a beast has been created that can’t function, then rethink the business model or SI. The clock is ticking… The Principals need to move out of the way and let someone else drive the bus before they destroy any remaining enterprise value.

Can anyone at Twitter spell ‘enterprise class’?

@angry_fan

Just to follow up, i’m not saying that C++ is not faster. Any fool knows that it is. And of course its great for the kind of real-time work, and we don’t know that they’re not already doing that for parts of the app that needs it.

C++ or Java or whatever compiled language would be great to use IF they can solve the underlying architectural challenges, but to just replace what they have now with Java or C++ without solving the base issues, it will just magnify the issue.

From what I know, the guys over at Twitter are not fools.

You know, I’ve been reading everywhere that Techcrunch has lost its purpose, and that specifically Michael’s public perception is going down the tubes because of attitudes he displays.

I have enjoyed reading Techcrunch for 2 years. However this post is the first that I’ve personally encountered where I have to say, “WTF Mr. Arrington?”

I live here in SF and enjoyed the free Iron Man preview at Metreon. Got a chance to meet a bunch of the people I follow and look up to in the industry, including Michael. You don’t seem to be a jerk in person, why are you getting so snarky here?

For someone who has interviewed hundreds of companies, are you getting so personally offended with Twitter because its an invaluable service to you? Dude, if that’s the case then put together a smart team of people and BUILD A NEW TWITTER.

You have the contacts with VC and talent. JUST DO IT.

@Ericson Smith:

I think we agree on most things except for this

The guys over at Twitter are fools – an app built around babysitting MySQL is badly designed.

Also, my point was that a compiled app acting as middleware should obviate the RDBMS. Does any IM system of scale use an RDBMS?

Twitter, Twitter, Twitter. Seriously need change.

I wouldn’t be shocked if Twitter answered you. Like you said, the one thing they are doing well these days is communicating with their users.

Twitter doesn’t need to fix and improve their architecture at the moment. The community is so big its too late for that. Now they need to create a complete new framework that can handle major load.

Oh my F*ing god that is why the down times during middle of months I say in their down report looked damn familiar as a pattern..

This is DBA incompetence on the highest order if all conclusions hold and I am sure they will

BTW, that is why they cannot handle track its Database intensive..basically the SQL query to do track approaches infinity thus queries have to be more than cached they have to be systematically optimized programmatically to get away from approaching infinity..

Your questions are funny Mike,

It would be like asking you, Mike Arrington.

Is it true that TechCrunch and all the crunch sites are actually a GPL word press install on a single server with a custom html template?

Is it true that Techcruch can’t handle so much as bad javascript as Seesmic pulled it down the other day?

This line of questioning is mundane and irrelevant to most people

As a technical expert and software programmer, even I do not see the point of them.

Twitter, aka ANY SITE is a package or product. Either it works or it doesn’t. Why would you want to go in and act like their sys admin?

If it were any site but twitter, I gather you would say it sucks in other words and that would be the end of it.

It’s almost like you and Scoble are trying to mold this site into your next Facebook or something. Trying to steer it in whatever direction you want.
I think that’s a little disturbing. I don’t think you would do this type of amateur technical consulting for any other website. It actually makes me question the relationship you have with them as well as Scoble’s.

@Angry Fan:
Um, yes, ejabberd, and most other OSS XMPP servers support mysql, postgresql, as well as possibly the other commerial RDBMS. Are you sure you know anything about them?

@55: Chris, I too question Arrington’s relationship with Scoble. I think it’s amazing that two narcissists can actually have sex with each other. Doesn’t that go against their prime urge of exalting their own selves above all else?

Plus, that’s one porn you don’t want to see. Though maybe you want to get a twattr from them when they start doing the monster mash together.

Well done Michael.

A handful of emotive words and you have 50 comments and counting.

@56: Mr. Church, it may be your dyslexia, but I said IM applications of scale. Not some pissant OSS XMPP douchebaggery.

Lordy, you’ve got problems with reading comprehension.

Heh, seems by the time I had read the comments you picked up another seven ;)

Is that why most of your major outages can be traced to periods of time when former Chief Architect/server watcher Blaine Cook was there to sit and monitor the system?

Bing! I think you finally figured it out (or you’re getting good advice from a smart techie).

When did Twitter start getting unreliable? When Cook and Lee Mighdoll left. I don’t know which one was keeping it running or if both of them were, but the current situation has all the signs of people fumbling around trying to figure out which dial to turn when this or that happens.

This was a good post. The previous one was way off (and I said so).

Keep on truckin Mikey.

@Angry Fan:
I’m not sure how dyslexia would come into play? Maybe you could explain?

Many corporate and govenrment IT dept. are running customized versions of OSS XMPP servers. What are they using for DBs? What do you think google is using for a db? Multi-value dbs? I don’t claim to know, you do.

So, quit being a douchebag and tell us all how data storage is done on the “IM applications of scale” you seem to know everything about.

BTW, why don’t you write a post like this for your friend who runs Mahalo. You’re good at Emperor Has No Clothes pieces. We both know JC is running around buck nekkid. Why give him a free pass??

Michael,
I didn’t read the comments, but they should pony up and do some sort of co-sponsorship with Microsoft and throw the DBs onto MSSQL and run the servers on IIS. IronRuby is coming along nicely, although the RoR app might want to be upgraded to a more robust and tested framework/server OS, even if it isn’t IIS 7 and .Net

Does this mean CrunchBase can’t scale?

Ha ha, “damage to the community.”

You ever notice when the author of the post gets up on his high-horse that the posts follow suit. Everybody’s an expert! I offered pro-bono for bragging rights and they responded with NIH syndrome! RoR suxxxx0rz!!1

Hey, pile on everybody. Arrington, isn’t your training supposed to be in economics? Why not second guess their spending of the 15mil instead of blathering about IT? It makes you sound like Gillmor without the marijuana.

@Ericson Smith: you’ve said everything I was about to say.

No language “magically” recovers from database slave failures, so why would ruby?

I’m not sure what’s wrong with Michael Arrington recently, but his quality has defiantly gone down hill, every other post is about twitter.

I have a few questions too.

* Does your customer contract with Twitter state ‘zero downtime’? Oh wait…it’s a free service, no contract.

* What damage to Twitter and ‘the community’ are you talking about? They continue to grow rapidly and you continue to use it like a crack addict.

* Why do you continue to personally attack one of the guys who built it? Without his hard work it wouldn’t exist.

Funny, but a bit harsh?

RoR = a Joke.

OMG – Michael Arrington talking tech – mentioning things like MySQL replication and stuff. Doesn’t everyone know he doesn’t know what the f*ck he’s talking about!!! He’s a goddamn lawyer and a pseudo writer hack. We’re at 14 minutes now. Time is almost up.

Arrington talks a good game, but I think that’s all he’s really capable of doing. I doubt he has even the first clue as to what any of this stuff means.

@57,

I am in no way implying that Mr. Arrington and Mr. Scoble are having gay sex together. I am simply saying that I question their ties to Twitter.

Former attorney turned journalist turned amateur system administrator and DBA.

Jack of all trades, or somebody who is a little too interested in Twitter? This is not really a story in my humble opinion.

@Dave Winer: Well, let’s not get carried away. Arrington has been pushing Twitter with its lack of business model for two years, Facebook shits bricks of gold in his eyes, Mahalo is the bestest search engine evarr, and Yahoo MUST SELL now (each post collecting some M$$$, I guess).

The source of his critique isn’t to be a serious tech blogger, it has more to do with his tendency to throw tantrums like an overgrown baby. You should see what he does when he runs out of cigars. He’s like that giant baby-thing in Spirited Away.

their problem are understandable. ruby, mysql are only suitable for content oriented sites. they just made a mistake of using it instead of something more solid.

Time to move on from the regular twitter bashing – you get what you pay for and at the end of the day its a free service.

ha, they don’t even have a dba position open :)

@Greg: Wrong, just because a service is free does not mean that they have any less of a responsibility to their users. They are building a business, make no mistake about it.

Hey, here’s to redefining the word “Tweetard”.

Tweetard: “A person who constantly bashes the Twitter service, yet can’t stop using it.”

Hey, i’ll give an iPod nano to the first person who can verifiably get this up in urbandictionary.com

Contact me through my URL above!!

Twitter

Way to go armchair journalist. Feel good beating this dead horse? Instead of running your mouth at Twitter constantly, actually do something in technology. Have you ever dealt with the technical difficulties in something the scale of Twitter? Yeah, didn’t think so. Try it yourself, then you’ll have some respect for the people who fuel the service you spam daily for your personal marketing message. Tool.

What really impresses me about Michael is that he can keep up (and/or lead) in so many realms.

And that includes technical discussions.

It’s amazing.

Now (and not that he needs a defender in any way) but if Michael was a bit nicer to Scoble in print (sometimes he is nice) perhaps Robert would take him along on such interviews (and not that Michael doesn’t lead as well with his own interviews — and scoops – leading ability)

Michael, why go on a personal Jihad againt Cook? Also, do you evidence that he was dismissed????? (maybe you’ve already published that, I don’t know).

How long will it be until you are able to undo the damage Cook has caused to Twitter and the community?

I stopped reading your post too soon — that’s totally unfair.

@Ericson Smith – LOL – this one wins ;)

I’m pretty confident in saying that Arrington has never once set foot inside any type of MySQL database, let alone setup any type replication/failover situation. There are much larger sites running on much less hardware, you don’t have a clue as to what you are talking about.

I’ve just set up TechCrunch Sans Arrington, a piped feed containing TechCrunch’s articles minus Michael Arrington’s painful diatribes. If posts like this one make your brain hurt as much as mine, be my guest and subscribe.

Eh. http://icanhaz.com/tcsansarrington

read comment 28 by bob wyman – and that’s all you need to read.

oh, look at the # of comments here – mission accomplished for the author.

@Ericson Smith: you win. “Tweetard” is more effective then my favorite replacement, twats twatting messages.

Only in SF would the overwhelming response to an article rightfully calling bullshit on a PR push to deflect attention from fundamental incompetency be “awww. Why you got to be so mean?”

Honest and unmerciful Arrington. Keep putting the feet to the fire.

Michael, like many, I am a TechCrunch addict but over the last few weeks I’ve become twittered out and I DON’T EVEN USE IT. I have never seen ANY company get as much (obsessive) coverage. You’d think Twitter was the most important thing to come along since Jesus himself.

This is the funniest TC post I have ever read.

You know what’s so funny Michael, you know sheeit about architecture let alone databases, well may be from a 1 million feet.

PUT YOUR MONEY WHERE YOUR MOUTH IS and contribute to Twitters CapX. Because right now you sound like a little girlie man.

BTW, question for you. Since your cred is worth zero amongst tech’s, who do you think will work with let along for you now??? Find the problem and fix it. Don’t blame people.

You’re probably just ticked because Scoble out scooped your egotistical self ;-)

There you got one more page view – go get a doggy snack

WOW!! I think we all totally forgot that you have sooo much IT experience and background and that you weren’t a lawyer at all and have tons of technical training. Right?

Dude, a little tip since you clearly can’t see beyond the 5 feet surounding your desk but NO ONE USES TWITTER OUTSIDE OF THE UBER GEEK/TECH COMMUNITY!! IT will never get mainstream adoption because…you know..we have these things.. called CELL PHONES, yeah it’s cool, they let you talk to people WITHOUT the Internet!! and they also have tese things, you might have heard of them, called TEXT MESSAGES that allow you to send short messages to a group of people with 160 characters! W.o.w.

You must be so smug, just sitting there in your chair looking like shit (we’ve seen your seesmic videos) and bashing other people’s hard work. I guess it’s soo easy to do when you don’t have the balls to try yourself huh? Oh, don’t get me wrong you may invest in various companies and call yourself a “founder” but you surely don’t have the pleasure of running any of them and dealing with their day to day issues huh?

And your job board company was a huge success, wasn’t it? Who burns thru MILLIONS with no significant revenue? I mean come on, no one is that stuipd. Looks like you should stick to “reporting” and leave the real jobs to the people who know what their doing.

After all, those who can’t do “teach” right? And your out to teach us all a big lesson about what everyone is doing wrong and how they suck, etc, etc, etc. Douch bag. You couldn’t even cut it as your own CEO, how lame.

So I tell you what, how about you do something significant, like run a startup that tries to develop a new business or technology from the ground up and then you get the right to tell other “owners” if they are doing a good job or not. Your so stupid you would have panned Google back in 1998 or Yahoo back in 1994 because it wasn’t XYZ enough.

Go screw yourself, you are NOT relevent to the world at large and you are NOT a journalist because you only write about companies that your VC cohorts tell you to write about. Way to go on independent thinking there! But I guess if you were a worth while person you wouldn’t be 38 and single huh?

Someone seriously needs to knock you down a peg, like a stupid HS girl who thinks she’s all that…the bigger they are the harder they fall.

Manny, Twitter doesn’t need Mike’s money – watch the video. They have the money. They need more experience and more help, along with a new architecture, which could take them months to develop.

OK, your a great developer and have experience that can help twitter scale and you want to help.

The secret is to shut up! The founding developers know they are over their heads and are hanging on to their jobs with their teeth. Do you think they want someone to come in who can clean up the mess and make them look as inept as they are?

Tweak your resume so you sound like a mid level developer who can help them solve the problem and make them look like the savers of the system!

Once your in then start fixing the system (remember to document your solutions) and remember to send firendly notes about what you found and one or two recommended fixes to the project lead and your manager.

I’ve worked at many companies that the managers wanted someone who could solve the problems but the interviewing developers were deathly afraid of someone who had more experience than they did.

Blaine Cook watching twitter server every second. When it failed, he will manually switch the cable from one server to another.

Wow, it is the future of server computing. The new approach in solving scability. ;)

I’m giving these guys a break. Sure there’s plenty of work to do and in this business they can ill afford to have the downtime and negative press they’ve experienced lately, but I’m never quick to judge. Unless you’re on the ground of a company, there’s no way of truly understanding the challenges within. Business culture, management, and investor relations sometime bleed into the tech requirements and resources.

We have NO IDEA what was being asked of these guys or what kind of pressures they’ve been under in the past year or so. I know from experience that when the term “monetization” begins to enter the discussion, tech is usually an afterthought…at least on the minds of the business folk. They do need to get their act together, but I sympathize greatly.

@95: Right on. A storm in a teacup. Or a tornado in a bucket of Arrington’s urine.

If the money isn’t going to hiring REAL talent and clearly isn’t going to hardware architecture, it must be lining someone’s pockets.

Tradition: hot discussion about Twitter :)

I thought you complained about Federated Media all the time, but it seems Twitter is still the favorite.

Best to focus on ways that actually may help them. This to me does not do it. More finger pointing. Would welcome a post on Twitter that was constructive.

Firstly, this reporting is hearsay. Oh.. I know a guy who knows a guy that says that Arrington is the long lost brother of Blaine and that Mike used to sysadmin at Twitter. What BS.. really people! don’t fall for flame bait. duuh.

Its really funny about all the people that are posting here, criticizing twitter and really have no idea how to scale a site.

If you *do* know to scale a site (and you’ve done it before.. not just read a bunch of blogs) you obviously don’t truly understand the problem domain that twitter has to solve.

It’s not just obtain a bunch of stuff from the database, cache it, load-balance it and serve it up.

Any reasonable competent sysadmin can do this. The guys there are NOT stupid.. There’s plenty of resources on the internet on how to scale. The problem isn’t as simple and trivial as you think it is.

I was at lunch with a group of guys (at railsconf) and blaine joined us. I wanted to go get the ‘scoop’ of what the whole problem was, but I thought, just leave the guy alone.. there’s too much anger focused on him personally and it must suck.

Remember you have a choice you use twitter or not use twitter.

If you think you can do better go ahead.. TRY.. I don’t think any of you whiners have the balls for that one.

A twitter fan.

I love the weekends. Bravo!

Before you launch again into a dissection of why Blaine’s done such a bad job (which is a bit rich since most of the recent problems seem to have happened immediately after he left), perhaps it would be worth looking back at Ev’s previous success – Blogger – which was also pretty much unusable for great tracts of time as a direct consequence of not being able to scale effectively. I’m not implying it’s his fault, just pointing – again – to the fact that a lot of things can impact on the ability of a service to stay up and much of it is going to be connected to the amount of financial and engineering resource you’re prepared to put onto a problem, as well as how effectively you hire and what product decisions you make. One option – for example – on the product side would be to stop new registrations as a way to maintain the service’s ability to function for existing users. That’s not an engineering decision, that’s a product decision and is one that is clearly not being made. You really really need to get away from thinking that how Twitter is architected, scales and is supported is all at the door of one man – particularly when the evidence would tend to suggest that while that man was at the company, the service was substantially less flaky.

… and humanity? That’s pretty funny.

I doubt any of these armchair architects can even configure a mysql instance from scratch.

Hey Michael I have a few questions too?

How many master MySQL servers should twitter be using?

What total number of ‘physical database machines’ would you find an acceptable number for powering Twitter?

How would you propose Twitter architect a fully redundant database architecture whilst maintaining an acceptable transaction throughput?

How long will it be until you are able to move on from Twitter and start doing some investigative journalism again?

You know – looking at your graph again at the top of the page, another way to read it would be that Blaine was the only thing holding that company together…

Sorry, I thought you said

…undo the damage Cook has caused to Twitter and TO HUMANITY?

not community.

Tweetards :-)

I don’t use twitter as I usually do work when I sit at my computer, but I must say the comments are fun to read.

I picture in my head a room full of nerds (rich nerds, but still nerds) swinging their arms trying to hit each other as they call each other names.

Leave the fighting to the UFC and move on folks…it’s just a web service..and it doesn’t work.

Go find a new toy.

Anybody know if twitter blocking yahoo pipes from getting feeds? I have set up a feed of my friends using pipes and it has been dead for last day. When I run the pipe directly from pipes instead of my site where pipes is embedded I get the following error:

warning error fetching http://twitter.com/statuses/friends_timeline/BrokerScience.atom (401 Unauthorized)

any ideas?

Tweetards, indeed. This has been the funniest post in a long time.

C Thomas Edwards :

“looking at your graph again at the top of the page, another way to read it would be that Blaine was the only thing holding that company together…”

YES. but the problem is, Blaine himself built it. He built a platform that required his constant attention to stay live. and a platform that no one else can keep live.

http://www.webomatica.com/wordpress/2008/05/31/twitter-reminds-me-of-the-it-crowd/

God that’s funny.

Great that the industry leaders are addressing the issue.. good for Twitter, great for the community. Do Web2.0 a favor! :-|

see update in post above.

“Twitter continues to be annoyingly and constructively responsive to criticism”

hahaha classic.

@117
A striking resemblance to every ones favourite Twitter punching bag!

I can’t believe they even allude to the Mars rover on their blog.

In their response, they’ve indicated an alternative filesystem-based approach. This is in fact a good and viable solution (in the lack of zillion servers available like Google), though not a long term solution. In any case, they’ve a new batch of $15M, don’t they?..

Our new architecture will move our reliance to a simple, elegant filesystem-based approach, rather than a collection of database.

So wait, all of the problems are database related now? What happened to Ruby on Rails? Which one is it?

I am not a real geek. I like cool things and try to keep up on tech but just so everyone here catches this. There is this thing in the universe called critical mass. Facebook, Google, Microsoft, and now twitter have all caught this. I was talking to one of my friends that just graduated high school about a week ago and I asked him if he had heard of twitter. He had just started using it within the last 2 weeks. he said it was a great way to have his friends join him for lunch without having to setup something “official”. So long has twitter does not have a complete meltdown it will continue to explode because it is hitting the main stream. Oh, and Arrington relax.

So they basically said that everything in this post is true, they’re patching it up for the short term, and re-architecting for the long term. Wow, they really took the fun out of this..

Arrington you redeemed yourself with the above video. You have relaxed good job.

The pics twitter uses are so nice and reassuring, quite frankly I don’t really mind when they are down or overloaded. If twitter becomes super-efficient and smooth running I shall miss the enforced twitter breaks.

But then, whilst I’m a twitter addict – I’m not twitter dependent as Michael is – that’s the real problem. Michael really needs twitter to function.

Aleph (128) means that he’s a tweetard.

Was this post ghost-written by Ted Dziuba?

It smacks of his brand of “I read an O’Reilly book and am now going to insult you.”

I’m pretty over all of these twitter outage and scoble accusation blog posts all over the internet. It’s like high-tech passing notes in middle school. Play nice.

The proof that this blog post is garbage is simple.

Replace the word “Twitter” with “MySpace” or “Rapidshare” or even “MSN” in the original blog post.

Now what? now what?

You wanna know something about the “big hit” Twitter that Mike Arrington won’t dare to show you.

http://www.alexa.com/data/details/traffic_details/youporn.com?site0=youporn.com&site1=twitter.com&y=r&z=3&h=300&w=610&c=1&u%5B%5D=youporn.com&u%5B%5D=twitter.com&x=2008-06-01T05%3A26%3A31.000Z&check=www.alexa.com&signature=3YhDu2C%2FDXlxwggMTiN4t6SrBVU%3D&range=3m&size=Medium

Twitter can’t even hold water against a YouTube CLONE. Not youtube itself, a CLONE of youtube that specializes in pr0n. What does that tell you about this new mega-site of the year?

michael, you seem to be obsessed with twitter and the supposed shortcomings of its architecture. lots of websites, from startups to established companies, have scalability problems. what the big deal about this one? i don’t feel the urge to post every time a page is slow to load or i encounter a 503.

Mike – the backlash of technology is everywhere. The other day I had my GPS fool on me at the Lincoln tunnel, right in the middle, and put me on to the wrong lane and then had to recalculate. I don’t think I can complain that to Garmin. Having said that I understand that Twitter should have made the checks and balances in place to not make this news. But again, it is still the same “technology” that has its own limitations.

Why always so mad at Blaine Cook? I saw you calling him out on Twitter last night or the night before too. I must have missed something but I’m pretty sure that one person can’t be to blame for all this.

I feel kind of bad for Twitter, they didn’t really plan for it to get this big, and I’m sure they’re glad it did but you can’t blame them too much for being unprepared. They just got some big funding and they’re working on making it better, what more do you want?

When you launch a Web service, this should be planned in your risk assessment plan… Why so much amateurism? I am speechless.

@Sarah Austin
I can’t stand FriendFeed. Tried it… Makes me think of Plaxo and all these wannabees. Hopefully some senior engineer will come and work @Twitter and fix this. Would be regretful to loose the mass currently on Twitter.

@Amanda: I never knew Dane Cook was Twattr’s architect.

But seriously, the amount of grief Arrington, Scoble and other no-talent ass clowns are pouring on him is pathetic and offensive. Whether he failed in his position or not that gives nobody license to personally attack him because every time Twattr fails it makes them throw a tantrum.

That Arrington is a giant blubbering baby doesn’t give him the right to trash Cook personally and professionally.

Honestly Michael, listen to yourself! You really think it’s the sole responsibility of the architect to keep the whole company going even after he’s left? You don’t think the CTO / CEO / Product lead or whatever have *any responsibility*? And you’re prepared to sat these things with no knowledge of the resources at his disposal, with no knowledge of how he was managed, with no knowledge of how free he was to make decisions or act upon them? With evidence that the head of the bloody company has had trouble scaling projects before and with substantial evidence that the product worked better when the architect was around?! I mean – I’m not saying that he’s not responsible, just that from outside how the hell can you tell!? You can put the best people in bad situations and they won’t be able to do good work. You can hire geniuses and then ignore them. You can have great people and then give them nothing to work with. Again – you cannot know this situation as well as you pretend to and in the process you’re basically eviscerating a man who – as far as I can tell – is utterly unable to defend himself without basically burning all of his own colleagues. From what I hear Twitter themselves don’t feel negatively towards Blaine! Having said that though it’s probably fairly helpful for them if you absolve them publically of any and all repsonsibility in this area in public over and over again.

If you want to give Twitter a hard time, having accepted that they provide an extremely entertaining service, i would suggest you do so in the round. Criticise the company for not having been able to scale, criticise the company for not being able to monetize. Or put the responsibility where it belongs – at the top – and let them redistribute it as they see fit. Because really you’re not going to be able to assign blame. You really can have no idea what’s going on behind closed doors. And until you can say confidently that Blaine got all the support he needed, you’re basically just mounting an unwarranted and sustained ad hominem attack. Which is – I’m afraid – indefensible.

http://www.alexa.com/data/details/traffic_details/xhamster.com?site0=xhamster.com&site1=twitter.com&y=r&z=3&h=300&w=610&c=1&u%5B%5D=xhamster.com&u%5B%5D=twitter.com&x=2008-06-01T06%3A34%3A30.000Z&check=www.alexa.com&signature=sWsFrq7Jwr2kHHXTPKCNPY7dLVc%3D&range=3m&size=Medium

I just noticed something, any website that clones youtube and hosts pr0n will automatically beat the crap out of Twitter.

Even a 1 man website can beat the entire twitter team in traffic and views.

Isn’t that something.

news.google.com/news?source=ig&hl=en&rlz=&q=california%20pornography%20tax&um=1&ie=UTF-8&sa=N&tab=wn

You guys may not have noticed this but California is passing a tough new pornography tax. That is going to push people to the internet for free content more than ever before.

Again, it seems pretty easy to beat twitter in a traffic matchup, with a script, ffmpeg, linux and a dream.

Arrington, remind me again how much you are paying to use this service?

@47. Supposedly there is no such thing as a stupid question. That has now been proven false with your question. What possible motivation does Twitter have to be “enterprise class”? Are you really trying to lump Twitter in with the likes of SAP, DB2, Oracle, SQL, Domino, Exchange, etc? All of which charge their customers thousands of dollars to justify being enterprise class applications? How much does Twitter charge their users? What SLA’s have they guaranteed their users?

Arrington, do you get equally upset when a radio or TV station goes off the air for a lengthy period of time? Another service YOU DO NOT PAY FOR!

@141: Last time his cable service went off for ten minutes Arrington shot the TV screen with his silver .22 handgun, after pulling it from his pink purse.

Erickson Smith #39

You nailed it. The problem is one of an exponential explosion of writes to the db that scales to the size of the community of listeners to a particular sender, which as I understand it is not limited by twitter accounts. The database setup they have is quite robust for normal high transaction applications. Unfortunately, those exponential write explosions is simultaneously twitters strength, it allows the quick reach out to the community of listeners) but it is also the weakness , all those simultaneous sends kill the db fast. What twitter needs is a way to slow down the propagation of new tweets *naturally* by setting up , distribution points architecturally.

I have a way to do this that I’ll be glad to share with twitter …for ehem…a fee. *grin*

[...] seiner unnachahmlichen Art formuliert Michael Arrington auf Hey Twitter I Have A Few Questions Too ein paar Fragen, die nahelegen, dass Twitter ein Dienst von unglaublicher Relevanz sein muß. Sie [...]

[...] seiner unnachahmlichen Art formuliert Michael Arrington auf Hey Twitter I Have A Few Questions Too ein paar Fragen, die nahelegen, dass Twitter ein Dienst von unglaublicher Relevanz sein muß. Sie [...]

“I have a way to do this that I’ll be glad to share with twitter …for ehem…a fee. *grin*”

Dude, totally get a clue. They’d turn to craigslist.org 80x10000power times before taking you up on an offer. We have teh best coders in the world here.
I’m so bored and drunk right now. Sorry for the sprinkle of reality.
I see startups like Twitter every single week in LA. I’ve toured the buildings, worn the id badges.

They’re focusing on this one for a reason, and I’m not sure why. I can tell you that they’re nothing special as far as this area goes.

if this is only de DB issue, SSD disks can help to feel the gap… but I doubt that this is the main issue.

If I may:

I am a non-tech-industry geek who enjoys staying ahead of the curve, on the bleeding edge, and generally keeping up with what’s going on in tech-land.

I discovered TechCrunch a few months ago and have enjoyed reading (most of) the stories covered here. It helps me stay informed.

But — I do not use Twitter. I find the occasional story about them interesting (‘new release’, ‘raised money’, etc.) – kinda like everything else on here.

This obsessive level of coverage that the company’s been getting recently is absolutely mind-numbing, and frankly it makes me less want to read this site every day. I REALLY do not care about the low-level scalability problems they’re having or who screwed up what or who did what to whom in the bathroom with the candlestick.

So as a mild-mannered, average non-tech-industry “regular guy”, please hear me when I beg you to STOP MAKING EVERY THIRD POST ABOUT F’ING TWITTER!!!!!

That is all.

Noah – just skip the Twitter posts, read what interests you. Or bail. We’ll both be fine either way.

I’ll keep writing whatever the hell I feel like writing, you keep reading what interests you, wherever that is on the Internet, and things will work out.

I don’t know, I don’t know.
Comments are fan though at twitter posts.
But it’s obvious that twitter posts drive traffic here… whatever.

By the way I like question 3. For what it seems twitter’s architecture was
not meant to sustain such load, that is one thing no one can argue.

[...] Techcrunch pone domande sulle risorse techiche di Twitter perennemente semi down … [...]

Yea. NOT HELPING!

Yeah, it’s the architecture not the implementation technology. It depends on the type of hardware too. There are single servers hosting 200’000 domains…

Dammit, stop bashing on Blaine Cook already – get a life.

I’m really suprised that twitter answered those kind of questions ( although the answers are not really to the point ).

Every stupid Arrington post is on http://www.techmeme.com/

Gabe is so lame!

Saturday morning TechMeme didn’t have a single post from TechCrunch on its home page. Not on the top, not at the bottom.

I became concerned. Really concerned. Did Arrington get ill? What did he smoke Friday night? Too much coke at D6? Comscore showing TechCrunch bleeding. Will TechCrunch survive?

But now, finally, the old FOX News of the Internets is back in shape. 150 comments and counting. Reader engagement and emotions. Scoble and Winer and Iskold (of ReadWriteWeb) as guest stars. Pageviews peaking.

And Arrington’s cash register is ringing so loud that neighbors in Atherton have called police.

Another TechCrunch weekend…

I’m so glad that Dave Winer invented blogs for the whole world to get UK-style tabloids. All Arrington all the time.

BTWI just noticed that the Feedburner icon in this page’s header lists just 758k readers — severly down from > 900k just two weeks ago.

I guess we will get another 2 posts of similar quality still this weekend.

wow, It’s exciting and I saw your blog in technorati with 22000+. OMG

Are there any other startups in the world besides Twitter? I love hearing about all of the petty little problems every day. Maybe there is another startup that is having petty little problems too?

Maybe they are waiting for either Yahoo, Google or Microsoft to buy them out.

Don’t forget just how powerful the Twitter Brand name is with the global web community. Ditto these posts and other relevant blog moans.

Michael, with all due respect… Have you ever coded? Have you ever learned ANY programming language?
You used to be a lawyer, right? Then don’t pretend you know computer science.
please.

They’re focusing on this one for a reason, and I’m not sure why.

thats why twitter will be gone in 5 years, tops.

geez, its not that hard to scale..i can lay it out in 3 steps

1) load balanced web servers. takes about 2 minutes to configre
2) mysql cluster backend. Easy
3) get rid of hack job RoR, and switch to a real language

The end.

Re: TechCrunch update above, why (would one wish to) get companies (and also readers) riled up. It’s only a short term traffic and attention gain strategy if employed.

[...] question to add to all of these posts asking Twitter [...]

[...] The debate about Twitter rages on. Scoble met with Twitter team and talked about the various issues in a video interview. I got an [...]

[...] but at least they are doing it now. They’ve even managed to foil Mike Arrington’s attempt to start a late-weekend bitchmeme by asking some rather pointed questions of the [...]

Technology screw-ups almost always cover management poor judgment. Here’s a post on that: http://blogs.zdnet.com/projectfailures/?p=798

Hey Mike what happened to your RSS numbers? Couple weeks ago it was 945k now its down a couple 100k….Hope the Jain story and the Twitter link bait is helping.

@Arrington: Fuck you, you’ll write about what we tells ya, or your cigars are getting broke.

Lots of very interesting news here. It sounds like Twitter is in desperate need of an infrastructure architect. The capacity planning sounds like it has been an afterthought.

Ryan
lesson in brevity: http://www.mofata.com

Jealousy no twitter interview? Whats eating ya Mike? Little brisk with your readers and your losing subs way to go dude shake your money maker!

[...] nur eine rhetorische Frage sein soll, hat Michael Arrington bei TechCrunch gleich ein paar richtige Fragen an Twitter [...]

Arrington, in case I wasn’t clear–that wasn’t a rhetorical question.

HOW MUCH DO YOU PAY TO USE TWITTER?

And don’t say that is not the point, because it is absolutely the point. As even a law school drop out will tell you, you have no contractual obligation with Twitter to deliver you a certain level of service.

Clearly, with the amount of funding Twitter has, they would have experts in the field. ;) Or has the money gone somewhere else…?

How sad you have nothing more to write about since they responded smartly.

Time for more 1938 media posts!

Twitter – We serve millions with a desktop and a geek!

@Robert Scoble, I’ll do just that. Thanks!

Using three database servers is ridiculous. I was under the impression that they were doing a lot of difficult processing behind the scenes, and was willing to excuse their downtime based on that. I’ve written a few more thought’s on what this means for Twitter in my blog – http://joshduck.com/blog/2008/06/02/twitter-is-crud/

@174
Nobody cares about your stupid blog. At all.

[...] Arringtoh wrote a post yesterday about architecture issues in Twitter (though his last question is very lawyerly and off [...]

[...] is losing potential new users as the site continues to go down. Earlier this weekend Mike Arrington singled out Blaine Cook, an ex-Twitter employee, as the source of the problems. In my own opinion it was an unprofessional [...]

You “new media” practitioners are a bunch of self-righteous freeloaders that don’t know anything about real technology but state yourselves as technologists.

Hey Mike,

Is it true that you recently changed your name from ‘Assington’?

A co-worker of mine posted the above as me, but to be fair I have been ranting about how stupid this post is and referring to you as ‘Michael Assington’, no harm no foul. Just thought I’d clear that up :)

Good luck with your new career in web application scalability.

RYAN, you nice book

I attended a talk on ‘scaling Rails’ yesterday at RailsConf in which Blaine Cook participated. He admitted that scaling was an afterthought because “twitter was the stupidest idea ever” and that it wouldn’t have been built if scaling had been a requirement up front. Twitter was an app that was built for fun, and it’s success surprised everyone involved.

2.0 Savvy Kids – Part 4…

Michael Arrington has some questions about Twitters architecture?…

Interesting. They are overloaded many a times. They show some rubish and childish pictures when they are overloaded.

Talking with other people about Twitter may show your lack of Knowledge, at least they are doing something ! Why would they have to respond to anyone !It seems Geeks have an Ego that only Geeks Understand ! Take a lesson from Scoble he has tact and understands building a business ! As with all Starbuck’s Democrats you are a pain in the butt ! As with all Geek’s obviously you only communicate with machines , people skills you LACK !

All the Twitter team’s response did was highlight your poor judgement in singling out Blaine Cook. Until you apologise for that, I think you can safely be considered a unprofessional jerk.

All the Twitter team’s response did was highlight your poor judgement in singling out Blaine Cook. Until you apologise for that, I think you can safely be considered an unprofessional jerk.

All the Twitter team’s response did was highlight your poor judgement in singling out Blaine Cook. Until you apologise for that, I think you can safely be considered an unprofessional %$£@.

[...] טפיל וובי מודרני. אבל הגדיל לעשות מכולם מר ארינגטון עם הפוסט ההזוי הזה. איך אומרים, אם זה לא אישי זה לא [...]

@LSF – an unprofessional ASSHOLE

Very hard questions to them.

@139.

Well said.

Os problemas do twitter, o que realmente esta contecendo….

Os Problemas do twitter são velhos conhecidos de todos seus usuários e muitos estão extremamente irritados, devido às constantes quedas do serviço.
A equipe do Twitter foi questionada por Michael Arrington do TechCrunch e não tiveram escolhas a n…

[...] us that they’ve “Lost a database” we know what that means – one of their three database machines has gone down and they are busy trying to rebuild it. At least that’s what I think it means. When it comes [...]

Does anyone think http:www://twitterfund.com will sign people up or raise any money?

Nice experiment on whether a fan community will actually pay a small amount to get something they’ve had for free. Sure beats ads, surely?

[...] more read-slave servers, and by fixing some bugs for improved efficiency” (until now they had just three database servers, possibly shown in the image above [...]

[...] more read-slave servers, and by fixing some bugs for improved efficiency” (until now they had just three database servers, possibly shown in the image above ) [...]

Twitter helps me a lot in website traffic … twitter is really good

_____________
John Glenn
There are a lot of sites out there showing book video. BookVideoTV, BookTelevision and of course CSPAN, but I like how BN.com and Reader’s Entertainment TV have specific genre channels and original shows. There’s just more to see and I can be specific in what genre I’m interested in. Anyone else watch online tv?

Twitter has been a help in the traffic world.

[...] certain features is Twitter’s recent attempt to keep their frail architecture from failing completely. They tried it out during Apple’s recent WWDC keynote and it worked, [...]

[...] Replies are still down.  Disabling certain features is Twitter’s recent attempt to keep their frail architecture from failing completely. They tried it out during Apple’s recent WWDC keynote and it worked, so [...]

twitter never impressed me much, at least not on first impressions, perhaps i just haven’t found it to be all that much use yet

twitter never impressed me much, at least not on first impressions, perhaps i just haven’t found it to be all that much use yet

[...] all this could happen at Twitter. But Twitter is a company, with a few guys and (apparently, gasp) ONE mysql database (with two slaves). They have enough problems just keeping the fail whale at [...]

[...] some significant critical press relating to the internal workings of their technical architecture. When faced with some pretty direct questions from a pretty influential personality about why their service goes through periods of downtime [...]

Cool news!

[...] half-baked” tomorrow.  Or show your Director of Marketing a crappy review from TechCrunch, or Mossberg or NetworkWorld — wherever your product plays.  Then, in your mind, imagine [...]

[...] half-baked” tomorrow.  Or show your Director of Marketing a crappy review from TechCrunch, or Mossberg or NetworkWorld — wherever your product plays.  Then, in your mind, imagine [...]

[...] the details provided in this post by Michael Arrington are correct then Twitter is definitely the victim of poor planning, I know from experience that [...]

[...] Hey Twitter I Have A Few Questions Too Share and [...]

Twitter is a great social thing

This facebook connect thing is interesting.

[...] it was growing rapidly but was still having uptime issues with blogs like Techcrunch frequently questioning its architecture. Meanwhile, I had signed up for FriendFeed & was there lurking. I found a lot [...]

nice tutorial great work, thx for excellent informations!

[...] tutti, compreso l’autorevole Techcrunch, che in un post datato 31 maggio intitolato “Hey Twitter, I have a few question too” pone alcune domande al team di [...]

Great post.

very nice

[...] because of widespread criticisms about privacy issues, and Twitter has many times been able to calm outrage about repeated downtime by publicly responding with details about their architecture and plans for [...]