We are telling our customers to use Google Checkout or call us in the meantime. What a crazy thing to happen on a holiday weekend.

Yeah, definitely sucks for people who rely on them for 100% of their payment processing. Just goes to show that any site accepting credit card transactions really needs to have a backup solution like PayPal or Google Checkout when something like this happens.

Wow, the title suggests that the company just went under, as in failed, bankrupt, etc. But the article says that there is a supposed data center issue. Big difference, especially in todays economy, when many businesses are actually failing.

You’re right, I updated the title.

Huh? Title’s pretty clear. Service isn’t functioning, so it’s down. Service is down, so clients are left hanging…

There’s no sensationalism at all in the title.

Ah, I saw it post-update, hah. Ignore me.

/me steps slowly away from the keyboard…

I thought the same when I read the title. o.0

[...] See Authorize.net Goes Down, E-Commerce Vendors Left Hanging for a little more info. Eric Longman posted at 2009-7-3 Category: Geek Stuff [...]

We switched to simple validation in our MIVA shopping cart, which just collects thw credit card info. Hopefully, we can charge it later.

Read somewhere else – they expect everything back up by 10 am PST…

Lousy title. Should have used the word “outage”. But a blog’s gotta do what a blog’s gotta do.

This is not PCIDSS compliant, storing credit card information on your internet connected web server can get you fined up to half a million dollars by Visa and MasterCard.

Any merchant who relies entirely on real-time processing and/or doesn’t store declined orders in their database isn’t taking their business seriously anyway.

What’s worse is many of these smaller merchants and home based business’ are currently on vacation and therefore won’t find out about all of these lost sales until Monday! Thankfully as it is a holiday weekend sales are slow to begin with.

Had a hard drive crash this morning, turns out the untested backup program doesn’t actually work, so I lost the reference IDs I need to bill about $400/month in subscriptions.

On top of that Authnet goes down, so I have to update 4 ecommerce sites to only accept PayPal for the time being.

This is looking to be a really, really bad day :)

What happened to redundancy? You would think that a company as big as Authorize.net would have some sort of data backup / failover system so this could not happen..

[...] http://www.techcrunch.com/2009/07/03/authorizenet-goes-under-e-commerce-vendors-left-hanging/?awesm... [...]

I agree – the idea that one location having problems can bring it down is just mad, considering how large they are.

Customers could have the process go through, as the payment information is automatically stored and charge the information later….

Another company where it seems impossible to get a hold of someone on the phone…?

Jeremy, maybe in the future assume the poster made a legitimate comment and save your asinine snarky post for yourself.

I understand that there was a fire but why could they have not issued a press release to let us all know? Instead, thousands of us lost sales and customers and had to figure it out for ourselves. No working phone numbers, no working websites and apparently CyberSource corporate has the day off and didn’t think this was important enough to send people in to answer the phones. I called and got a message that the office was closed. This is despicable.

That would cut into their margins. Might as well have a single point of failure in this economy.

Yes, because it’s totally legal and Visa/MC won’t mind a bit if authorize just holds onto that info for a little while.

This not only affects e-commerce but also many Card Present/In-Store operations. Authorize.net has a card present API used by many retail systems.
Awesome timing with the holiday weekend.

They’re bringing the world of internet quality to customer service.

So much for geographical diversity and hot standby for critical services providers.

I suggest that every mid market business take their incumbent distributed server based point of sale gateways and messaging queues, and throw it all on a PAAS platform owned by a company that is that is actually and truly negative cash flow and in debt to their managed host – oh yes, Rack Space, or Amazon, who you cant complain to.

Go do it. Oh, they admit that yes, they are down, and no, they have no idea…..and yes, you will get a credit, and no, you are not indemnified against business loss due to the outage.

[...] Authorize.net Goes Down, E-Commerce Vendors Left Hanging (techcrunch.com) – July 03, 2009Talk about a serious outage. Payment gateway service provider Authorize.net has been down and out for several hours, a number of tipsters inform … [...]

Glad we use trustcommerce.com

Fire at the Fisher Plaza in Seattle. Not only has it taken down authorize.net, but also geocaching.com, plus probably others. A TV and radio station also have been affected and are using alternate transmission sites.

http://www.seattlepi.com/local/6420ap_wa_fisher_plaza_fire.html

interesting that 3+ years after acquiring authorize.net cybersource did not integrate them into the main cybersource datacenters, which they have multiple and is the system that processes all their medium and large customers.

Hmm… Conspiracy Theory – CyberTerrorism is trying out various things: first rackspace, then app engine, and now Authorize.net

Hasn’t this week seen some action…

[...] an outage at your payment processor’s datacenter may be the cause. According to TechCrunch, a fire broke out in Authorize.net’s seattle datacenter last night shortly before [...]

PCI Compliance does allow you to store CC info on your webserver, with the following caveats. The Primary Account Number (PAN) that appears on the front of the card can be stored. So can the cardholder name, service code and expiration date. These cannot be stored indefinitely. Usually only long enough to allow for returns or disputed charges (60-120 days depending on credit card used). Also, the info should be encrypted and your server requires a firewall and anti virus protection, as well as limited access to your server; that should be password protected. Paper records need to be shredded and/or burned after storage time limits exprire. CC numbers stored on paper with only last 4 digits appearing are ok.
The full magnetic stripe, chip, CAV2 CVC2 (number that appears on the back of the card) and PIN cannot be stored.

This is not the first problem at this building. Fisher Plaza previously experienced major outages due to electrical issues in 2008 and 2006.

Transactions are still not working as of 09:20 PST.

[...] Authorize.net Goes Down, E-Commerce Vendors Left Hanging __________________ TSS-Radio is Everything SIRIUS [...]

Looks like the Authorize.net gateway is back up.

Authorize.net has reported that transaction processing is back up.

In a short statement issued just a few minutes ago, a representative of the gateway said “ I understand transaction processing is back up with the exception of Global processing. We are working to bring that up ASAP.”

Follow the latest development on Authorize.net Twitter page at http://twitter.com/authorizenet

http://twitter.com/authorizenet

Nettica DNS service down too – they host out of Fisher Plaza and their homepage redirects to their news blog right now http://www.nettica.com

our transaction began hitting again a few minutes ago.

Has anyone successfully cut over to an alternate provider? We don’t have any ecommerce but we do a lot of Virtual Terminal stuff and we’re stuck as well. PayPal offers a Virtual Terminal but it takes 48 hours to open an account.

No… the Authorize.net gateway is not back up, regardless of what they say

Hope they will recover fast!

That isnt exactly true. You can store payment info assuming you meet other requirements of the PCIDSS about encryption and network protection. While every effort should be made to reduce or eliminate the storage of payment information and consumer information where not required there are cases where a company must store this data. I think this would be a case where a company should be collecting order for later processing. Any good order management service should have payment processing fault tolerance built into it.

Wow

The economy is down already – do we need more problems

[...] This is a true testament to the stinge-factor of financial services companies.  Those fucktards at authorize.net* put all their eggs in one basket and kept their entire infrastructure in a single datacenter. In which there was a fire.  And now they’re gone. [...]

[...] payment processors is having a service outage reportedly due to a fire in a Seattle datacenter.  Here’s a post at TechCrunch on the [...]

Definitely not a fun morning for many merchants and customers too.

This brings up two very important issues…

1 – The value of having alternative payment options for crashes and users who prefer to use them (paypal, google checkout, phone, etc…).

2 – Having good error messages. Many commerce sites just say “error” or have their site timeout when these issues come up. This doesn’t instill any trust with the customer who isn’t sure if they got charged, should reorder or what to do.

Give people options & information.

Hey Rober, good synopsis of the rules, thanks. I was recently trying to wade through a document about an inch think on PCi compliance put out by the bank and it was totally unclear

I still can’t log on and process through virtual terminal.

why don’t they have their own generators and back-up gas supply like our data center does? Ours can go for 2 days without electricity using their own. Or was the fire structurally damaging?

Google Checkout needs to lose the “registration” process if they want anyone to really use it.

From: http://www.merit.edu/mail.archives/nanog/msg19026.html

Fisher Plaza, a self-styled carrier hotel in Seattle, and home to multiple
datacenter and colocation providers, has had a major issue in one of its
buildings late last night, early this morning.
The best information I am aware of is that there was a failure in the
main/generator transfer switch which resulted in a fire. The sprinkler
system activated. From speaking to the fire battalion chief, I am under the
impression that Seattle Fire did use water on the fire as well, but I am
unsure of this.

Given the failure location, generator power was not available, and cooling
failed. UPS power to systems continued, and I can personally vouch that
they held out for well over an hour. When we were able to access our
equipment, ambient air temps were well over 100 degrees in the room our
equipment is located in.

At least some, if not many circuits were affected. Several large
co-location providers and other datacenters are located in the facility,
these facilities have no power.

As this was the main/generator switch, and it is now highly damaged, the
circuits in the area are damaged, and the entire area is doused in water, a
rapid restoration of power does not seem likely. Fisher Plaza’s phone
numbers now result in fast-busy signals, so I have no recent update from
them directly.

Interestingly, this building is also the production studios for several
Seattle TV and radio stations.

There is no ETA for resolution.

[...] Originally posted here: Authorize.net Goes Down, E-Commerce Vendors Left Hanging (TechCrunch) [...]

I feel bad for authorize.net customers, and at the same time I am glad we switched to Braintree Payment Solutions last year instead of authorize.net.

We contacted authorize.net in Nov 2008 and they forwarded us to some merchant account company I had never heard of. A week later they couldn’t answer my questions and wanted me to contact authorize.net about those, I was turned off by that and by their discount rates.

I had heard of Braintree on hacker news so I contacted them. We went live with Braintree in January and they are awesome. My account rep answers questions related to payment settlements and she has even helped me troubleshoot some api issues I had.

If anyone is considering switching because of this outage, you can find tons of advice on hacker news by searching on searchyc.com.

[...] company authorize.net. Nothing is known yet when the processor will be back online. A comment on this thread says they will be back online around 10:00 AM [...]

[...] Don MacAskill 10:11 am UTC on July 3, 2009 Permalink Tags: amazon, amazon payments, authorize.net, credit card, processing Authorize.net, one of our payment processors, is having an outage.  They’ve been offline since 11:15pm PST July 2nd (last night) due to a fire at their datacenter in Seattle.  Apparently their backup datacenter, in San Jose, couldn’t handle the load and is also having problems.  You can find more information on their Twitter account and this TechCrunch article. [...]

My Authorize.net seal was causing my site to load slowly, so I solved the problem by disabling the seal.

I’ve switched my payment settings so my cart will accept credit card information. However, it will not automatically authorize or capture a sale. This seems to be working fine.

No doubt – Lesson learned in a big way. Installing PayPal backup now for future situations.

[...] Techcrunch var addthis_pub = ‘atronao’; var addthis_brand = ‘Atronao’;var addthis_language = ‘es’;var addthis_options = ‘email, favorites, digg, delicious, myspace, google, facebook, reddit, live, more’; Post Relacionados:Lo que lleva la adicción al vídeo juegoGoogle lanza servicio de red socialGoogle lanza nuevo servicio sobre la saludHakean datos personales de 6 Millones de personas en ChileGoogle lanza a ‘Mail Goggles’ para esos emails impulsivos « Sarcófago de Michael Jackson costó $25,000 dolares [...]

[...] it was Google App Engine that went down and today it’s Authorize.net that has gone down. The provider of payment gateway service has been down for more than 8 hours (almost). And those 8 [...]

authorize.net back up for me now at 1:30pm EST

I was able to finally log into Authorize.net – 1:50pm – but when I tried to add a new order to the Virtual Terminal – the transaction would not go thru – for unknown errors. Will wait a little while.

Here’s my affiliate sign up link for authorize.net. Don’t all of you jump on it at once!

As a network admin I am totally blown away that a company we entrust with our lively-hoods does not have the necessary redundancy built in to their systems. I blame myself though. The thought that they didn’t have a disaster plan to meet every scenario imaginable never even crossed my mind. I’m sure they would have told me this could never have happened even if I had asked though.

I would bet that a cost cutting management team ignored the recommendations of the Admin team, who I am confident were pushing for more redundancy if they were competent. I know Admins can sometimes appear to be over zealous but this is the reason why. It is very stressful managing a system that is not prepared for catastrophic failure do to management oversight.

I hope other managers can learn from this what the true cost/benefit of redundancy is.

Authorize.net back up for me at 1:45pm EST. Don’t know for how long what with everybody trying to send all their “held” transactions at once, but we shall see. Will be looking for backup or alternative method of taking payments for future issues though.

This maybe a good place to ask you guys –

We are looking for a hosted payment solution that allows full customization. So, PCI compliance is taken care of by the payment solution firm, but the payment page can still be customized to match our site’s theme. Do you guys have any recommendations for this?

I heard that they were launching fireworks INSIDE of the data center!

I think it is working now our payments are going thru . so make test transaction , but authorize.net site is still down

Leo

What a crazy thing to happen at all… “banking stuff” should not rely on one physical location… I have more security guarding my personal photos…

1:30 p.m. CDT, Authorize and Global back up for us, both for order processing and remote/bulk capture.

Once again the importance of redundancy, failover, and GSLB comes into focus. How many more outages like this are going to happen before all companies outsource their DNS?

http://dynect.com
http://bit.ly/McUTc

You’ve nailed it. Yesterday most of us would have assumed that Authorize.net was, ahem, “too big to fail”. But that very assumption reveals the same problem in our IT processes that seems to have just been revealed in theirs.

Authorize.net Goes Down, E-Commerce Vendors Left Hanging…

Talk about a serious outage. Payment gateway service provider Authorize.net has been down and out for several hours, a number of tipsters inform us. That has big implications: since the service is used by tens of thousands of e-commerce vendors to acce…

my current employer uses authorize. it’s API error documentation is horrific and you can’t manually hit customers in your vault. both a stark contrast to braintree payment solutions which i have used in the past – highly recommended. no i don’t work there.

Check out rocketgate as an alternative payment processing company. RocketGate uses multiple live active datacenters so there is no ‘failover’ needed to keep your business up and running

[...] including off-site redundancy is critical. Authorize.net is a recent example of such a failure, Authorize.net Goes Down, E-Commerce Vendors Left Hanging Payment gateway service provider Authorize.net has been down and out for several hours… That [...]

Hopefully, this is a wake up call for all payment processors to re-evaluate their backup plans. Thankfully, our merchant CDGCommerce had a cost-free alternative Quantum Gateway for us to utilize for the time being.

In the last 7 hours, there have been approximately 340 tweets using the #authorizenet hashtag.

The situation, in general, indicates several things…

– Payment acceptance is the lifeblood of the small business.

Not a surprising piece of information…but something that is oft forgot when discussing payments acceptance and initiation modalities in general

– Redundancy…redundancy…redundancy

As someone who lives in the payments space, I am intrigued with the information that will flow out over the next few days regarding the situation. A fire in the datacenter is simply a nightmare…but, why did it take so long for customers to be notified? Why wasn’t there a fail-over to a backup site?

I suspect, believe, that true, full analysis of the situation won’t ever be fully released. But, as information trickles out, it will be intriguing.

– PCI-DSS education is still necessary

As emphasized by some of the comments toward the top of this list…Simply capturing the payment information for “future” processing (in cleartext) is wholly unacceptable. And yet, due to the situation, merchants are faced with the distasteful task of either putting their customer data at risk or losing sales.

An unfortunate situation…wholly unpleasant…and somewhat intriguing.

Thanks for the link! Glad I could help.

The first thing I did was look for @AuthorizeNet on Twitter. Their account was there but no followers or updates. I see now they must have just made it a little bit prior to my search.

It’s amazing how Twitter is helping to break big stories quicker than ever. It’s just a matter of finding credible sources to back up the tweets. Great job as always TC.

[...] link: Authorize.net Goes Down, E-Commerce Vendors Left Hanging [...]

So long as you’re setup to save both abandoned carts & the customer info from non-finalized orders, you should be able to follow up by email or phone and get payment info after the fact.

But of course preemptively notifying the customer while in checkout and having them choose an alternate option such as “mail in payment” would prevent said customer from receiving an authorize error message in the first place.

[...] shot to report it since it has to do with a lot of e-store sales (includig our own JAMM Store).  We just got word that the service that handles a lot of e-vendor transactions, Authorize.net, went down for a while [...]

Authorize.Net Web site at http://www.authorize.net is back up and seems to be responding quicker than before.

[...] shot to report it since it has to do with a lot of e-store sales (includig our own JAMM Store).  We just got word that the service that handles a lot of e-vendor transactions, Authorize.net, went down for a while [...]

[...] just reading a blog post on Techcrunch about Authorize.net going down due to a fire in Seattle.  Crazy [...]

Authorize.net is supposed to have a redundant backup system. Why didn’t it work. Does this call for a class action suit?

That is a serious outage, but non-SaaS enterprise apps experience similar outages, and to one degree or another, these things are to be expected.

Sure everybody should be concerned with availability (and “scalability”) when they select a SaaS vendor. I reduce risk by only playing with the “big boys,” but they have gone down from time to time as well.

The question is not whether there will be some downtime, but whether a company, especially a small to mid-size company, is in a better position than a trusted SaaS vendor to manage the app in question?

Whether or not a businesses operates in a cloud or not is a irrelevant. Both models still operate from physical locations. The issue is redundancy plain and simple and down time is unacceptable for any mission critical service. Don’t take my word for it, sit in on a disaster preparedness class that all Network Administrators should have taken and you will find that there is no mention of acceptable down time.

However; if your argument is that “these things are to be expected” because of incompetence, I agree. They should not be the norm but rather the exception though and if we give them a pass on outages such as this then we get what we deserve.

PaymentsGateway.net offers exactly what you’re looking for. If you go to their developer forum at: http://forum.paymentsgateway.net/showthread.php?t=48 take a look at their “Secure Web Pay” solution. Additionally, you might also take a look at their CMI solution for tokenized payments which allows you to make calls to store and charge credit cards (and e-checks) on their platform without having to store the sensitive payment information on your server. This takes you out of scope for PCI but still allows you to charge, refund, void, etc. the cards through API calls.

We were one of the businesses hit. Luckily for us, our customers had cash.

So fucking retarded! How did these assholes think that is was ok not to have a second data center set up with a redundant backup with the amount of customers they have?

We noticed the gateway go down, shortly after 1:30am not later as they claim. I’ve been on with my lawyer all day, expect a claim next week assholes.

[...] Authorize.net Goes Down, E-Commerce Vendors Left Hanging By Robin Wauters Talk about a serious outage. Payment gateway service provider Authorize.net has been down and out for several hours, a number of tipsters inform … TechCrunch – http://www.techcrunch.com/ [...]

Getresponse was down this morning as well on west coast time.

[...] midnight Pacific time. I’m also trying to encounter out how some businesses are affected; TechCrunch says it’s “tens of thousands of e-commerce vendors,” but when Authorize.net was [...]

I couldn’t agree more, we are a UK based recent start-up small company distributing through out Europe with suppliers in the US & Japan we RELY on the data centre for emails we have been with out communication for over 24hrs. I have the utmost sympathy for you guys who rely on web commerce. All our orders come in via email as well as most of our European communication and to not have a back up data centre is as you say totally fucking retarded, I would decribe the data centre as a bunch of some thing begining with C and ending in TS rather than assholes

We were affected by their down-time as well; however, we were able to effectively use Google Checkout and PayPal as backups.

Authorize.Net has given us the confirmation that we were RIGHT in the recent partnership with a server management company that offers TRIPLE redundancy for all of their server solutions.

For such a system to go down and only have ONE backup, which was local, is unacceptable.

[...] major media outlets were also covering the outage. At around 11 AM EST, Authorize.net setup an official Twitter [...]

Bing Travel is down too.

The whole point of a managed DNS service is to add/help with redundancy on top of content. Check out http://dynect.com and http://204.13.248.102 (watch video on how to Break Free).

[...] and this post as we learn more. More information can also be found in this article from [...]

[...] updates after the jump) Read more Share and [...]

Very glad that I found this information here – I’m in the middle of developing a Auth.net payment module. It seems that also the test accounts are not working right now.
Did anybody of you try to send data to https://test.authorize.net/gateway/transact.dll in the last 12 hours?

[...] The biggest question I received was: “How could one of the biggest payment processors in the world be brought down by an incident at only one of their locations and why did they not have some type of redunduncy or backup systems in place that would prevent a the entire system from going down?” (Article at Tech Crunch) [...]

This might bee a serious catastrophe, I think paypal is the payment gateway of all the time, great service for a lower price and 100% uptime, and for the “free people” nice supoort also, c’mon Authorize.net its pretty dam bad really.

John
http://www.encuentratujob.com.mx/jobCountry/JobUsa.htm
————————-
US Employer

We are still getting a General Error when we try to process credit cards using Authorize Net

[...] the strange here: Authorize.net Goes Down, E-Commerce Vendors Left HangingSHARETHIS.addEntry({ title: “AUTHORIZE.NET GOES DOWN, E-COMMERCE VENDORS LEFT HANGING”, url: [...]

let me sign up to show features.

[...] According to reports, Authorize.net was down on July 3, 2009 from around 2:15 a.m. until Noon Eastern Time. Authorize.net set-up a twitter account to provide updates and respond to questions. [...]

I just registered at http://www.authorizenetclassaction.com

True, a very BAD thing to happen during the weekend. But the guys have rather handled it well and we hope the site is fully up and running soon.

dude.. http://www.authorizenetlawsuite.com/ seriously

[...] Goes Down-ShopVisible Clients Still Live and Selling Friday was a grim day for Authorize.Net. For online retailers using this payment gateway service provider, holiday selling plans online were [...]

Is Authorize down again or is the website just running slow? It happened about 2:30EST.

[...] reading about Authorize.net’s fun holiday weekend adventure of a fire in their data center, I decided that I better find a solution to automatically backup my servers data and database [...]

Since you asked…Commerce Lab from IP Commerce offers a hosted payments page that can be integrated to your existing process. http://commercelab.ipcommerce.com/Integration/Integration_Tools/Hosted_Payment_Page.aspx

The Commerce Hosted Payments Page presents a seamless payment process based on your CSS allowing full customization with your branding (or theme) and is a PCI-certified payment system.

We were just told by an Authorize.net rep that there would be no (i.e. zero) compensation for the outage. They wouldn’t even waive processing fees for the day of the outage, or credit us in any way for the downtime.

For anyone from Authorize.net reading this: “THIS IS TERRIBLE BUSINESS”. Between the downtime itself, near lack of communication about it, and total denial of any compensation, you’ve just given us serious incentive to find another provider.

We got that word too. They’re claiming it was an ‘act of god’. Right.

The fire department determined the cause of the fire to be improper maintenance on old equiptment, and who on earth has ever heard of a server bank with a SPRINKLER system instead of Halon?

Act of stupidity, not an act of God.

I use Trust Commerce like the other posts….used Authorize at one time. The switch was easy. I’ve had nothing but good experiences with Trust.

Why store it at all? Why take the liability?

Heard about this, but wasn’t aware it affected so many sites. I know a lot of merchants who use Authorize, but there are many gateway companies out there. Figured the market share would be a bit more spread out given there are more options now than ever. It’s a good idea to have a backup gateway or plan of attack. PayPal is a great alternative and you can use their gateway solution in much the same way as authorize. Thanks for sharing this, the comment were very interesting as well.

[...] this month, it seems Authorize.net was temporarily taken down by a fire at their datacenter. Thankfully, they had a disaster recovery [...]

[...] FormSpring arbeitet mit PayPal, First Data (in Deutschland ist die Firmentochter TeleCash* für ‘Internet Payment’ zuständig), Google Checkout und Authorize.net (dort gab es allerdings im Sommer Probleme). [...]

that would make the third fire at Fisher Plaza this year. http://www.king5.com/results?search=site&keywords=Fisher+Plaza&searchbutton=Go

Seems an odd location and track record for a major hosting site.