I hate it when people want to push code on a Friday. Here it is, Friday, I was working to wind up a few last tasks before going home when my phone went off saying part of my company’s site was not working right.
After some investigation with a developer we discovered it was an issue with Facebook (ugh, how I hate thee) and their code was breaking because Facebook was broken (they were not aware this would happen I am sure they will fix it going forward).
So while they are working to work around the issue I ran a search for it, and it seems to be a more wide spread problem caused by a Facebook software deployment done today, Friday at nearly 7PM!
F F S
My monitor first detected the failure at 6:49 PM so they weren’t even done deploying by the time it failed. It was intermittant for a few minutes then went hard down at around 7:11PM.
@$#$ facebook. Thanks for screwing me, and who knows how many others by deploying code on a Friday night.
Of course the developers on our end deserve some of the heat as well. But I am nit picking about code deployments on a Friday, not code bugs..
Pretty common for the big guys who operate with Devops. Most of them are deploying code at all times of the day. Continus Deployment/Integration technologies are good, but only as good as your weakest test case.
The key for your side is to gracefully fail on your website when the third party you rely on isn’t working for some reason.
Too often have a I ran into situations where customers demand maintenance pages, etc for when our service will be unavailable. My comment is always.. “why have you not designed your interface with us as a toggle, that you can flip on and off, and not have your customer suffer.
I may blog post about this later this week…
Comment by Justin — February 6, 2012 @ 2:10 pm
yeah probably true. I’ve never been a fan of the whole ‘devops’ concept. I’ve been working very closely with developers for many years now(going back to 2006, prior to that was at a company where there was more firewalls in place between teams). I think working closely with them is good but for sure believe better lines need to be drawn about when and how stuff gets put into production. Now if I don’t get paged and don’t have to respond to the problems then I don’t care.
A few weeks ago my company pushed out an ‘important update’ to our web site on a Sunday afternoon (I wasn’t involved). Guess what, by Monday morning at 10AM we were rolling it back because it was destroying our database (no load on Sunday).
I remember one deployment in particular at my last company, a bit over a year ago, around Christmas time. The code was supposed to go to prod at around 3PM — at 10:30 *PM* they had *just* gotten it to work in the QA environment. The CTO had a meeting with the team at around 11:30 and said – we can push it to prod now, or wait till tomorrow morning. I suggested a radical idea – let it sit and bake in QA for a week and then push it to prod. He said “we can’t do that we have to push it soon”. I said “Don’t be surprised if it blows up”. He said “Is it going to blow up?”, I said “I don’t know but don’t be surprised if it does”.
So we took a hard downtime the following day at around noon for 30 minutes to deploy the code.
90 minutes into the 30 minute downtime we decided to roll half of the code back because it didn’t work. Then we brought the site up. About an hour after that the developers were frantically trying to fix the other half of the code that wasn’t working.
Rinse and repeat over..and over..and over..I suppose you could say I’m jaded against these sort of rushed schedules, forced deployments etc. I’ve been burned so many times I stopped keeping track years ago.
The company with the firewalls was quite complex, it was probably the opposite of everywhere else I have worked. There were multi month development cycles, towards the end of my time there we would spend 2-3 weeks preparing for a production release, all hands on deck for an all nighter while we released, and we still had major issues mainly due to the complexity of the app (nearly 1000 XML-based config files, ugh the horror). Of course, everything was managed and deployed by hand, nobody had the time to write tools or make things better.
Comment by Nate — February 6, 2012 @ 2:38 pm
Finally got my thoughts posted.
http://www.justinbrodley.com/code-deployment
Leaving my current gig and moving onto something new, but transitions are involved.
Comment by Justin — February 29, 2012 @ 1:23 pm