When The Merde Hits The Fan...

How can Drupal protect its reputation when projects go wrong?
France.fr website is down
France.fr website is down
France.fr, a major showcase for France as a tourist destination, went down after just one day. It was built by an agency that seems to have no prior Drupal experience. How can Drupal protect its name when any agency can claim to be 'Drupal specialists'? Steve Parks reports...

Drupal is becoming the CMS framework of choice as governments around the world move to using open source solutions, but this creates a problem - most governments’ approved suppliers lists are limited to large agencies that have previously only dealt with proprietary solutions.

This isn’t just a problem for the government agencies that commission sites only to encounter problems - it’s a big problem for the Drupal community too. The agencies concerned aren’t going to say ‘We blame our inexeprience with Drupal’, they’re going to say ‘This is the problem with open source solutions, let us show you our .Net solution’. So it’s Drupal that stands to lose credibility. The open source community has to deal with this issue head on to maintain the lead it’s building up in the public sector.

The latest example is the website France.fr - intended as a major showcase for tourism to France - but if you go to the website today you find this message:

We are in the final phase of our audit to find out the causes of the technical failures which led to us closing the site, and are close to finding a solution. We would like to thank all the web users who helped us spot the bugs.

We aim to have the site fully operational again in the second half of August, and can confirm that a version with a participatory dimension will be released in November.

Yoran Brault, a French Drupal developer (and author of the book ‘Concevoir et deployer ses sites web avec Drupal’), has been investigating and has blogged about his findings. France-Info has also been reporting the details, as has Le Monde, as well as many other outlets, and more information has been posted on the official Facebook page of France.fr. Here’s a summary:

  • The site was targeted at tourists. France hosts approximately 74 million tourists a year, but didn’t have a national visitor website until now;
  • The site budget is widely reputed to have been 860,000 Euros - although Le Monde says it was 1.6 million Euros;
  • It launched on the 14th July;
  • It was hosted on 11 servers, reportedly then doubled to 22, by host Cyberscope;
  • This, officials say, wasn’t enough to cope with a peak of 50,000 unique visitors a day, and the site developed problems;
  • The site at this stage did not involve any user login or interaction (these features were due to be added in November), so could have been served entirely from cache;
  • The site went down just 24 hours later on the 15th July. At first they said it would be back up in a few hours - but then the holding page message changed to that shown above, saying it would be ‘fully operational again in the second half of August’;
  • The head of hosting at Cyberscope is quoted as blaming the site developers: “Elle a été construite d’une manière qui ne correspond pas à l’usage” - translation: “It was built in a manner that doesn’t correspond with the level of usage”.

The Google cache of the page at www.france.fr/page/mentions-legales says:

The design and implementation, editorial production, development and integration of the pages have been made by the company Modedemploi in 2010.

Drupal can easily deal with such volumes of traffic, as has been seen on many other projects (just take a look at Lullabot’s client list for examples), especially as the site served only anonymous traffic (there was no user login). So in what way was the site “built in a manner that doesn’t correspond with the level of usage”? I set out to investigate.

Le Monde states that Modedemploi is an advertising agency, “more specialised in viral marketing than in the creation of sites”.

So I’ve contacted Modedemploi today with a list of questions, shown here in English:

  1. What part of the site build did Modedemploi do?
  2. Who were your partners in the project, and what parts of the build did they do?
  3. How much experience did the project team have with Drupal? (please provide links to other sites you’ve built if any)
  4. We can’t find any reference to Modedemploi having ever built a Drupal site. Was France.fr your first ever experience with a Drupal site?
  5. What platforms do you build the rest of your sites in? ASP.net? Something else?
  6. Why did you feel you were well placed to bid for the contract?
  7. What price did you charge for the site?
  8. What went wrong with France.fr to cause the site to go down?
  9. Why is the site going to be down for so long?
  10. What lessons have been learned?
  11. Specifically, what do you think other high traffic websites did right, that you didn’t get right with France.fr?
  12. Are you open to approaches from experienced Drupal developers to help fix the problem? Who should they contact?

I’ll keep you posted about any answers I receive (but I’m not holding my breath - they refused to comment to Le Monde).

I also want to hear more from the Drupal community. Do you know of any Modedemploi experience with Drupal? Do you have any more information about this site build? Do you know what the problem with the site is? Did any Drupal shops bid for the work but get turned down?

Please contact me with more information: editor [at] drupalradar [dot] com or tweet @drupalradar - and watch this space for more information.

But then the big question is - what can the Drupal community do about this? How can we make sure that Drupal doesn’t get a bad name because of bad implementations by large agencies inexperienced with Drupal? Do we need to provide them with training, better documentation or other help?

More than this though - I strongly believe we need to make sure Drupal shops are in a better position to get this work in the first place. But how do we achieve that?

Over to you in the comments…

Update

I’m already seeing some of the backlash against Drupal over this. ZDnets French site has an article about this story which includes experts saying that Drupal was not the right choice for this kind of site. I’ve also seen a number of tweets today like this one:

France.fr is unavailable since its launch and it’s built with Drupal #coincidence

Regardless of the fact that Drupal can very easily be configured for high traffic sites - a few bad projects like this could very seriously affect its reputation. What can be done to ensure quality?

Update: 1st August 2010

Dries Buytaert, Drupal’s project lead, has sent me an email with this comment:

I don’t know anything more about France.fr, really. I’ve not spoken with any of the people directly involved in this project. I’ve sent them an e-mail but never heard back. I don’t think they have reached out to anyone else in the Drupal community either. It certainly looks like they need help, but they aren’t asking for any, it seems.

We all know Drupal scales far beyond France.fr’s requirements so I’m not sure what the problem is. It is hard to comment on what happened with France.fr and how to avoid situations like that until we know and understand what went wrong.

DrupalRadar is still researching what went wrong with the France.fr project and plans to feature a detailed post analysing the problems and possible solutions. Please email us with any information or insight you have that may be useful to this article (in confidence if necessary): editor [at] drupalradar [dot] com

[Credits to: Greg Harvey of CMSPros for the tip-off and Alessandra Sturani for help with translation]

Comments

Interesting story - still doing better than the Italian equivalent which cost 36 million (and also does not work!). As for how to protect Drupal I think what you are doing is part of it (getting the story and facts straight) and also pointing to all the high traffic websites that do work with Drupal without problems.

As Drupal grows we cannot, as a community, be held responsible for every failure no more than PHP, Django, Rails or any other generic technology is held responsible. Now can we hope to have in place mechanisms that prevent that (a Drupal High Council of sorts that checks each website for quality assurance).

Most governments permitted suppliers lists are restricted to significant businesses which have previously only dealt with proprietary solutions. rf remote control transmitter

Really impressed! Whatever is the explanation very open and very clearly the issues. That actually contains the information. Your website is very useful. Thanks for sharing. Looking forward to more! There are many cool sites that are even specific to each phone. They should be more informative too after the run of competitive market.

Excellent write-up,your blog is certainly one of the most inspiring and informative blogs i’ve visited. Thank you for sharing!
logo design competition

I second this, such a insightful post. I am just reading up the articles in here and it just amazes me. be yourself quotes

this is such a great thing to know “We are in the final phase of our audit to find out the causes of the technical failures which led to us closing the site, and are close to finding a solution. We would like to thank all the web users who helped us spot the bugs” thanks so much for sharing…

background check

A very nice post with good information in it. I hope this blog stays up for a long time.

Cheers.

TheGateKeeper (Muay Thai | Self Defense Laws | Martial Arts for Children)
 Thanks

I believe the key is in being passionate about what you write, being focused, and being dedicated to get the work done. Of course, you an write about pretty much any topic, but you need to have faith and ability to get the job done.
research paper writer | write my research paper | research paper proposal

Thanks so much for this! I have not been this thrilled by a blog post for quite some time! You’ve got it, whatever that means in blogging. Anyway, You’re definitely someone that has something to say that people should hear. Keep up the wonderful job.
cell phone spying

fixed this happened in our country. results did not come late, most failure … after failure. Unfortunately, we had to compromise quite hard.

I am looking forward to reading new articles.
Scot

Great of meaning that you want to try to explain to online readers, I think this is contribute many of knowledgement.

commonwealth netbank

I completely agree with you. I have no point to raise in against of what you have said I think you explain the whole situation very well

I totally agree with you. Drupal must no blamed if all things that went wrong.

Kollagen|HGH Reviews|Best Wrinkle Creams

I just found your site and wanted to say that I have really enjoyed browsing your posts.
custom logo design

to be honest to the head, die-hard fans have been about lately. Bad answer, respond bellicose. nothing good in it. in place to encourage their favorites. very great sin

Very nice and helpful information has been given in this article. I like the way you explain the things. Keep posting. Thanks… I came across this post bookmarked and i definitely preferred a few things i examine. will truly take a note of it as well and in addition move through the some other.
Clubmz e-spy

This was really so very interesting and fun for me to read. I really do think that this was really very cool and fun to read. Thanks.
Cell Spyware

Very nice and helpful information has been given in this article. I like the way you explain the things. Keep posting. Thanks… I came across this post bookmarked and i definitely preferred a few things i examine. will truly take a note of it as well and in addition move through the some other.adult toys

Very nice and helpful information has been given in this article. I like the way you explain the things. Keep posting. Thanks… I came across this post bookmarked and i definitely preferred a few things i examine. will truly take a note of it as well and in addition move through the some other.reverse mortgage calculator,

Really interesting article, and truly engaging, I must add. I’ve subscribed to your RSS so I can look forward to more good posts from you. best kids apps

I was very encouraged to find this site. I wanted to thank you for this special read. I definitely savored every little bit of it and I have you bookmarked to check out new stuff you post.. nursing colleges

InterLawyeresting story - still doing better than the Italian equivalent which cost 36 million (and also does not work!). As for how to protect Drupal I think what you are doing is part of it (getting the story and facts straight) and also pointing to all the high traffic websites that do work with Drupal without problems.

Good article! Thank you so much for sharing this post. Your views truly open my mind.
Free Calls Bangladesh | O2 Calls

As Drupal grows we cannot, as a community, be held responsible for every failure no more than PHP, Django, Rails or any other generic technology is held responsible. Now can we hope to have in place mechanisms that prevent that (a Drupal High Council of sorts that checks each website for quality assurance

Some interesting thoughts on the subject. Looking forward to see what else you post in the future.
Massage London

I hope there will be more successful your site is very wonderful site. porna Thanks to the efforts of.

Interesting story - still doing better than the Italian equivalent which cost 36 million (and also does not work!). As for how to protect Drupal I think what you are doing is part of it (getting the story and facts straight) and also pointing to all the high traffic websites that do work with Drupal without problems. Lawyer

Stumbled upon your website while I was searching for information and facts for our internet site, obviously this isnt all releative however its interesting how one can reach a web site accidentally and find your self looking at something different but just as useful. I tend to get diverted a really often like this.Visiting Peru

Great blog article about this topic,I have been lately in your blog once or twice now.I just wanted to say hi and show my thanks for the information provided. clubmz reviews

Stumbled upon your website while I was searching for information and facts for our internet site, obviously this isnt all releative however its interesting how one can reach a web site accidentally and find your self looking at something different but just as useful. I tend to get diverted a really often like this.
braces cost

Stumbled upon your website while I was searching for information and facts for our internet site, obviously this isnt all releative however its interesting how one can reach a web site accidentally and find your self looking at something different but just as useful. I tend to get diverted a really often like this.
braces cost

Stumbled upon your website while I was searching for information and facts for our internet site, obviously this isnt all releative however its interesting how one can reach a web site accidentally and find your self looking at something different but just as useful. I tend to get diverted a really often like this.

Let’s say that each unique user visits four pages. I would say that four pages per visit is probably being generous but just to show the numbers.http://kustomkulture.info

Hey dude, was just looking through the internet looking for a bit of information and came across your site. I am very impressed by the information that you have written on this page. It tells me how good you understand this subject. Bookmarked this page, and will come back for more. My friend, YOU ROCK!!! mba programs in california

PIC16F877A Thank you for making the sincere effort to talk about this. I feel very strong approximately it and would like to learn more. If its OK, as you gain more extensive knowledge, might you thoughts including extra articles similar to this one with additional information? It could be extremely useful and helpful for me and my friends.

Nonsense is everything, so we pauperism to sure that there are values on what we are sex to fact our subscribers participating. Trait them vexed is one perception we moldiness connectedness do!Quotes About Love

It is a good article and inspires everyone to read it completly. This is indeed very interesting and attract the readers. I appreciate your thoughts……..Funny Life Quotes

Drupal Radar is very good. They can be a tool for us to use for our website. refurbished dishwasher

I have been really glad after reading this blog as the knowledge which has been given via this blog is simply tremendous. I would congratulate and appreciate the blogger for doing this much hard work.

I just wanted to comment your blog and say that I really enjoyed reading your blog post here. It was very informative and I also digg the way you write! Keep it up and I’ll be back to read more soon mate
Writing Essay || Coursework Help || University Essays || Do My Assignment For Me || Help With C++ assignment || Statistics Help || Cover Letter Service || Essay Editing || Graduation Thesis || Cv Editing || Scholarship Essay

If there are problems with humidity of air, use volcano vaporizer

Wow 11 servers for 50,000 daily uniques.

Let’s say that each unique user visits four pages. I would say that four pages per visit is probably being generous but just to show the numbers.

That adds up to 200,000 pageviews spread across 11 servers.

Roughly per Drupal server:
Per hour that is 1389
Per minute that is 24
Per second that is 0.4

Those numbers are based on the setup having 5 mysql servers and 6 drupal servers serving the content in some sort of master/slave configuration. Any of us who have used Pressflow in Master/Slave configurations know that the master still takes most of the load by default.

If they only had one mysql server and 10 drupal servers then we get

Roughly per Drupal server:
Per hour that is 834
Per minute that is 14
Per second that is 0.25

My opinion is that, within reason, no matter how many servers were thrown at this site it would have blown up. Perhaps the theme registry was being rebuilt on every page? Perhaps a view was poorly written, or constructed, that was fetching huge amounts of data to display only a small portion of it. Perhaps no load testing was done against a site that was clearly going to attract good traffic levels.

Who knows? We probably won’t find out whether they developers ever visited the Performance page.

He might *invite* some other, smaller, specialist businesses so he can say he did if anyone asks, but he’ll have a whole host of reasons why he couldn’t possibly use them because they represent to high a “risk” to the tax payer’s money.youtube videos

Doesn’t Acquia sort of want to be the Drupal High Council, with the partnership system, etc. I’m not sure how well that works as a mark of quality though. Depends on how diligently partners are checked - especially the higher ones, but frankly, at every level. (Staff CVs scrutinised, references checked, source code/key site set-up examined even?)

And anyway, does it matter if a measure of quality, trusted and agreed upon, exists if purchasing managers in large government departments aren’t aware of it and still just “call Capita” (to quote David Cameron, when he was in opposition). If Sir Bert is already set on hiring Big Outsourcing Group PLC, he will.

He might *invite* some other, smaller, specialist businesses so he can say he did if anyone asks, but he’ll have a whole host of reasons why he couldn’t possibly use them because they represent to high a “risk” to the tax payer’s money. Ha!

You either need to be able to change the attitude of government purchasing managers (something David Cameron says he is committed to doing, but I don’t think he can - governments come and go but the same civil servants are there for decades - they know they don’t really have to listen to the politicians…) *or* you have to tool up to take on the big outsourcing and advertising people head on. Tooling up might mean hiring specialist consultants in winning government business to write your RFPs for you. It might mean slipping on a suit and a fake moustache and re-branding, but you have to do something.

IMHO. ;-)

This is the problem with open source solutions, let us show you online bachelors degree
our .Net solution’. So it’s Drupal that stands to lose online doctorate degree credibility. The open source community has to deal with this issueonline associates degree head on to maintain the lead it’s building up in the public sector.online undergraduate certificate

He might *invite* some other, smaller, specialist businesses so he can say he did if anyone asks, but he’ll have a whole host of reasons why he couldn’t possibly use them because they represent to high a “risk” to the tax payer’s money. Ha!Seo
Ecommerce
Ecommerce solution
Web Designing
Liposuction
Company formation
SEO
web designer
seo services
Local seo | local seo services
PPC
Virtual offices
Virtual address
ecommerce cms
ecommerce website

So far I visit your blog and found very informative post and your sharing is such a useful for me and other really thanks for sharing such a great post. essays | term papers | research papers

Post new comment