Saturday, July 2, 2011

Realtime Hadoop usage at Facebook: The Complete Story

I had earlier blogged about why Facebook is starting to use Apache Hadoop technologies to serve realtime workloads. We presented the paper at the SIGMOD 2011 conference and it was very well received.

Here is a link to the complete paper for those who are interested in understanding the details of why we decided to use Hadoop technologies, the workloads that we have on realtime Hadoop, the enhancements that we did to Hadoop for supporting our workloads and the processes and methodologies we have adopted to deploy these workloads successfully. A shortened version of the first two sections of the paper are also described in the slides that you can find here.

20 comments:

  1. On the flip side of the coin, Cloudera is also stepping up a notch with the announcement of Enterprise 3.5, the latest version of its Hadoop management offering. Some of the new features include real-time monitoring.

    ReplyDelete
  2. Nice blog post. I'm not that much aware of this thing but then, thanks.

    ReplyDelete
  3. Thank you..its very helpfull dhruba.. actually i'm planning to work with HBase

    ReplyDelete
  4. What a wonderful world. What a good job for this post. Very rich and constructive at the same time. I want to say a thumbs up to the creator for keeping this web site simple. Congratulations finally a web site of top-level. Have a nice day!

    ReplyDelete
  5. Finally! I've been waiting for months for Facebook to begin using an efficient technology like Apache Hadoop. I've noticed a real improvement in speed since this change has taken place. Keep up the good work Hadoop!

    ReplyDelete
  6. Hi Drubha,
    Thanks for the post and links to the white paper. Intersting read.

    Do you have any additional details you can share on muti data center replication. Do you have that in facebook today, if not what are some of the ideas on accomplishing a minimal downtime in case a data center is down.
    thanks

    ReplyDelete
  7. @mridula, I do not yet have much details about multi-data center replication. We are working on such a product, will update you here when we get to some sane design :-)

    ReplyDelete
  8. Thanks for sharing the links.
    _________________
    Jobs In Brisbane

    ReplyDelete
  9. Hello Sir,

    Thanks for posting these files
    that was good. but I am not aware of these things.

    Can you suggest a project which is easy and done in 3 months time.

    Thankk you sir.

    ReplyDelete
  10. Is NFS irrelevant going forward? Can a linux node/cluster run on hdfs alone?

    ReplyDelete
  11. Apache HTTP is one of the most widely used Web server software in the world today, with more than 100 million Web sites using it. And because it is open source, it is a very dynamic and has become a robust Web server it is today. What is more, it is free to download and use.

    apache jobs

    ReplyDelete
  12. They could also switch your existing application servers to J2EE or Java platforms such as WebSphere, Weblogic or Apache.

    apache jobs

    ReplyDelete
  13. Thanks for sharing with us.This provide more useful information for us. I have one of my Recycling Center Chicago business, i hope you like it.

    ReplyDelete
  14. I would be very thankful if you continue with quality what you are serving right now with your blog…I really enjoyed it…and i really appriciate to you for this….its always pleasure to read so….Thanks for sharing!!

    ReplyDelete
  15. I am desire associated with examining rejuvenating content. Keep the great run! Excellent web page. My wife and i adored disregarding it?s fundamentally an incredible read for several. We now have included in our favswill lemon juice help acne
    Whenever i must condition, as being a pile while
    does Clear Skin Max work
    i precious researching what you may important to state,
    External Hemorrhoids causes
    Internal Hemorrhoids Symptoms
    When i won't aid nevertheless decline curiosity suitable just before a long time. For the reason that while you obtained any incredible love regarding the matter topic.

    ReplyDelete
  16. This discussion is very focused on the topic and I’m satisfied with the researched material as is is authentic and unbiased.

    ReplyDelete