Saturday, July 2, 2011

Realtime Hadoop usage at Facebook: The Complete Story

I had earlier blogged about why Facebook is starting to use Apache Hadoop technologies to serve realtime workloads. We presented the paper at the SIGMOD 2011 conference and it was very well received.

Here is a link to the complete paper for those who are interested in understanding the details of why we decided to use Hadoop technologies, the workloads that we have on realtime Hadoop, the enhancements that we did to Hadoop for supporting our workloads and the processes and methodologies we have adopted to deploy these workloads successfully. A shortened version of the first two sections of the paper are also described in the slides that you can find here.
Posted by Dhruba Borthakur at 6:02 PM
Labels: ,

20 comments:

  1. driving school worcesterJuly 4, 2011 at 1:55 AM

    On the flip side of the coin, Cloudera is also stepping up a notch with the announcement of Enterprise 3.5, the latest version of its Hadoop management offering. Some of the new features include real-time monitoring.

    ReplyDelete
  2. plumbingJuly 9, 2011 at 2:44 AM

    Nice blog post. I'm not that much aware of this thing but then, thanks.

    ReplyDelete
  3. vamsJuly 12, 2011 at 3:24 AM

    Thank you..its very helpfull dhruba.. actually i'm planning to work with HBase

    ReplyDelete
  4. SocialkikJuly 20, 2011 at 4:40 AM

    What a wonderful world. What a good job for this post. Very rich and constructive at the same time. I want to say a thumbs up to the creator for keeping this web site simple. Congratulations finally a web site of top-level. Have a nice day!

    ReplyDelete
  5. Driving Schools WorcesterSeptember 13, 2011 at 3:57 AM

    Finally! I've been waiting for months for Facebook to begin using an efficient technology like Apache Hadoop. I've noticed a real improvement in speed since this change has taken place. Keep up the good work Hadoop!

    ReplyDelete
  6. mridula@ning.comOctober 19, 2011 at 1:19 PM

    Hi Drubha,
    Thanks for the post and links to the white paper. Intersting read.

    Do you have any additional details you can share on muti data center replication. Do you have that in facebook today, if not what are some of the ideas on accomplishing a minimal downtime in case a data center is down.
    thanks

    ReplyDelete
  7. Dhruba BorthakurOctober 22, 2011 at 12:06 AM

    @mridula, I do not yet have much details about multi-data center replication. We are working on such a product, will update you here when we get to some sane design :-)

    ReplyDelete
  8. mridula@ning.comNovember 1, 2011 at 4:56 PM

    Thanks :)

    ReplyDelete
  9. AnonymousDecember 11, 2011 at 8:49 AM

    Thanks for sharing the links.
    _________________
    Jobs In Brisbane

    ReplyDelete
  10. Edward SagayarajJanuary 4, 2012 at 7:58 PM

    Hello Sir,

    Thanks for posting these files
    that was good. but I am not aware of these things.

    Can you suggest a project which is easy and done in 3 months time.

    Thankk you sir.

    ReplyDelete
  11. AnonymousJanuary 7, 2012 at 7:36 AM

    Is NFS irrelevant going forward? Can a linux node/cluster run on hdfs alone?

    ReplyDelete
  12. alfredNovember 3, 2012 at 2:55 AM

    Apache HTTP is one of the most widely used Web server software in the world today, with more than 100 million Web sites using it. And because it is open source, it is a very dynamic and has become a robust Web server it is today. What is more, it is free to download and use.

    apache jobs

    ReplyDelete
  13. alfredNovember 3, 2012 at 2:59 AM

    They could also switch your existing application servers to J2EE or Java platforms such as WebSphere, Weblogic or Apache.

    apache jobs

    ReplyDelete
  14. Teen Social Networking SitesNovember 3, 2012 at 3:26 AM

    Hi nice .. Thanks for this posting..

    Thanks

    ReplyDelete
  15. Recycling Center ChicagoNovember 7, 2012 at 1:46 AM

    Thanks for sharing with us.This provide more useful information for us. I have one of my Recycling Center Chicago business, i hope you like it.

    ReplyDelete
  16. Air Conditioning Repair Bonita SpringsDecember 5, 2012 at 1:43 AM

    Nice blog post. Air Conditioning Repair Bonita Springs

    ReplyDelete
  17. ferrugiamechanicalDecember 6, 2012 at 2:44 AM

    Thanks for sharing the links.... AC Service The Woodlands

    ReplyDelete
  18. SocialkikDecember 23, 2012 at 11:10 PM

    I would be very thankful if you continue with quality what you are serving right now with your blog…I really enjoyed it…and i really appriciate to you for this….its always pleasure to read so….Thanks for sharing!!

    ReplyDelete
  19. AnonymousMarch 1, 2013 at 1:09 PM

    I am desire associated with examining rejuvenating content. Keep the great run! Excellent web page. My wife and i adored disregarding it?s fundamentally an incredible read for several. We now have included in our favswill lemon juice help acne
    Whenever i must condition, as being a pile while
    does Clear Skin Max work
    i precious researching what you may important to state,
    External Hemorrhoids causes
    Internal Hemorrhoids Symptoms
    When i won't aid nevertheless decline curiosity suitable just before a long time. For the reason that while you obtained any incredible love regarding the matter topic.

    ReplyDelete
  20. Facebook Connect ApplicationApril 22, 2013 at 10:55 PM

    This discussion is very focused on the topic and I’m satisfied with the researched material as is is authentic and unbiased.

    ReplyDelete

Subscribe to: Post Comments (Atom)