Saturday, July 2, 2011

Realtime Hadoop usage at Facebook: The Complete Story

I had earlier blogged about why Facebook is starting to use Apache Hadoop technologies to serve realtime workloads. We presented the paper at the SIGMOD 2011 conference and it was very well received.

Here is a link to the complete paper for those who are interested in understanding the details of why we decided to use Hadoop technologies, the workloads that we have on realtime Hadoop, the enhancements that we did to Hadoop for supporting our workloads and the processes and methodologies we have adopted to deploy these workloads successfully. A shortened version of the first two sections of the paper are also described in the slides that you can find here.

29 comments:

  1. On the flip side of the coin, Cloudera is also stepping up a notch with the announcement of Enterprise 3.5, the latest version of its Hadoop management offering. Some of the new features include real-time monitoring.

    ReplyDelete
  2. Nice blog post. I'm not that much aware of this thing but then, thanks.

    ReplyDelete
  3. Thank you..its very helpfull dhruba.. actually i'm planning to work with HBase

    ReplyDelete
  4. What a wonderful world. What a good job for this post. Very rich and constructive at the same time. I want to say a thumbs up to the creator for keeping this web site simple. Congratulations finally a web site of top-level. Have a nice day!

    ReplyDelete
  5. Finally! I've been waiting for months for Facebook to begin using an efficient technology like Apache Hadoop. I've noticed a real improvement in speed since this change has taken place. Keep up the good work Hadoop!

    ReplyDelete
  6. Hi Drubha,
    Thanks for the post and links to the white paper. Intersting read.

    Do you have any additional details you can share on muti data center replication. Do you have that in facebook today, if not what are some of the ideas on accomplishing a minimal downtime in case a data center is down.
    thanks

    ReplyDelete
  7. Thanks for such informative blog, i hope this blog is helpful for my plumber london business. I hope you like it.

    ReplyDelete
  8. @mridula, I do not yet have much details about multi-data center replication. We are working on such a product, will update you here when we get to some sane design :-)

    ReplyDelete
  9. Thanks for sharing the links.
    _________________
    Jobs In Brisbane

    ReplyDelete
  10. Hello Sir,

    Thanks for posting these files
    that was good. but I am not aware of these things.

    Can you suggest a project which is easy and done in 3 months time.

    Thankk you sir.

    ReplyDelete
  11. Is NFS irrelevant going forward? Can a linux node/cluster run on hdfs alone?

    ReplyDelete
  12. Apache HTTP is one of the most widely used Web server software in the world today, with more than 100 million Web sites using it. And because it is open source, it is a very dynamic and has become a robust Web server it is today. What is more, it is free to download and use.

    apache jobs

    ReplyDelete
  13. They could also switch your existing application servers to J2EE or Java platforms such as WebSphere, Weblogic or Apache.

    apache jobs

    ReplyDelete
  14. Thanks for sharing with us.This provide more useful information for us. I have one of my Recycling Center Chicago business, i hope you like it.

    ReplyDelete
  15. I would be very thankful if you continue with quality what you are serving right now with your blog…I really enjoyed it…and i really appriciate to you for this….its always pleasure to read so….Thanks for sharing!!

    ReplyDelete
  16. I have just found the Press Releases and news about Green Coffee Bean Max, Zeta Clear, Provillus, Garcinia Cambogia Select, Raspberry Ketone Max, Breast Actives, Phen375, Triactol, Venapro, Wartrol and African Mango Plus on top Social media sites and Newspaper, so i thought to share it with you guys!

    ReplyDelete
  17. Dr. OZ has recently endorsed 4 products on his Show as best Fat Burner Supplement for quick and Safe Weight Loss! Garcinia Cambogia Select, Green Coffee Bean Max, Raspberry Ketone Plus and African Mango Plus are Those Top 4 Dr OZ Weight Loss Pills!

    ReplyDelete
  18. Garcinia Cambogia Select is a breakthrough medical discovery in natural health and wellness and in sustainable fat loss and weight loss. Garcinia Cambogia is a weight loss and eating disorder remedy. It is a safe and effective non-stimulant that is made from all-natural fruit extracts that are being used at a low or high dosage depending upon the desired result. Garcinia Cambogia Select leads in melting off the awful inches from your waist. It makes you attain the desired shape and a gorgeous flat belly that you have always dreamt of! The Garcinia Cambogia Extract could be the most recent fat loss solution which comes highly recommended by non other that Dr Oz in fact he calls the Garcinia Cambogia Extract the Holy Grail Of Weight-loss. For more review about Dr OZ Garcinia Cambogia Select, Visit www.garciniacambogiaselects.com or find complete Garcinia Cambogia Select Review at http://www.prweb.com/releases/garciniacambogiaselect/reviews/prweb10331994.htm

    ReplyDelete
  19. Provillus is rated hair loss product of the year for men and women hair loss! Provillus is FDA approved solution to stop hair loss and regrow hair back! Read Provillus review before order at www.managehairloss.com

    ReplyDelete
  20. I just found Review on Adiphene, Triactol, Provillus and ZetaClear, Phen375, Garcinia Cambogia Select and Acai Plus with Detox Plus Today, so thought it would be great piece of information to share!

    ReplyDelete
  21. I am desire associated with examining rejuvenating content. Keep the great run! Excellent web page. My wife and i adored disregarding it?s fundamentally an incredible read for several. We now have included in our favswill lemon juice help acne
    Whenever i must condition, as being a pile while
    does Clear Skin Max work
    i precious researching what you may important to state,
    External Hemorrhoids causes
    Internal Hemorrhoids Symptoms
    When i won't aid nevertheless decline curiosity suitable just before a long time. For the reason that while you obtained any incredible love regarding the matter topic.

    ReplyDelete
  22. This discussion is very focused on the topic and I’m satisfied with the researched material as is is authentic and unbiased.

    ReplyDelete
  23. Really good piece of knowledge, I had come back to understand regarding your website from my friend Sumit, Hyderabad And it is very useful for who is looking for HADOOP.

    ReplyDelete