tag:blogger.com,1999:blog-8556003786324804973.post3318542386990715590..comments2024-01-27T03:43:02.667-08:00Comments on HDFS: Hadoop and Solid State DrivesDhruba Borthakurhttp://www.blogger.com/profile/10832366855372649190noreply@blogger.comBlogger31125tag:blogger.com,1999:blog-8556003786324804973.post-2354546875855098912014-01-06T20:58:40.325-08:002014-01-06T20:58:40.325-08:00Very nice article Dhruba. Hadoop as is will not ge...Very nice article Dhruba. Hadoop as is will not get much speed up using SSD as you described, HBase may be slightly better. But I see Spark / Shark should get a significant speed up using SSD. <br />I wanted to get your thoughts on Spark / Shark Suraj Waghuldehttps://www.blogger.com/profile/16695272171725231126noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-19564750434691256212013-09-27T05:14:36.242-07:002013-09-27T05:14:36.242-07:00This comment has been removed by a blog administrator.Anonymoushttps://www.blogger.com/profile/15897344725999558513noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-79495133791378784932013-09-24T05:16:09.945-07:002013-09-24T05:16:09.945-07:00This comment has been removed by the author.Anonymoushttps://www.blogger.com/profile/05484923246522375928noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-77665392425048936032013-08-04T20:39:50.131-07:002013-08-04T20:39:50.131-07:001. yes
2. yes
3. I do not think that the improveme...1. yes<br />2. yes<br />3. I do not think that the improvement in overall job latency will decrease by 5X. The copy phase is something that is a big bottleneck that could be partially addressed by a fast storage for map outputs. Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-14485020913264652132013-08-04T19:27:19.874-07:002013-08-04T19:27:19.874-07:00Dhruba,
This is one of the most informative, pre...Dhruba, <br /><br />This is one of the most informative, precise, concise and well written posts on Hadoop/SSD topic. Kudos and thanks for sharing!<br /><br />What are your thoughts on the following regarding harnessing SSDs smartly for Hadoop:<br /><br />1. Would server-side SSD caching (read only I/Os) be a better and cost-effective performance boost alternative to using SSD as Hadoop/HBase storage device? This could potentially avoid SSD write pitfalls.<br /><br />2. Regarding your observation "In a data warehouse, a large portion of queries access only recent data...Another use case would be to put map outputs on SSDs, this could potentially reduce map-output-fetch times" - well, again, would it make more sense to deploy SSD read-only cache for frequently accessed 'recent' Map output instead of using SSD as a storage device for that data? <br /><br />3. Another aspect to look at is - Let's assume that SSD itself (whether used as storage or cache) brings in 5X I/O performance boost over HDD when Map output is harnessing SSD. In the overall Hadoop workflow and scheme of things - does it matter? Is I/O boost an incremental performance boost to overall Hadoop application or does it translates to say ~5X overall application performance boost? <br /><br />Look forward to your response.<br /><br />Thanks.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-10481800119407466782013-06-29T04:26:06.532-07:002013-06-29T04:26:06.532-07:00This comment has been removed by the author.Anonymoushttps://www.blogger.com/profile/05828706836812988577noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-22892521543281013562013-06-28T01:37:39.308-07:002013-06-28T01:37:39.308-07:00Quick question on HBase benchmark - you said 45 % ...Quick question on HBase benchmark - you said 45 % cpu with 30K ops per second. Did you try with 2 region server processes to see if you could push this to 70K qps and cpu to 90 %. Running two processes would help you get some freedom from concurrency issues.<br /><br />Another comment - is the workload batched multi gets for HBase ?Varun Sharmahttps://www.blogger.com/profile/08488650712521430947noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-28778327113393862872013-01-18T09:51:31.998-08:002013-01-18T09:51:31.998-08:00Hi Alex ,the FB version of HDFS (https://github.co...Hi Alex ,the FB version of HDFS (https://github.com/facebook/hadoop-20) has some fixes that improves random read performance of local short-circuit reads from a mere 92K ops/sec (when I measured it) to a 1million reads/sec. I do not know what those particular fixes are, but I am assuming that HDFS will make some significant improvements to latency going forward.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-53714850315358253722013-01-18T05:27:48.437-08:002013-01-18T05:27:48.437-08:00Any updates on this in the months since May? I am ...Any updates on this in the months since May? I am considering SSD for Hadoop but this basically tells me not to bother...Anonymoushttps://www.blogger.com/profile/15847604769667639885noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-40698960509717179412012-10-19T22:56:30.049-07:002012-10-19T22:56:30.049-07:00Very nice one..Very nice one..Ramkrishnahttps://www.blogger.com/profile/04410426244006877440noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-56408230364088949552012-09-13T04:43:12.970-07:002012-09-13T04:43:12.970-07:00I am relatively new to HDFS administration. I am ...I am relatively new to HDFS administration. I am really struggling to find good information on ideal HDFS block sizing to support the efficient operation of MapReduce jobs as well as HBase datafile/index storage. Any suggestions on good resources? Thanks<br />Jon<br />jfortneyhttps://www.blogger.com/profile/02772501201065633490noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-47136283497140885812012-08-03T10:36:50.360-07:002012-08-03T10:36:50.360-07:00@Vladimir: I have not tried MapR. And we have not ...@Vladimir: I have not tried MapR. And we have not yet committed the above-mentioned-change to Apache hdfs.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-56159118464381917552012-08-03T09:40:27.292-07:002012-08-03T09:40:27.292-07:00" We found that the read-shortcircuit code pa..." We found that the read-shortcircuit code path spent considerable time in DNS lookup calls and updating metric-counters. We fixed these two pieces of code and observed that HDFS could sustain a peak random-read-throughput of around 92K ops/sec, the CPUs was now close to 95% usage. "<br /><br />Have you committed your changes back to Apache?Vladimir Rodionovhttps://www.blogger.com/profile/18159080975174540442noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-46882575622929738002012-08-03T09:37:01.446-07:002012-08-03T09:37:01.446-07:00Dhruba,
Have you tried MapR? Definitely, won'...Dhruba,<br /><br />Have you tried MapR? Definitely, won't help improve HBase numbers, but should improve raw DFS IO and reduce CPU%%.Vladimir Rodionovhttps://www.blogger.com/profile/18159080975174540442noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-20506745230304693252012-06-06T02:23:05.744-07:002012-06-06T02:23:05.744-07:00I am really amazed by this site and really gonna s...I am really amazed by this site and really gonna share this site to my friends.Custom Facebook Applicationhttp://www.socialcubix.com/services/facebook-application-developmentnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-35837504247480625222012-05-14T15:22:45.293-07:002012-05-14T15:22:45.293-07:00@Sujoy: you are absolutely right. In fact, we curr...@Sujoy: you are absolutely right. In fact, we currently run multiple servers instances per SSD just to be able to utilize all the IOPs. This is kindof-a-poor man's solution to the problem. Also, you have to have enough CPU power on the server to be able to drive multiple database instances on the same machine.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-54232676733579894632012-05-14T13:02:29.535-07:002012-05-14T13:02:29.535-07:00The last comment was by me (Sujoy)...not sure why ...The last comment was by me (Sujoy)...not sure why it showed up as unknown.Sujoyhttps://www.blogger.com/profile/05655988445627582166noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-9860843490911602462012-05-14T13:01:15.444-07:002012-05-14T13:01:15.444-07:00Thanks for the article Dhruba.
If the SSD could be...Thanks for the article Dhruba.<br />If the SSD could be shared so that a handful of servers would share the 200K+ IOPS of a SSD (thereby ammortizing the cost), then do you thing that a jump from 400 IOPs (from HDDS) to 35-92K IOPS on a SSD would help a real workload on Hbase/HDFS?Sujoyhttps://www.blogger.com/profile/05655988445627582166noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-70904958105669658692012-05-08T11:32:02.975-07:002012-05-08T11:32:02.975-07:00Agree again, Matt. A HBase workload that currently...Agree again, Matt. A HBase workload that currently runs on spinning disks could run much better if/when the data is served from SSDs. But when it is time to benchmark/compare HBase with some other database on SSD, then there is lots to be done.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-84543961167018380972012-05-07T23:30:29.618-07:002012-05-07T23:30:29.618-07:00Even if it's poorly utilized, could the SSD st...Even if it's poorly utilized, could the SSD still be a big improvement? Maybe you only get 4k reads/second, but if your app was previously bottlenecked by the disks at 400 reads/second then it could make a huge difference to the overall application.mcorganhttps://www.blogger.com/profile/07373089632944145014noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-75607520578500954332012-05-07T22:43:19.749-07:002012-05-07T22:43:19.749-07:00Agree with you Matt. A hybrid storage reduces the ...Agree with you Matt. A hybrid storage reduces the storage cost compared to a pure-SSD based storage. But the thing is that an application can only do a few thousand 16K random read from a HDFS datanode. So, you won't be able to utilize the max potential of the iops offered by the SSD.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-18310310028307871592012-05-07T22:35:58.173-07:002012-05-07T22:35:58.173-07:00Following on to Andrew's comment about hybrid ...Following on to Andrew's comment about hybrid storage, one potential setup would be to store the primary replica of a file on SSD and the 2 backup replicas on spinning disks. The write path in hbase is always sequential where spinning media does ok. If the regionserver can work with hdfs to localize and read from the copy of a file that is on an SSD, then you get most of the benefits of the SSD at a third of the cost.<br /><br />You could then go further and get more granular about what's stored on the SSD's: certain tables, certain regions, non-datablocks, evicted block cache entries, etc. But, seems like storing just the primary replica on SSD could buy you a lot relative to the implementation complexity.mcorganhttps://www.blogger.com/profile/07373089632944145014noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-46941355444566948902012-05-07T15:54:48.717-07:002012-05-07T15:54:48.717-07:00Thanks for your comments Andrew. It is very much p...Thanks for your comments Andrew. It is very much possible to make HDFS expose APIs so that data can go to spinning disks and other metadata objects to SSD. It will possibly improve HBase performance no doubt. And it is worth doing. The one question that I am still doubtful is that if we get to machines with more and more cores, than some of our existing database technologies might not be able to effectively utilize all the cores.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-54021702473862867482012-05-07T15:08:19.832-07:002012-05-07T15:08:19.832-07:00... of course I am not talking about a pure (cold)...... of course I am not talking about a pure (cold) read workload, but something more akin to FlashStore: http://www.vldb.org/pvldb/vldb2010/papers/I04.pdfAndrew Purtellhttps://www.blogger.com/profile/14087522700337885728noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-69492597339169924092012-05-07T15:02:10.094-07:002012-05-07T15:02:10.094-07:00If we consider a HDFS that adds storage device typ...If we consider a HDFS that adds storage device type information to volume and block metadata, and an extension of the HDFS API to specify storage device affinity, then we might see an HBase that stores data to spinning media, but WALs and flush files and other short lived and frequently accessed objects to SSD. Do you think we may see enough benefit with this, plus the (theoretical) improvements you mention, to make a hybrid storage architecture with Hadoop+HBase+SSD+SATA make some sense?Andrew Purtellhttps://www.blogger.com/profile/14087522700337885728noreply@blogger.com