tag:blogger.com,1999:blog-8556003786324804973.post7208542049539081164..comments2024-01-27T03:43:02.667-08:00Comments on HDFS: HDFS block replica placement in your hands now!Dhruba Borthakurhttp://www.blogger.com/profile/10832366855372649190noreply@blogger.comBlogger26125tag:blogger.com,1999:blog-8556003786324804973.post-42632815524996667972014-05-22T09:28:29.079-07:002014-05-22T09:28:29.079-07:00does hadoop version 2.2 onward support this?does hadoop version 2.2 onward support this?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-83523218468340759862014-05-22T09:27:49.409-07:002014-05-22T09:27:49.409-07:00Does Hadoop 2.2 and on support this feature?Does Hadoop 2.2 and on support this feature?Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-61081496158482386752013-09-08T01:25:55.314-07:002013-09-08T01:25:55.314-07:00Hello Sir,
I am implementing my own replicating po...Hello Sir,<br />I am implementing my own replicating policy for hadoop which is work for my desertion,plz guide me and link that u have mentioned is not working "404 page not found" error.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-74146155940284116352013-09-04T22:29:25.565-07:002013-09-04T22:29:25.565-07:00Sir, I have an one doubt if one file of block stor...Sir, I have an one doubt if one file of block stored in same node then how will you get the scalablity at the processing time in context of parallel processing?rvihttps://www.blogger.com/profile/18019578323349073883noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-35730243885275913122013-08-07T01:55:38.188-07:002013-08-07T01:55:38.188-07:00hey can anyone tell me how can u limiting the bloc...hey can anyone tell me how can u limiting the blocks to be located on a single node??i am working on this rite nw..Anonymoushttps://www.blogger.com/profile/08638187165535216178noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-43738971197481315712013-03-28T06:52:55.872-07:002013-03-28T06:52:55.872-07:00 we would like store 1 of the 3 replicas off-site ... we would like store 1 of the 3 replicas off-site for (possible) recovery on a site<br />failure.<br /><br /><br />we have HDFS cluster as two replication in two different racks( rack0 , rack1). Now we need to keep one more replica iin aonther gepgraphical location for backup.Is it possible to configure? Do you know datacenter rack awareness in hdfs?Anonymoushttps://www.blogger.com/profile/09886544254591642465noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-3108982581300751402013-01-28T07:51:57.312-08:002013-01-28T07:51:57.312-08:00where can i find details on how to use this plug-i...where can i find details on how to use this plug-in??<br />My requirement is actually very simple.When i place a block, i want to place it in the same datanode as another existing block. So i jus want to retreive the location of the previous block( using FileSystem) and force hdfs to place the new block in the same set of nodes. Will this plug-in be useful for my purpose??<br />Is there a simpler way to achieve what i want to implement??Anonymoushttps://www.blogger.com/profile/18294224900789200391noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-57488487692561098732013-01-17T09:31:08.046-08:002013-01-17T09:31:08.046-08:00Reetha, HDFS does bufferedIO which means data in H...Reetha, HDFS does bufferedIO which means data in HDFS files will be cached in the OS buffer. That is the reason why your CopyToLocal times might be faster the second time around.<br /><br />Regarding "why reading from slave2" is much faster, I have no idea. You could analyze network traffic, disk IO, etc that was happening simultaneously at the time of your test run to figure out what was going on.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-32411871850604733162013-01-16T22:18:14.087-08:002013-01-16T22:18:14.087-08:00Hi
I have a small hadoop experimental setup with ...Hi <br />I have a small hadoop experimental setup with 1 Namenode and 3 Datanodes(say slave1, slave2, slave3) and replication factor of two<br />I did a CopyFromLocal command to store a small file into hadoop<br />When I tried hadoop fsck -files -blocks -racks , I found it is stored in slave1 and slave3<br />then when I tried CopyToLocal from slave1 and slave3 (where the data is stored) It takes more time than from slave2( where data is not present)<br />I checked "duration" in datanode log file as well as <br /><br />And one more thing for the first CopyToLocal it takes more time (27341012) than corresponding CopyToLocal(562176)<br /><br />Please tell me why it is so ?<br />ReethaAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-32611774676590663432013-01-07T22:28:18.201-08:002013-01-07T22:28:18.201-08:00thanks for fast response ,a portion is missed in t...thanks for fast response ,a portion is missed in the question posting it again <br />RamAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-20668957090402380572013-01-07T22:24:47.991-08:002013-01-07T22:24:47.991-08:00Dhruba,
Can you modify your git hub file movement ...Dhruba,<br />Can you modify your git hub file movement code to move ,<br />a < block >from <192.168.1.1> to <192.168.1.2><br /><br />Thanks and Regards<br />Ram<br />ReplyAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-38995741562349518452013-01-07T22:24:39.886-08:002013-01-07T22:24:39.886-08:00Ram: I am not sure I understand your request.Ram: I am not sure I understand your request.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-29320404886903791222013-01-07T22:22:50.225-08:002013-01-07T22:22:50.225-08:00Dhruba,
Can you modify your git hub file movement ...Dhruba,<br />Can you modify your git hub file movement code to move ,<br /> a from <192.168.1.1> to <192.168.1.2><br /><br />Thanks and Regards<br />RamAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-1454168490258550222012-09-23T00:28:14.253-07:002012-09-23T00:28:14.253-07:00The distance is calculated via the network topolog...The distance is calculated via the network topology script, more details here: http://developer.yahoo.com/hadoop/tutorial/module2.html#rackDhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-85416552642647053292012-09-22T07:04:51.529-07:002012-09-22T07:04:51.529-07:00on read the replicas are selected according to the...on read the replicas are selected according to the distance with the client ,<br />how hdfs calculate the distanceVijuPoonthottamhttps://www.blogger.com/profile/12460338614332426045noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-56180598827708189702011-11-09T18:02:03.335-08:002011-11-09T18:02:03.335-08:00Thank you, Dhruba.Thank you, Dhruba.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-14221156105810997932011-11-09T16:27:08.554-08:002011-11-09T16:27:08.554-08:00https://github.com/facebook/hadoop-20/blob/master/...https://github.com/facebook/hadoop-20/blob/master/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyConfigurable.javaDhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-63332660454351155502011-11-09T02:28:59.859-08:002011-11-09T02:28:59.859-08:00Hey Dhruba,
It seems the link to Configurable Bloc...Hey Dhruba,<br />It seems the link to Configurable Block Placement Policy example is broken.<br />Could you please re-post it?<br /><br />Thank you.Hemanthnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-33991685866370676822011-09-25T22:16:55.442-07:002011-09-25T22:16:55.442-07:00Thanks a lot for your kind help...But I have still...Thanks a lot for your kind help...But I have still a difficulty on working with hadoop under eclipse..Please suggest me simple ways if you have..I think your help is very useful for me...Thanks you in advance..Ju Junoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-8155241055647317422011-09-23T05:38:13.108-07:002011-09-23T05:38:13.108-07:00here is an example of ho we wrote our own block pl...here is an example of ho we wrote our own block placement policy:<br />https://github.com/facebook/hadoop-20-append/blob/master/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlockPlacementPolicyConfigurable.javaDhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-28776074243245317062010-10-08T05:21:07.565-07:002010-10-08T05:21:07.565-07:00fitness reviews weight loss reviews<a href="http://www.fitnessreviewsguru.com/" rel="nofollow">fitness reviews</a> <a href="http://www.fatlossprophet.com/" rel="nofollow">weight loss reviews</a>Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-84152332221190107392010-03-23T06:55:21.489-07:002010-03-23T06:55:21.489-07:00Nobody is working on HDFS snapshots (as far as I k...Nobody is working on HDFS snapshots (as far as I know). Here is a v v preliminary design document that I had written <br /><br />http://issues.apache.org/jira/browse/HDFS-233Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-83552395013433936952010-03-22T16:14:19.828-07:002010-03-22T16:14:19.828-07:00Dhruba,thanks for the answer. May be it is not the...Dhruba,thanks for the answer. May be it is not the best place to ask but what happened to HDFS snapshot capability feature? Is anybody working on it right now? If not then why? Incremental snapshot based backup is the only one from many other use cases. Actually, we have at least two of them at our company.<br /><br />-VladimirVladimir Rodionovhttps://www.blogger.com/profile/18159080975174540442noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-83007654411327341882010-03-22T06:39:45.791-07:002010-03-22T06:39:45.791-07:00Hi Jenium, your reasoning is correct. You can now ...Hi Jenium, your reasoning is correct. You can now supply your ownblock placement policies to HDFS.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-72043116959037012212010-03-21T17:07:52.734-07:002010-03-21T17:07:52.734-07:00Correct me if I am wrong. Using this HDFS-385 patc...Correct me if I am wrong. Using this HDFS-385 patch we can make HDFS to store all blocks of a file on a same node (physically)?Vladimir Rodionovhttps://www.blogger.com/profile/18159080975174540442noreply@blogger.com