tag:blogger.com,1999:blog-8556003786324804973.post743442804924971131..comments2024-01-27T03:43:02.667-08:00Comments on HDFS: Hadoop AvatarNode High AvailabilityDhruba Borthakurhttp://www.blogger.com/profile/10832366855372649190noreply@blogger.comBlogger61125tag:blogger.com,1999:blog-8556003786324804973.post-67941939492921675412012-02-23T05:18:52.231-08:002012-02-23T05:18:52.231-08:00Hi,
I need to do the fail over testing on a clust...Hi,<br /><br />I need to do the fail over testing on a cluster having 5-10 nodes.<br />Can you let me know what are all the cases do i need to take care of?Anonymoushttps://www.blogger.com/profile/10439085223402950400noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-11440200573531699922012-01-22T20:01:01.733-08:002012-01-22T20:01:01.733-08:00Thanks for sharing it with us. Keep it up!
walmart...Thanks for sharing it with us. Keep it up!<br /><a href="http://iamdavie.com/walmart-credit-card/" rel="nofollow">walmart credit card</a>airahttps://www.blogger.com/profile/17913256714950356365noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-88576539204349677672011-09-08T12:57:40.725-07:002011-09-08T12:57:40.725-07:00The datanodes send blockReceived and block reports...The datanodes send blockReceived and block reports to both AvatarNodes. One of the Avatarnode is running as the primary and the other one as the standby. From the perspective of sending heartbeats, the datanode does not distinguish between primary AvatarNode and standby AvatarNode.<br /><br />If the datanode cannot send a report to one of the namenodes, it retries for a certain period (with backoff), but if the buffer becomes too big, it drops all the blockReceived and remembers to send a full blockreport at the next successful call to that AvatarNode.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-14504237811971097122011-09-08T09:58:07.701-07:002011-09-08T09:58:07.701-07:00Would you mind providing a brief description of th...Would you mind providing a brief description of the NN/DN heartbeat process from time T1 when NN is alive, through time T2 when NN is dead but AN (Avatar Node) switchover has not yet occurred, through T3 where AN is now what DN heartbeat is talking to. Specifically, I am trying to understand how and when DN knows where to send its heartbeat report. Thanks.Paulhttps://www.blogger.com/profile/18168120866664624709noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-29117007934810511432011-09-08T03:52:25.443-07:002011-09-08T03:52:25.443-07:00Hi Sid...
I have done that and i am facing this pr...Hi Sid...<br />I have done that and i am facing this problem after that <br />ivy-retrieve-common:<br />[ivy:retrieve] :: retrieving :: org.apache.hadoop#hmon [sync]<br />[ivy:retrieve] confs: [common]<br />[ivy:retrieve] 0 artifacts copied, 8 already retrieved (0kB/0ms)<br />No ivy:settings found for the default reference 'ivy.instance'. A default instance will be used<br />DEPRECATED: 'ivy.conf.file' is deprecated, use 'ivy.settings.file' instead<br />:: loading settings :: file = H:\Webproject\hadoop\Hadoop\facebook-hadoop-20-warehouse-bbfed86\ivy\ivysettings.xml<br />compile:<br /> [echo] contrib: hmon<br />compile:<br /><br />BUILD FAILED<br />H:\Webproject\hadoop\Hadoop\facebook-hadoop-20-warehouse-bbfed86\build.xml:522: The following error occurred while executing this line:<br />H:\Webproject\hadoop\Hadoop\facebook-hadoop-20-warehouse-bbfed86\src\contrib\build.xml:30: The following error occurred while executing this line:<br />H:\Webproject\hadoop\Hadoop\facebook-hadoop-20-warehouse-bbfed86\src\contrib\hod\build.xml:29: Execute failed: java.io.IOException: Cannot run program "echo" (in directory "H:\Webproject\hadoop\Hadoop\facebook-hadoop-20-warehouse-bbfed86\src\contrib\hod"): CreateProcess error=2, The system cannot find the file specifiedtiruhttps://www.blogger.com/profile/05148747628433396198noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-26996032821990947562011-09-05T23:11:59.265-07:002011-09-05T23:11:59.265-07:00hi tiru, please do a clean build by first running ...hi tiru, please do a clean build by first running "ant clean", that should solve your problem.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-31492764687671754332011-09-05T04:25:48.369-07:002011-09-05T04:25:48.369-07:00hi I am trying to build avatar node build has fail...hi I am trying to build avatar node build has faild here with the jsp<br />Compiling 5 source files to C:\Chaitanya\Projects\Hadoop\Sample\facebook-hadoop-20-warehouse-bbfed86\build\classes<br /> [javac] C:\Chaitanya\Projects\Hadoop\Sample\facebook-hadoop-20-warehouse-bbfed86\build\src\org\apache\hadoop\hdfs\server\namenode\corrupt_005ffiles_jsp.java:77: cannot find symbol<br /> [javac] symbol : method getNamesystem()<br /> [javac] location: class org.apache.hadoop.hdfs.server.namenode.NameNode<br /> [javac] FSNamesystem fsn = nn.getNamesystem();<br /> [javac] ^<br /> [javac] C:\Chaitanya\Projects\Hadoop\Sample\facebook-hadoop-20-warehouse-bbfed86\build\src\org\apache\hadoop\hdfs\server\namenode\corrupt_005ffiles_jsp.java:80: cannot find symbol<br /> [javac] symbol : class CorruptFileBlockInfo<br /> [javac] location: class org.apache.hadoop.hdfs.server.namenode.FSNamesystem<br /> [javac] Collection corruptFileBlocks = <br /> [javac] ^<br /> [javac] C:\Chaitanya\Projects\Hadoop\Sample\facebook-hadoop-20-warehouse-bbfed86\build\src\org\apache\hadoop\hdfs\server\namenode\corrupt_005ffiles_jsp.java:81: cannot find symbol<br /> [javac] symbol : method listCorruptFileBlocks(java.lang.String,)<br /> [javac] location: class org.apache.hadoop.hdfs.server.namenode.FSNamesystem<br /> [javac] fsn.listCorruptFileBlocks("/", null);<br /><br />please help on thistiruhttps://www.blogger.com/profile/05148747628433396198noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-69349823970376837192011-07-10T13:03:47.405-07:002011-07-10T13:03:47.405-07:00If you make the BackupNameNode process block repor...If you make the BackupNameNode process block reports/blockReceiveds from datanodes, then it will be hot in nature. This is one step closer to the design of the AvatarNode. The difference being that the AvatarNode still uses the shared NFS mount point to store transaction logs.<br /><br />If you decide to make the primary namenode synchronously replicate transactions to the backupnode, you would have to measure the performance of transactions. will the response from the backupnode be as fast/reliable as an NFS appliance? If the backup node falls out of sync, how quickly/reliably does it get the full transaction log from the primary? These are some issues that are worth measuring on a real cluster.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-71447499732157885982011-07-09T16:22:22.034-07:002011-07-09T16:22:22.034-07:00The datanodes send block reports to both namenode ...The datanodes send block reports to both namenode and backupnode, so the last can act as a hot-standby.Anonymoushttps://www.blogger.com/profile/08214765587656957936noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-68061101587636591832011-07-06T18:25:32.669-07:002011-07-06T18:25:32.669-07:00I am not sure what you mean by "version of Av...I am not sure what you mean by "version of Avatar Node that uses the BackupNode", can you pl explain?<br /><br />My point is that if u use the BackupNode from 0.20 (or 0.22), the system will be as reliable as the combined availability of both nodes (which is typically lower than the availability of just the NN). The NN synchronously sends a transaction to the BN.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-54474365032673100372011-07-03T20:11:03.029-07:002011-07-03T20:11:03.029-07:00Hi Dhruba,
I was working in a version of Avatar...Hi Dhruba, <br /><br />I was working in a version of Avatar Node that uses the BackupNode. But on the paper for SIGMOD '11 is said that this does not bring any advantage over current approach. I though it was because the backup node was not available on 0.20 . I also tought that failures on BackNode would not affect Namenode, but it does not look so. Could you comment on that?<br /><br />Could also give some overview about the implementation of DAFS? I understood that you replaced the ClientProtocol proxy ( dfs,namenode) by FailoverClientProtocol, which redirect requests to the current active avatar. But I did not understood why in the event of failover, failoverFS, a new DistributedFileSystem, is created and FailoverClientProtocol is set to point to its dfs.namenode. Why couldn't we just create a new ClientProtocol proxy ? <br /><br />Thanks,<br />AndréAnonymoushttps://www.blogger.com/profile/08214765587656957936noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-54099159896272784642011-07-03T12:05:31.703-07:002011-07-03T12:05:31.703-07:00A block report processing takes about 100 ms. This...A block report processing takes about 100 ms. This means we can process about 100 nodes per second. On namenode startup, our 3000 node cluster takes about 5 minutes to process all block reports. <br />Then it takes another 10-15 minutes to exit safemode, the reason being that when the namenode decides to exit safemode it loops though all existing blocks to determine underreplicated/excess-replicated blocks. Since we have close to 300 million blocks it takes a long long time to loop through all those blocks. Another reason is that since we use HDFS RAIFD, there could be lots of blocks with excess replicas, and excess replica deletion code in the NN is very heavy-weight.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-18511714758146641472011-07-03T08:59:44.497-07:002011-07-03T08:59:44.497-07:00Konstantine once mentioned in his paper (http://de...Konstantine once mentioned in his paper (http://developer.yahoo.com/blogs/hadoop/posts/2010/05/scalability_of_the_hadoop_dist/) that NameNode is able to process about 10 block report per second. So if we have a cluster of around 2000 nodes, it will take around 200 seconds to process all block report. Can you comment on this?Thanh D. Dohttps://www.blogger.com/profile/08210169726398878947noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-10444423203709503182011-07-03T08:54:53.925-07:002011-07-03T08:54:53.925-07:00Hi Dhruba. Can you shred some light on why process...Hi Dhruba. Can you shred some light on why processing block reports take so much time?Thanh D. Dohttps://www.blogger.com/profile/08210169726398878947noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-56384161186786689002011-06-27T09:03:14.554-07:002011-06-27T09:03:14.554-07:00absolutely right, most of the restart time is from...absolutely right, most of the restart time is from processing block reports.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-25623063355370734622011-06-27T07:40:52.712-07:002011-06-27T07:40:52.712-07:00Dhruba, if I remember correctly, the checkpoint wi...Dhruba, if I remember correctly, the checkpoint will be done once the edits file exceeds 64MB right? So restarting a non-avatar NameNode shouldn't spend much time on processing the transaction log. And the major time spent on restarting comes from processing block reports, right? Please correct me if I am wrong.Thanh D. Dohttps://www.blogger.com/profile/08210169726398878947noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-12880630635287046532011-06-27T04:34:58.171-07:002011-06-27T04:34:58.171-07:00Hi Thanh, our average transaction log is about 2GB...Hi Thanh, our average transaction log is about 2GB every half hour, but peaks are much higher than this.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-33920725007468308332011-06-26T09:55:17.497-07:002011-06-26T09:55:17.497-07:00Hi Dhruba!
I am curious about the restart time o...Hi Dhruba! <br /><br />I am curious about the restart time of non-avatar NameNode. You mentioned that it could take 1 hour to restart, including:<br />- 6 minutes to load the 12GB fsimage<br />- 35 minutes to process block reports<br />So, did it take about 20 minutes to process the transaction log? What is the typical size of a edits file in your cluster?Thanh D. Dohttps://www.blogger.com/profile/08210169726398878947noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-11519622186152988542011-06-12T20:01:58.579-07:002011-06-12T20:01:58.579-07:00Hi, would you please tell me how to configure and ...Hi, would you please tell me how to configure and startup?<br />and you metioned that have to 'runs a manual command to trigger the failover' , and what is it?<br />Thank you very muchAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-24194486444120742282011-05-27T04:04:04.333-07:002011-05-27T04:04:04.333-07:00It belongs.
They don't need to, but they need...It belongs. <br />They don't need to, but they need to have one interface in it, cause it is only one unique VIP that belongs to only one subnet if I understand you right? It should be made available somehow on the machines.<br />Everything else can be subnet-indepedendent. Pacemaker uses Multicast often to have a cluster-wide communication which is independent to the underlying topology.Lars Froniushttp://twitter.com/erstlingswerknoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-21541296597531744272011-05-25T02:13:30.525-07:002011-05-25T02:13:30.525-07:00Hi Lars, when we talk about VIP, isn't it true...Hi Lars, when we talk about VIP, isn't it true that the pair of machines have to be in the same subnet?Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-21862929866141696572011-05-25T01:43:11.281-07:002011-05-25T01:43:11.281-07:00Is it possible do have both Avatar/NameNodes in a ...Is it possible do have both Avatar/NameNodes in a state where they can write? If so you could put an LVS-Loadbalancer in the front that can hold the VIP and can route to NameNodes through direct-routing which would also solve the problem of the NameNode as a bottleneck in HDFS.<br /><br />If this is not possible I do want to make a remark on the first comment. I do like pacemaker very much for it's flexibility, and I think it could fit here. DRBD might be the false approach, but you can have a shared-storage like NFS as you did and put your NameNodes as Cloned-Ressources (or Master-Slave) through pacemaker on the different Hosts. On of them gets the VIP. You do have a hot-standby through the cloned/master-slave-ressource-concept and you are very flexible in scripting for it through the OCF-Framework. Pacemaker does the monitoring and failover for you.<br /><br />I think it would be nice to have this integration into the "standard" linux-ha suite.Lars Froniushttp://twitter.com/erstlingswerknoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-28411210841743889422011-05-22T17:54:53.055-07:002011-05-22T17:54:53.055-07:00The workload on our largest 30 PB HDFS cluster wri...The workload on our largest 30 PB HDFS cluster writes t transaction log at around 6 MByte/second. We do not use a fast -interconnect.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-26051369818750885762011-05-20T12:01:35.462-07:002011-05-20T12:01:35.462-07:00Elegant solution.
Was curious to know what the I/...Elegant solution.<br /><br />Was curious to know what the I/O rates are for the secondary trying to keep up with the primary. <br /><br />Did you consider a fast interconnect between primary and secondary as a means for the secondary to read transactions and go to the filer only as a fallback?<br /><br />Regards<br />TW @shrusamiraAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-8556003786324804973.post-10589645075352912612011-03-12T22:55:54.054-08:002011-03-12T22:55:54.054-08:00when the avaratnode-standby starts up, it reads th...when the avaratnode-standby starts up, it reads the fstime of the remote primary namenode to remember the time wen the primary ast checkpointed. if, somehow, the primary checkpoints again by itself (without consulting the standby), the standby can still detect this fact by looking at the fstime file.Dhruba Borthakurhttps://www.blogger.com/profile/10832366855372649190noreply@blogger.com