資源描述：
《基于網(wǎng)絡爬蟲的有效URL緩存 畢業(yè)論文外文翻譯》由會員上傳分享，免費在線閱讀，更多相關內(nèi)容在學術論文-天天文庫。
1、外文資料原文外文資料原文EfficientURLCachingforWorldWideWebCrawlingAndreiZ.BroderIBMTJWatsonResearchCenter19SkylineDrHawthorne,NY10532abroder@us.ibm.comMarcNajorkMicrosoftResearch1065LaAvenidaMountainView,CA94043najork@microsoft.comJanetL.WienerHewlettPackardLabs1501PageMillRoadPaloAlto,CA94304jan
2、et.wiener@hp.comABSTRACTCrawlingthewebisdeceptivelysimple:thebasicalgorithmis(a)Fetchapage(b)ParseittoextractalllinkedURLs(c)ForalltheURLsnotseenbefore,repeat(a)–(c).However,thesizeoftheweb(estimatedatover4billionpages)anditsrateofchange(estimatedat7%perweek)movethisplanfromatrivialpr
3、ogrammingexercisetoaseriousalgorithmicandsystemdesignchallenge.Indeed,thesetwofactorsaloneimplythatforareasonablyfreshandcompletecrawloftheweb,step(a)mustbeexecutedaboutathousandtimespersecond,andthusthemembershiptest(c)mustbedonewellovertenthousandtimespersecondagainstasettoolargetos
4、toreinmainmemory.Thisrequiresadistributedarchitecture,whichfurthercomplicatesthe外文資料原文membershiptest.Acrucialwaytospeedupthetestistocache,thatis,tostoreinmainmemorya(dynamic)subsetofthe“seen”URLs.ThemaingoalofthispaperistocarefullyinvestigateseveralURLcachingtechniquesforwebcrawling.W
5、econsiderbothpracticalalgorithms:randomreplacement,staticcache,LRU,andCLOCK,andtheoreticallimits:clairvoyantcachingandinfinitecache.Weperformedabout1,800simulationsusingthesealgorithmswithvariouscachesizes,usingactuallogdataextractedfromamassive33daywebcrawlthatissuedoveronebillionHTT
6、Prequests.Ourmainconclusionisthatcachingisveryeffective–inoursetup,acacheofroughly50,000entriescanachieveahitrateofalmost80%.Interestingly,thiscachesizefallsatacriticalpoint:asubstantiallysmallercacheismuchlesseffectivewhileasubstantiallylargercachebringslittleadditionalbenefit.Weconj
7、ecturethatsuchcriticalpointsareinherenttoourproblemandventureanexplanationforthisphenomenon.1.INTRODUCTIONArecentPewFoundationstudy[31]statesthat“SearchengineshavebecomeanindispensableutilityforInternetusers”andestimatesthatasofmid-2002,slightlyover50%ofallAmericanshaveusedwebsearchto
8、findi
