資源描述：
《基于網(wǎng)絡(luò)爬蟲(chóng)的有效url緩存畢業(yè)論文外文文獻(xiàn)翻譯》由會(huì)員上傳分享，免費(fèi)在線(xiàn)閱讀，更多相關(guān)內(nèi)容在工程資料-天天文庫(kù)。
1、外文資料原文EfficientURLCachingforWorldWideWebCrawlingAndreiZ.BroderIBMTJWatsonResearchCenter19SkylineDrHawthorne,NY10532abroder@us.ibm.comMarcNajorkMicrosoftResearch1065LaAvenidaMountainView,CA94043najork@microsoft.comJanetL.WienerHewlettPackardLabs1501PageMillRoadPaloAlto,CA
2、94304janet.wiener@hp.comABSTRACTCrawlingthewebisdeceptivelysimple:thebasicalgorithmis(a)Fetchapage(b)ParseittoextractalllinkedURLs(c)ForalltheURLsnotseenbefore,repeat(a)—(c).However,thesizeoftheweb(estimatedatover4billionpages)anditsrateofchange(estimatedat7%perweek)move
3、thisplanfromatrivialprogrammingexercisetoaseriousalgorithmicandsystemdesignchallenge.Indeed,thesetwofactorsaloneimplythatforareasonablyfreshandcompletecrawloftheweb,stepmustbeexecutedaboutathousandtimespersecond,andthusthemembershiptest(c)mustbedonewellovertenthousandtim
4、espersecondagainstasettoolargetostoreinmainmemory.Thisrequiresadistributedarchitecture,whichfurthercomplicatesthemembershiptest.Acrucialwaytospeedupthetestistocache,thatis,tostoreinmainmemorya(dynamic)subsetofthe"seen"URLs.Themaingoalofthispaperistocarefullyinvestigatese
5、veralURLcachingtechniquesforwebcrawling.Weconsiderbothpracticalalgorithms:randomreplacement,staticcache,LRU,andCLOCK,andtheoreticallimits:clairvoyantcachingandinfinitecache.Weperformedabout1,800simulationsusingthesealgorithmswithvariouscachesizes,usingactuallogdataextrac
6、tedfromamassive33daywebcrawlthatissuedoveronebillionHTTPrequests.Ourmainconclusionisthatcachingisveryeffective-inoursetup,acacheofroughly50,000entriescanachieveahitrateofalmost80%.Interestingly,thiscachesizefallsatacriticalpoint:asubstantiallysmallercacheismuchlesseffect
7、ivewhileasubstantiallylargercachebringslittleadditionalbenefit.Weconjecturethatsuchcriticalpointsareinherenttoourproblemandventureanexplanationforthisphenomenon.INTRODUCTIONArecentPewFoundationstudy[31]statesthat"SearchengineshavebecomeanindispensableutilityforInternetus
8、ers"andestimatesthatasofmid-2002,slightlyover50%ofallAmericanshaveusedwebsearchtofindinformation.Hence,
