資源描述：
《Crowdsourcing for Relevance Evaluation 關聯(lián)評價的眾包》由會員上傳分享，免費在線閱讀，更多相關內(nèi)容在學術論文-天天文庫。
1、PAPERCrowdsourcingforRelevanceEvaluationOmarAlonsoDanielE.RoseBenjaminStewartA9.comPaloAlto,CA{oalonso,danrose,bstewart}@a9.comAbstractRelevanceevaluationisanessentialpartofthedevelopmentandmaintenanceofinformationretrievalsystems.Yettraditionalevaluationapproacheshaveseverallimitations;inparticul
2、ar,conductingneweditorialevaluationsofasearchsystemcanbeveryexpensive.WedescribeanewapproachtoevaluationcalledTERC,basedonthecrowdsourcingparadigm,inwhichmanyonlineusers,drawnfromalargecommunity,eachperformsasmallevaluationtask.1IntroductionRelevanceevaluationforinformationretrievalisanotoriouslyd
3、ifficultandexpensivetask.Intheearlyyearsofthefield,asetofvolunteereditors–oftengraduatestudents–wouldpainstakinglyreadthrougheverydocumentinacorpustodetermineitsrelevancetoasetoftestqueries.Thisprocesswassufficientlydifficultthatonlyafewsmalltestcollections(Cranfield,CACM,etc.)werecreated.Withthea
4、dventofTRECin1992[9],researchershadaccesstotestcollectionswithmillionsoffull-textdocuments.However,thescaleofTRECwasonlypossiblebyeliminatingthenotionthateverydocumentwouldbereadandevaluated.Instead,thepoolingapproachwasdeveloped,inwhichonlythetopNdocumentsretrievedbyatleastoneoftheparticipatingsy
5、stemswereexamined.TheotherfactorthatmadeTRECpossiblewastheavailabilityofalargenumberofprofessionalassessors–retiredintelligenceanalysts–whowerepaidfortheirworkwithfundsfromthesponsoringagencies.WhiletheTRECcollections–andasimportantly,thequerysetsandevaluations–havebeeninvaluableinfurtheringIRrese
6、archoverthepast15years,theystillhavesomelimitations.ThemostobviousoftheseisthatresearchersarelimitedtothetypesofIRtasksthatTREChasstudied.Forexample,ifaresearcherwishestostudysearchinaparticularverticaldomainarea(forexample,ayellow-pages-stylesearchforlocalbusinesses)orexperimentwithanewsearchinte
7、ractionparadigm(forexample,collaborativesearch),thenexistingTRECcollectionsmaynothelp.Furthermore,despitethepresenceofaWebtrack,evaluatinggeneralWebsearchhasuniquechallenges[7],whichoftenrequireanotherapproach.Fo
