model selection for small sample regression

《model selection for small sample regression》由會員上傳分享，免費在線閱讀，更多相關(guān)內(nèi)容在工程資料-天天文庫。

1、MachineLearning,48,9–23,2002c2002KluwerAcademicPublishers.ManufacturedinTheNetherlands.ModelSelectionforSmallSampleRegressionOLIVIERCHAPELLEolivier.chapelle@liple.frLIP6,15rueduCapitaineScott,75015Paris,FranceVLADIMIRVAPNIKvlad@research.att.comAT&TResearchL

2、abs,200LaurelAvenue,Middletown,NJ07748,USAYOSHUABENGIObengioy@IRO.UMontreal.CADept.IRO,CP6128,UniversitedeMontr′eal,Succ.Centre-Ville,2920Chemindelatour,Montr′eal,Qu′ebec,′Canada,H3C3J7Editor:DaleSchuurmansAbstract.Modelselectionisanimportantingredientofmany

3、machinelearningalgorithms,inparticularwhenthesamplesizeinsmall,inordertostriketherighttrade-offbetweenover?ttingandunder?tting.Previousclassicalresultsforlinearregressionarebasedonanasymptoticanalysis.Wepresentanewpenalizationmethodforperformingmodelselectio

4、nforregressionthatisappropriateevenforsmallsamples.Ourpenalizationisbasedonanaccurateestimatoroftheratiooftheexpectedtrainingerrorandtheexpectedgeneralizationerror,intermsoftheexpectedeigenvaluesoftheinputcovariancematrix.Keywords:modelselection,parametricre

5、gression,uniformconvergencebounds1.IntroductionConsidertheproblemofestimatingaregressionfunctioninthesetoffunctions∞f(x,α)=αk?k(x)(1)k=1where{?}formabasisofL(Rp),e.g.aFourierorwaveletbasis.k2Givenacollectionofdata(x1,y1),...,(xn,yn),whereyi=f(xi,α0)+ξiandxi

6、,ξiareindependentlygeneratedbyunknowndistributionsP(x)andP(ξ),onewantsto?ndthefunctionf(x,α?)thatprovidesthesmallestvalueoftheexpectedlossR(α)=L(y,f(x,α))dP(x)dP(ξ)(2)whereL(y,f(x,α))isagivenlossfunction,usuallythequadraticlossL(y,f(x,α))=(y?f(x,α))2.Tomini

7、mizetheexpectedrisk(2),oneminimizestheempiricalrisk10O.CHAPELLE,V.VAPNIK,ANDY.BENGIOfunctional1nRemp(α)=L(yi,f(xi,α))ni=1Howeversincetheset(1)hasanin?niteexpansion,thisideadoesnotwork:forany?nitenumberof(different)examplestherearefunctionswhichhavezeroempir

8、icalriskandalargevalueoftheexpectedloss.Toguaranteeasmallexpectedrisk,onecanminimizetheempiricalfunctionaloveronlythe?rstd=d(n)functions?k(x).Thisisreasonableifthe?kareorderedinsuchwaythatputsth