cross validation - Tune parameters with nested SVM in MATLAB -

i have dataset of 20 test subjects 50 variables , result vector of 1 , 0 determines state. set nested cross validation such in inner folds perform feature selection tuning hyperparameters of svm. these parameters should tested on outer fold.

i have done in regards feature selection logistic regression (using sequentialfs), @ loss how both feature selection , tuning of hyperparameters @ same time.

some sample code ideal, general explanation of setup , interpretation of given output helpful, new svm.

the feature selection must forwards. if possible, output best overall svm parameters , chosen features. naturally know errorrate on test sets. kernel used polynomial of second order, suppose there 1 parameter tune?

furthermore 5-fold on both inner , outer cross validation.

edit: have found sample code, should want, can't seem make work. can explain how set in matlab?

https://stats.stackexchange.com/questions/40906/nested-cross-validation-for-classification-in-matlab

i propose following answer, alldata contains data. every row entry , every column feature. variable called targets contain objective classes. assumptions going explain code attached below, part part.

a variable featsize have number of features.

featsize = size(alldata, 2); 

a variable kfolds contain number of folds want. said 5 suggest @ least 10, if have sufficiency of data.

kfolds = 5; 

a structure bestsvm "output" need. can clear variables @ end, except one. structure contain best svm , parameter , index of features yield best performance.

bestsvm = struct('svmmodel', nan, 'c', nan, 'featuresidx', nan, 'score', inf);      

a variable kidx contain cross validation indices based on data , number of folds. uses matlab's function crossvalind().

kidx = crossvalind('kfold', length(targets), kfolds); 

now, main outer loop run many folds specified in kfolds , prepare training set , testing set (traindata, testdata) corresponding targets (traintarg, testtarg) follows:

for k = 1:kfolds     traindata = alldata(kidx~=k, :);     traintarg = targets(kidx~=k);     testdata = alldata(kidx==k, :);     testtarg = targets(kidx==k); 

now, prepare feature selection. initialize variable bestfeatscore inf later can compare performance of svm (score) value (it may make more sense further down). initialize structure bestfeatcombo contain best svm among possible feature combinations corresponding feature indices feat, , parameter c.

    bestfeatscore = inf;     bestfeatcombo = struct('svm', nan, 'feat', nan, 'c', nan); 

the possible number of feature combinations 2^featsize - 1. e.g., if have 2 features in total, have 2^2 - 1 = 3 choices: 1) choose feature 1 only, 2) choose feature 2 only, 3) choose feature 1 , 2 only. thus, need for loop go on possible combinations of features.

    b = 1:(2^featsize) - 1 

however, there tricky part (for sure there better solution out-there) want start choosing sets of features first. thought of binary representation problem. have 3 features in total [f1, f2, f3], then, binary vector [1, 0, 0] represents selection of feature f1 , neglects rest. passing binary vector matlab give error of indexing. so, best way found solve use matlab's find() function find "indices , values of nonzero elements". thus, if featcombo = find([1, 0, 0]), variable featcombo equal 1. so, did use variable b (from loop above) contain number indicating current possible combination of features , convert binary vector of size featsize using matlab's function de2bi(). instance, de2bi(1, 3) gives [1, 0, 0] which, noticed uses left digit least significant. de2bi(3, 3) gives [1, 1, 0], de2bi(5, 3) gives [1, 0, 1], , on. then, if use find() function on de2bi() result on indexes of features want choose from.

        featcombo = find(de2bi(b, featsize)); 

for example, if featsize = 3, this:

for b = 1:(2^featsize) - 1; display(de2bi(b, featsize)); end; 

would give something like this:

 1     0     0  0     1     0  1     1     0  0     0     1  1     0     1  0     1     1  1     1     1 

which combined find follows:

for b = 1:(2^featsize) - 1; display(find(de2bi(b, featsize))); end; 

would give something like this:

 1  2  1     2  3  1     3  2     3  1     2     3 

which suitable use logical indexing. so, featcombo contain set (vector indices) of features chosen.

the next part initialize variables grid search boxconstraint c known hyper-parameter (depending on svm type may have other parameters). bestcscore contain best performance of svm during grid search, bestc contain best c parameter, , bestcsvm have best trained svm during search. variable gridc contain search space, in example goes 2^-5, 2^-3, 2^15. if have plenty of computational resources, suggest change 2.^(-5:2:15) smaller increment, 2.^(-5:1:15), or smaller, 2.^(-5:0.1:15) careful take while finish. similarly, if have low computational power (or limited time) increase interval 2.^(-5:3:15) or 2.^(-5:4:15), knowing poorly chose hyper-parameter.

        bestcscore = inf;         bestc = nan;         bestcsvm = nan;         gridc = 2.^(-5:2:15); 

next, start grid search based on array gridc , train svm using matlab's function fitcsvm(). svm trained current traindata determined current fold, , specific chosen features determined featcombo. training targets traintarg determined current fold well. note a) using rbf kernel type (since did not specify it), , b) letting matlab auto determine kernel scale. if want use kernel, code need modification.

        c = gridc             ansvmmodel = fitcsvm(traindata(:, featcombo), traintarg, ...                 'kernelfunction', 'rbf', 'kernelscale', 'auto', ...                 'boxconstraint', c); 

the next step determine how svm performed parameter, set of features, fold. function loss() svms.

            l = loss(ansvmmodel,testdata(:, featcombo), testtarg); 

if current svm (ansvmmodel) performed better previous best performance save score bestcscore, best parameter bestc, , best svm bestcsvm.

            if l < bestcscore                 bestcscore = l;                 bestc = c;                 bestcsvm = ansvmmodel;             end         end 

at end of inner-most loop, should have best hyper-parameter , svm current feature set. so, if svm has better score other trained svm other feature set, save svm, set of features, , hyper-parameter structure given bestfeatcombo.svm, bestfeatcombo.feat, bestfeatcombo.c` respectively.

        if (bestcscore < bestfeatscore) || ...                 ((bestcscore == bestfeatscore) && ...                 (length(featcombo) < length(bestfeatcombo.feat)))             bestfeatscore = bestcscore;             bestfeatcombo.svm = bestcsvm;             bestfeatcombo.feat = featcombo;             bestfeatcombo.c = bestc;         end     end 

however, note in if statement above, making special case in or clause. saying if current svm performs (scores) same best far, current svm has smaller set of features give same performance, choose replace best far svm uses less features.

now, @ end of intermediate loop, should have best svm set of features, best hyper-parameter c (to extent grid search allows). thus, can compare current best (stored in structure bestfeatcombo) against overall best far (stored in structure bestsvm).

    if bestfeatscore < bestsvm.score         bestsvm.svmmodel = bestfeatcombo.svm;         bestsvm.c = bestfeatcombo.c;         bestsvm.featuresidx = bestfeatcombo.feat;         bestsvm.score = bestfeatscore     end     end 

that end of it. said, output want in bestsvm, contains best svm out of kfolds, best combination of features, , best hyper-parameter c grid search allows.

i hope makes sense. below working example using matlab's fisheriris dataset has 100 samples , 4 features.

working code:

load fisheriris inds = ~strcmp(species,'setosa'); alldata = meas(inds,:); targets = species(inds); featsize = size(alldata, 2); kfolds = 5;     % specify number of folds bestsvm = struct('svmmodel', nan, ...     % store best svm     'c', nan, 'featuresidx', nan, 'score', inf);       kidx = crossvalind('kfold', length(targets), kfolds); k = 1:kfolds     traindata = alldata(kidx~=k, :);     traintarg = targets(kidx~=k);     testdata = alldata(kidx==k, :);     testtarg = targets(kidx==k);      % forward feature selection starts     bestfeatscore = inf;     bestfeatcombo = struct('svm', nan, 'feat', nan, 'c', nan);     b = 1:(2^featsize) - 1         % choose features. e.g. [1 0 0] selects first         % feature out of 3 features.         featcombo = find(de2bi(b, featsize));          % grid search boxconstraint         bestcscore = inf;         bestc = nan;         bestcsvm = nan;         gridc = 2.^(-5:2:15);         c = gridc             ansvmmodel = fitcsvm(traindata(:, featcombo), traintarg, ...                 'kernelfunction', 'rbf', 'kernelscale', 'auto', ...                 'boxconstraint', c);             l = loss(ansvmmodel,testdata(:, featcombo), testtarg);             if l < bestcscore        % saving best svm on parameter                 bestcscore = l;      % selection                 bestc = c;                 bestcsvm = ansvmmodel;             end         end          % saving best svm on feature selection         if (bestcscore < bestfeatscore) || ...                 ((bestcscore == bestfeatscore) && ...                 (length(featcombo) < length(bestfeatcombo.feat)))             bestfeatscore = bestcscore;             bestfeatcombo.svm = bestcsvm;             bestfeatcombo.feat = featcombo;             bestfeatcombo.c = bestc;         end     end      % saving best svm on folds     if bestfeatscore < bestsvm.score         bestsvm.svmmodel = bestfeatcombo.svm;         bestsvm.c = bestfeatcombo.c;         bestsvm.featuresidx = bestfeatcombo.feat;         bestsvm.score = bestfeatscore     end end 

edit

to answer edited question, in want have 5-fold cross-validation parameter selection, here need do.

please change inner most loop following:

        % grid search boxconstraint         bestcscore = inf;         bestc = nan;         gridc = 2.^(-5:2:15);         c = gridc             % cross validation parameter c             kidxc = crossvalind('kfold', length(traintarg), kfolds);             l = zeros(1, kfolds);             kc = 1:kfolds                 traindatac = traindata(kidxc~=kc, :);                 traintargc = traintarg(kidxc~=kc);                 testdatac = traindata(kidxc==kc, :);                 testtargc = traintarg(kidxc==kc);                 ansvmmodel = fitcsvm(traindatac(:, featcombo), traintargc, ...                     'kernelfunction', 'rbf', 'kernelscale', 'auto', ...                     'boxconstraint', c);                 l(kc) = loss(ansvmmodel,testdatac(:, featcombo), testtargc);             end             l = mean(l);             if l < bestcscore                 bestcscore = l;                 bestc = c;             end         end         % need retrain here , save svm best c         bestcsvm = fitcsvm(traindata(:, featcombo), traintarg, ...             'kernelfunction', 'rbf', 'kernelscale', 'auto', ...             'boxconstraint', bestc);         bestcscore = loss(bestcsvm,testdata(:, featcombo), testtarg); 

the code uses standard cv. note following line important:

l(kc) = loss(ansvmmodel,testdatac(:, featcombo), testtargc); 

in line save performances across folds. however, note have average that, purpose of cv.

l = mean(l); 

you have retraining, once.