MATLAB Implementation of SVM Classifier - EECS 152B Winter 2019

This assignment focuses on designing a linear Support Vector Machine (SVM) classifier using MATLAB's quadratic programming tool. The goal is to classify d-dimensional feature vectors into two classes (+1 or -1) based on supervised learning. The solution involves implementing a soft-margin SVM model to solve a constrained optimization problem with slack variables. The MATLAB function 'quadprogsoftsvm' is used to determine the optimal classifier by finding the weights (w), bias (b), and error (e) for different training datasets (x1, x2, x3, x4) provided in 'hw5data.mat'. The assignment also includes plotting the classifier boundary and analyzing the impact of different 'c' values on the classification results. The results and surface plots are generated to visualize the data separation achieved by the SVM classifier.

Running head: ASSIGNMENT 5
Assignment 5
Name of the Student
Name of the University
Author Note

1Assignment 5
Assignment Introduction:
In this particular assignment the supervised learning method is used for designing a linear
support vector machine which can classify the vectors the d-dimensional feature vectors in
any one of the two classes given by y = +1 or -1. The function of the optimal binary classifier
is given by,
f(x) = sign(wTx + b)
The 4 different training data sets are loaded in MATLAB from hw5data.mat. The labels of
the dataset are x1, x2, x3 and x4.
1.
The given constraint optimization problem is
Min ( 1
2 zT Qz + c∑
i=1
N
εi)
Constraints:
yi ( wT xi+b ) ≥1−εi for all i
ε i ≥ 0
Here,
∈= [ ∈1 ∈2 … ..∈ N ] T
z = [ w1 w2 … . wd b ]T
Now, quadprog in MATLAB is used to solve the problem. The quadprog
objective function in standard form is given by,

2Assignment 5
( 1
2 xT Hx+ f T x)
x
min
such that,
A*x <= b, Aeq∗x=beq, lb≤ x ≤ ub
This problem is a Soft-margin SVM model. The MATLAB function which
solves the problem is shown below.
MATLAB function:
function [w,b,e] = quadprogsoftsvm(data,yvals,c)
rows = size(data,1);
cols = size(data,2);
H = diag([ones(1, cols), zeros(1, rows + 1)]);
f = [zeros(1,cols+1) c*ones(1,rows)]';
p = diag(yvals) * data;
A = -[p yvals eye(rows)];
B = -ones(rows,1);
lb = [-inf * ones(cols+1,1) ;zeros(rows,1)];
z = quadprog(H,f,A,B,[],[],lb);
w = z(1:cols,:);
b = z(cols+1:cols+1,:);
e = z(cols+2:cols+rows+1,:);

3Assignment 5
end
2.
Now, the above MATLAB function is applied to primal form of classifier on the four training
data x1,x2, x3 and x4 and this outputs the w, b and e(error) from original y values. Here, the
value of constant c = 10 is considered.
Input and output for training data x1:
load('hw5data.mat')
>> c=10;
>> data = x1; yvals = x1(:,3);
>> [w,b,e] = quadprogsoftsvm(data,yvals,c)
Minimum found that satisfies the constraints.
Optimization completed because the objective function is non-decreasing in
feasible directions, to within the default value of the optimality tolerance,
and constraints are satisfied to within the default value of the constraint tolerance.
<stopping criteria details>
w =
-0.2062
0.0607
0.9515

4Assignment 5
b =
0.4720
e =
1.0e-11 *
0.4566
0.4609
0.4610
0.4576
0.4584
0.4561
0.4733
0.4714
0.4584

5Assignment 5
0.4889
0.4674
0.4713
0.4481
0.4591
0.4706
0.7002
0.4259
0.4578
0.3341
0.4572
Input and output for training data x2:
load('hw5data.mat')
>> c=10;
data = x2; yvals = x2(:,3);
[w,b,e] = quadprogsoftsvm(data,yvals,c)
Minimum found that satisfies the constraints.
Optimization completed because the objective function is non-decreasing in

6Assignment 5
feasible directions, to within the default value of the optimality tolerance,
and constraints are satisfied to within the default value of the constraint tolerance.
<stopping criteria details>
w =
-0.0000
-0.0000
1.0000
b =
5.1297e-08
e =
1.0e-11 *
0.6009
0.6374
0.6263
0.6433
0.6145
0.7037
0.5898
0.6131
0.5796

7Assignment 5
0.6402
0.7133
0.6957
0.5871
0.6591
0.6758
0.5988
0.5804
0.6122
0.5507
0.6205
Input and output for training data x3:
load('hw5data.mat')
>> data = x3; yvals = x3(:,4); c= 10;
>> [w,b,e] = quadprogsoftsvm(data,yvals,c)
Minimum found that satisfies the constraints.
Optimization completed because the objective function is non-decreasing in
feasible directions, to within the default value of the optimality tolerance,
and constraints are satisfied to within the default value of the constraint tolerance.

8Assignment 5
<stopping criteria details>
w =
-0.0720
0.0689
-0.0156
0.9897
b =
0.1261
e =
1.0e-12 *
0.3102
0.3100
0.3106
0.3102
0.3130
0.3109
0.3102
0.3105
0.3234
0.3103

9Assignment 5
0.3104
0.3088
0.3200
0.3105
0.3172
0.3105
0.3105
0.3094
0.3136
0.3103
Input and output for training data x4:
load('hw5data.mat')
data = x4; yvals = x4(:,6); c= 10;
[w,b,e] = quadprogsoftsvm(data,yvals,c)
Minimum found that satisfies the constraints.
Optimization completed because the objective function is non-decreasing in
feasible directions, to within the default value of the optimality tolerance,
and constraints are satisfied to within the default value of the constraint tolerance.
<stopping criteria details>

10Assignment 5
w =
0.0000
-0.0000
-0.0000
-0.0000
0.0000
1.0000
b =
1.0669e-07
e =
1.0e-12 *
0.7957
0.8173
0.7971
0.7979
0.7977
0.8226
0.7978
0.7961

11Assignment 5
0.7978
0.7980
0.7972
0.7960
0.8001
0.7974
0.7970
0.7927
0.8173
0.7976
0.7971
0.8194
0.7967
0.7971
0.8099
0.7960
0.7966
0.7975
0.7962
0.7981

12Assignment 5
0.7960
0.7978
3.
Now, it can be seen that for the output of training data x1, the column of
e vector is very close to zero. Hence, all the slack variables has
numerically gone close to zero.
Now, the classifier boundary where two data in x1 are separated is
plotted in a scatterplot.
MATLAB code:
f=zeros(4,1);
e1=10.*(-ones(35,1)); % c = 10
A1=[-a1' ones(20,1);a2' ones(15,1)];
z=quadprog(Q,f,A1,e1);
x1axis=[0:.1:4];
x2axis=[0:.1:4];
length(x1axis)
for k=1:41
for n=1:41
x3axis(k,n)=(-z(1)/z(3))*x1axis(k)+(-z(2)/z(3))*x2axis(n)+(-z(4)/z(3));
end

13Assignment 5
end
surf(x1axis,x2axis,x3axis')
hold
a2=a2+ones(3,1)*ones(1,15);
scatter3(a1(1,:),a1(2,:),a1(3,:),'b')
hold
scatter3(a2(1,:),a2(2,:),a2(3,:),'r')
for k=1:41
for n=1:41
x3axis(k,n)=(-z(1)/z(3))*x1axis(k)+(-z(2)/z(3))*x2axis(n) +(-z(4)/z(3));
end
end
surf(x1axis,x2axis,x3axis')
Plot:

14Assignment 5
-1
4
0
1
3 6
2
1015
3
2 4
4
5
1 2
0 0
4. Now, quadprog is applied with a different c value other than 10 on the training data x2.
Here, we have assumed c = 20.
MATLAB code:
load('hw5data.mat')
a1 = x2'; % training dataset 1
a2=[2;2;2]+randn(3,15)/2;
Q=[eye(3) zeros(3,1);zeros(1,3) 0];
f=zeros(4,1);
e1=20.*(-ones(35,1)); % c = 20 is assumed
A1=[-a1' ones(20,1);a2' ones(15,1)];
z=quadprog(Q,f,A1,e1);
x1axis=[0:.1:25];
x2axis=[0:.1:25];
for k=1:length(x1axis)
for n=1:length(x2axis)
x3axis(k,n)=(-z(1)/z(3))*x1axis(k)+(-z(2)/z(3))*x2axis(n)+(-z(4)/z(3));
end
end
surf(x1axis,x2axis,x3axis')
hold
a2=a2+ones(3,1)*ones(1,15);
scatter3(a1(1,:),a1(2,:),a1(3,:),'b')
hold
scatter3(a2(1,:),a2(2,:),a2(3,:),'r')
for k=1:length(x1axis)
for n=1:length(x2axis)

15Assignment 5
x3axis(k,n)=(-z(1)/z(3))*x1axis(k)+(-z(2)/z(3))*x2axis(n) +(-z(4)/z(3));
end
end
surf(x1axis,x2axis,x3axis')
The Surface plot is shown below.
Plot:
Hence, from the plot it can be seen that by changing the value of c or the values of constant
matrix in the linear in-equality equations changes the values of the w and b and thus those
point are not inside the surface.
5.
Now, for the data set x3 the training data points are plotted and classifier boundary is
identified by MATLAB function scatter3 and surf with the following MATLAB code.

16Assignment 5
MATLAB code:
a1 = x3'; % training dataset 3
a2=[2;2;2;2]+randn(4,15)/2;
Q=[eye(4) zeros(4,1);zeros(1,4) 0];
f=zeros(5,1);
e1=10.*(-ones(35,1)); % c = 10
A1=[-a1' ones(20,1);a2' ones(15,1)];
z=quadprog(Q,f,A1,e1);
x1axis=[0:.1:25];
x2axis=[0:.1:25];
for k=1:length(x1axis)
for n=1:length(x2axis)
x3axis(k,n)=(-z(1)/z(3))*x1axis(k)+(-z(2)/z(3))*x2axis(n)+(-z(4)/z(3));
end
end
surf(x1axis,x2axis,x3axis')
hold
a2=a2+ones(4,1)*ones(1,15);
scatter3(a1(1,:),a1(2,:),a1(3,:),'b')
hold

17Assignment 5
scatter3(a2(1,:),a2(2,:),a2(3,:),'r')
for k=1:length(x1axis)
for n=1:length(x2axis)
x3axis(k,n)=(-z(1)/z(3))*x1axis(k)+(-z(2)/z(3))*x2axis(n) +(-z(4)/z(3));
end
end
surf(x1axis,x2axis,x3axis')
Plot: