Machine Vision Coursework: Tracking Ping-Pong Ball and Bat Location

This document presents a comprehensive solution to a machine vision project focused on tracking a ping-pong ball and bat in a ping-pong game video. The solution details the implementation of various image processing techniques using MATLAB, including the creation of a custom function "createMask.m" utilizing color thresholding for object identification. It explains the use of the colorThresholder app and the L*a*b color space for segmentation. The solution also covers pixel connectivity, converting RGB images to grayscale and binary, and labeling using the bwlabel function. Furthermore, the document describes the application of the regionprop function to identify objects based on properties like orientation and axis length, along with the use of for loops to label objects with their centroids. It also discusses threshold-based color segmentation techniques, including a two-pass thresholding method and the use of a region of interest (ROI) to improve computational efficiency. The results section shows the ball recognizing and following framework and the experimental evaluation of the system's performance using two video sequences. The document provides a detailed overview of the project, including the algorithms, techniques, and results achieved.

Answer 1-c:
For specific “createMask.m” that function is helping in the identification of bat and ball is created
using color thresholder application of matlab. For creating this function first, you have to load your
image in the workspace of matlab using “imread’ function. After loading the image, you can see
your image in the workspace in the matrix form. Now open the color thresholder app from the
matlab tool bar in the image processing and computer vision tool box option or you can open the
application using “colorThresholder” command in the command window of matlab. After that an
application will open, click on “load image” option on the tool bar of the application and select
“from workspace” option, a small pop up window will open in which you can select your image
which is in the workspace in the form of matrix (true color version). After selection of your image
you will get the option to choose the color space so as per our requirements, we need to work in
L*a*b color space, select the L*a*b color space in the color space selection option.
After selection of color space app opens the image with set of different controls for each color
component of the image. For L*a*b color space the application also shows the histograms which
represents the color components of the image. In this specific color space three histogram will
appear on the right side of the window of the app “L”, which indicated the brightness of the image
second ‘a’ which represents the red-green coordinates of the image and last ‘b’ which represent
the yellow-blue coordinates of the image. You can perform segmentation using the slider controller
of the histogram and by moving them across the spectrum. As we need to create mask for the bat

and ball so we used the draw button to select the ball and bat from the image, after selecting the
bat and all or can say region of interest from the image, the color thresholder automatically
thresholds the image based on the color selected in the region you draw . Also, the color changes
to reflect the segmentation, although the automatic thresholding does not segment the part properly
but you can refine the part using the histogram controls.
And in the image above you can see the minimum and maximum value of threshold, you can
change it accordingly to more refine the results. After the whole process when you got your desired
result you can save the code as well as results by using the export option of the application. You
can use that function by passing and RGB image and this function will create a mask for the desired
ROI without using the application.
Answer 1-d
Labeling-:
1) First important part of labeling to identify the pixel connectivity in the image, pixel
connectivity defines which pixel is connected with the other pixel of the image. Selecting

the connectivity is core part to identify the pixel connectivity. There two main connectivity
type 4 and 8. following table describes about the connectivity.
2) Second step is to convert the RGB image to gray scale and then in binary. A binary image
is required that has value of 1 (white) for all pixel in the input image with luminance greater
than variance and 0 (black) for all other pixel.
3) Now everything is ready for labeling the binary Image that is result of previous level. Type
of Connectivity has been mentioned before can be used in this stage to label our Image.
4) General procedure is as follows-:
• Run-length encodes the input image.
• Scan the runs, assigning preliminary labels and recording label equivalences in a
local equivalence table.
• Resolve the equivalence classes.
• Relabel the runs based on the resolved equivalence classes.

5) This whole algorithm is the part of bwlabel function of matlab, by default the connectivity
setting of function is 8. This function returns label matrix that contains labels for the 8-
connected objects found in binary image.
Answer 1-e
For this specific part we used the “regionprop” function of matlab. Before this step we had
identified the label of each object, so we know the label of each label like the label =2 is for ball
and label =3 is for bat, similarly for other objects in the image. Now how the regionprop function
works. A label matrix which contains the label for each object in the image is passed through the
regionprop function. Element of L equals to 1 is belongs to the first object of the figure or image
and element of matrix L equal to 2 is belongs to the second object of the image and so on. The
question which property is used to identify the bat and ball as you can see in the property table the
orientation and the major, minor axis length are the defining property for the identification of the
objects. Angle between the x-axis and the major axis of the ellipse that has the same second-
moments as the region, returned as a scalar. The value is in degrees, ranging from -90 degrees to
90 degrees.
Answer 1-f
So, the region prop function is the main part of this property finding step of the code. Region prop
function works on the basis of L matrix that is labelled matrix calculated using the above described
algorithm, using regionprop function we can calculate the centroid, major axis, minor axis, area
etc. In the regionprop function either you can define each property separately or you can use “all”
property to calculated various properties of the objects based present in the labeled matrix. Second
part is the for loop, in the for loop basically we are just putting the label of the image in the centroid
of each object. As we have calculated the object’s centroid so we used centroid property to put the
label on the object.
Answer 2-b
The threshold-based color segmentation technique in was at first connected to make a paired
picture distinguishing pixel with a color like the reference object. These pixels were in this manner
converged with neighboring pixels to frame objects. The downside anyway was that identification
was exclusively subject to the limit, which is delicate to commotion. To diminish the sensitivity,

a two-pass thresholding method which has been effectively connected to recognize table tennis
balls in high-goals still pictures, has been reached out to video groupings with changing goals.
This method used different thresholds in each go with the first applying a coarse limit, to discover
all pixels with a comparative shading to the object of interest, which are combined with
neighboring pixels to shape hopeful balls. The point is to inexact the area of every hopeful ball, as
characterized by the focal point of these articles. This methodology can anyway evacuate parts of
the ball, particularly in and around its base because of varieties in shading and light shading. To
recuperate the missing segments, the second pass utilizes a progressively loosened up edge, which
is just connected to those areas in the first edge in the area of an applicant ball. The advantage of
two pass threshold is it slackens limit choice so the incentive in each pass can be less correctly set.
The primary pass gauges the area of a competitor ball, with the decision of various edges just
affecting upon the article estimate while the area remains around consistent. In the second pass,
the district of intrigue (ROI) is confined to it is possible that one or few territories so any blunder
in the edge is unimportant. In simple words you can think it manner that the image is passed two
time from the createMask.m function but every time the threshold values are different (loop).
While two pass threshold makes threshold setting for article division increasingly clear, it is as yet
alluring if the two qualities could be consequently decided. To accomplish this reason, when an
underlying evaluation of the object of interest area is given by the client, the accompanying
alliteratively method tunes the two-pass threshold system, where m, g, u and v are experimentally
characterized constants:
1) In the first pass set the ROI for the present edge to m times the ball measurement, where m
is set to give a bigger ROI so as to tune the Pass I edge. At that point look for the object of
interest utilizing the two-pass threshold calculation.
2) If various competitor balls (objects) are delivered in Pass 1, increase the dimension of the
edge by u% for the following emphasis and retune Pass 1 limit. if no another ball or object
remain after Pass 1, decrease the limit by u% and retune Pass 1 edge.
3) If the maximum number of Pass1 iterations is reached and no suitable threshold is found,
then select the threshold that produced the minimum number of candidate balls and start
tuning the Pass 2 threshold.

4) Calculate area difference between the ball (Ab) and the object (Ao) closest to the given
ball location.
5) If the maximum number of Pass2 tuning iterations is reached with no suitable threshold
being found, select the threshold with lowest (Ab – Ao) value.
Answer 2-b
In a frame, the object of interest involves a little region and searching the whole image is
computationally costly. On the off chance that the area of the object of interest can be evaluated,
a ROI would then be able to be built up which constrains the scan zone for the object of interest
and is focused about the anticipated article area. In this application, when a ball is identified in an
edge, its area in the following casing is anticipated and a versatile method connected to alter the
inquiry region as needs be. This calculation might be condensed as pursues where j and k are the
two constants which are experimentally set amid initialization:
1) In the first frame, set the size of ROI equal to the size of the frame.
2) If the OOI is found, reduce ROI for next frame to a small square of which the length of the
side is j times the diameter of the ball.
3) If no OOI is found then scale the length of the ROI in the next frame by k.
4) If the width (height) of the ROI is greater than the frame width (height), reduce to the frame
width (height).

Answer 2-c
For the specific results.
The video successions demonstrate the ball recognizing and following framework, which is
intended to assess table tennis administration. The top sheet demonstrates the edge that is being
prepared. The yellow square box demonstrates the locale of intrigue (ROI), which is the region the
calculation predicts where the ball is. In the event that a ball is discovered, it will be set apart by a
red circle and its middle will be set apart by a cross. The last three sheets demonstrate the handling
subtleties. The left sheet demonstrates an augmented binarized form of the ROI, where ball
hopefuls are shown by white items. The center sheet demonstrates the ball hopefuls that are being
handled. The location calculation assesses each ball competitor by checking its roundness, adjusted

upper form (red) and size and so on. The blue circle and cross are the anticipated ball form and
focus individually. At long last, the correct sheets demonstrate a plot of the x and y area of the ball
versus the casing number. The plot can be utilized to gauge the ball ascent of the administration.
To experimentally evaluate the performance of the system, two different test video sequences were
used. Each sequence had multiple moving objects and distinctive characteristics in terms of the
type of motion and object occlusion. While the detection rate in the first sequence is 72% and in
second sequence is 45%. In the second sequence the rate is low because of different zoom positions
of camera.
Answer 2-d
To obtain more accurate and reliable measurements, use of multiple cameras should be used. With
many data now feeding in from these cameras, it is suggested to employ a Multi-agent System to
coordinate these complicated processes. Each camera is associated with an agent who can make
independent decision based on the data it received. Many important parameters for this system
should be known like the distance of camera from the object of interest, angle of camera from the
region of interest, height of camera etc.
Answer 3-a
This error is not due to wrong code but it suggests that computer vision system toolbox not being
installed and we are using matlab 2018a/2018b version of matlab. So please check your version of
matlab and which tool box is installed in your system.
Answer 3-b
You can easily calculate the velocity multiplying the distance of centroids between previous frame
and current frame, the frame rate of the video and the scale of the unit is meter/pixel.
Suppose that the video frame rate is fps, (x1, y1) is the centroid position of moving target in the t
frame in video, (x2, y2) is the centroid position of moving target in the t1 frame. So, the speed of
moving object in video can be calculated by formula (5), the unit for it is /s.
The speed calculated based on the above formula is the speed of the moving object in the image,
so the calculation of the actual vehicle movement speed need through the mapping of the pixel

distance with the actual distance. Camera calibration technology which uses the image captured
by camera to restore the objects in space is a commonly and good method to solve the mapping
relationship. Or you can understand it in this way.
To determine the time, it takes for a certain action to occur, use the formula given below:
After getting both your distance and time, use the equation below to determine speed.
Answer 3-d
it is first important to choose and characterize what data is to be utilized for discovering
correspondences. As indicated by this thinking, the main thing evoked in the psyche is taking a
gander at the surface, shading or force of the neighboring zone of a chose point in the principal
outline and expecting that its correspondence on the following casing must likewise has the
equivalent or almost the equivalent auxiliary properties by the methods for surface, power or
shading, and so forth. On the off chance that this thought and related desire truly happen in the
reality, at that point those surface and power data might be utilized for arrangement of the
correspondence issue. Obviously, just this data isn't sufficient alone for the arrangement of the
issue. Since, there may exist numerous hopeful focuses which may have a similar surface and
power attributes. For this situation, an exceptional arrangement can't be ensured. Provided that this
is true, there must be other assumption(s) which will figure the right geometric area of the relating
point.
1) Assumption 1: Intensity values are unchanged.
2) Assumption 2: Location of a point between two successive frames changes by only a few
pixels.

3) Assumption 3: A point behaves together with its neighbors.
The three assumptions help build up a successful target following calculation. So as to follow the
focuses and to figure their paces by utilizing the above suppositions, it is important to express
those presumptions with scientific formalisms and after that speed conditions must be removed by
utilizing these formalisms.
where I(p,t) is the power estimation of a point p on the picture I(t) which was taken at the time
moment t. Note that the geometric area of the fact of the matter is communicated by its position
vector p € R2 (i.e., in 2D space). Here I(p,t) communicates the force estimation of a pixel at the
point p on the casing picture I(t). In comparative manner, the correct side of the condition
communicates the power estimation of the relating pixel at the point p + Δp on the casing picture
I(t + Δt). Likewise, Equation (5) says that the force estimation of the point p on the present picture
outline does not change amid the timespan Δt that passed. As such, it communicates that the force
I(p,t) does not change when Δt. In the more scientific sense, change rate of I(p,t) iz zero over the
timespan Δt.
The values of Ix, Iy and It in Equation can easily be computed from the frame images. The
variables vx and vy are two unknown components of the velocity vector v and these are
respectively the components in the directions x and y axes of the image coordinate system. we
have two questions to be understood, however we just have one condition. Since just a single
condition isn't sufficient for one of a kind of the questions, right now it appears impractical to
illuminate these questions. So as to tackle these two questions, we need progressively autonomous
conditions. For this reason, the third supposition of the LK calculation is utilized. That is, point p

carries on together with its neighbors. So, its neighbors should likewise fulfill the last equation. At
the end of the day, neighbor focuses (or pixels) of point p must move with a similar speed v(vx,vy).
As per these clarifications, indistinguishable conditions from above equations are composed for 3
× 3 or 5 × 5 neighborhood of the point p. For this situation, we absolutely have 9 or 25 conditions
with similar questions vx and vy. Presently the questions can be explained with overdetermined
set of equation by utilizing least squares or all out least squares estimation technique. Amid the
continuous following, some chosen focuses may not be seen on the following edge. This
circumstance may emerge in light of various reasons. Particularly, when the vehicle is going into
or leaving from the FOV of the camera, the likelihood of event of this circumstance is excessively
high. So as to avert such circumstances, we have deciphered the calculation with the picture
pyramid approach, which utilizes coarse to fine picture scale levels. For this situation, in the coarse
dimensions of the pyramid we use equation
to discover the speed vector of each point by finding the uprooting vector Δp of each point by
utilizing coordinated directions and in the better dimension of the pyramid we discover the speed
vectors absolutely by utilizing last equation. We use Equation:
to figure immediate paces of the vehicle, and to register the normal speed of the vehicle we use
Equation
by substituting the processed point speed vectors.
Comment on results -: there are some assumption we made for calculating the real speed from
the sequence of images. Better result and precise calculation of speed of ball and other moving
object we need to calculated precise value of those assumed quantity. As we do not have the data
for comparing our results with the real time speed of ball, we can’t determine how much precise
our result is in comparison to the real calculated speed of the ball.

Answer 4-a
This error is not due to wrong code but it suggests that computer vision system toolbox not being
installed and we are using matlab 2018a/2018b version of matlab. So please check your version of
matlab and which tool box is installed in your system.
Answer 4-b
As in the second sequence the camera was not focused on the same spot and position of the ball
was much more variable as compare to the first sequence. For this specific sequence we have to
basically work on the sensitivity of the threshold part so we changed the mask for second sequence
of images with much more refine process to determine the position of ball and bat. To construct
the mask for this sequence we followed the same process as in the first part using color thresholder
application of the matlab.
Answer 4(d)
it is first important to choose and characterize what data is to be utilized for discovering
correspondences. As indicated by this thinking, the main thing evoked in the psyche is taking a
gander at the surface, shading or force of the neighboring zone of a chose point in the principal
outline and expecting that its correspondence on the following casing must likewise has the
equivalent or almost the equivalent auxiliary properties by the methods for surface, power or
shading, and so forth. On the off chance that this thought and related desire truly happen in the
reality, at that point those surface and power data might be utilized for arrangement of the
correspondence issue. Obviously, just this data isn't sufficient alone for the arrangement of the
issue. Since, there may exist numerous hopeful focuses which may have a similar surface and
power attributes. For this situation, an exceptional arrangement can't be ensured. Provided that this
is true, there must be other assumption(s) which will figure the right geometric area of the relating
point.
1) Assumption 1: Intensity values are unchanged.
2) Assumption 2: Location of a point between two successive frames changes by only a few
pixels.
3) Assumption 3: A point behaves together with its neighbors.