UCL COMPM090 Applied Machine Learning Assignment Solution 2016

This document presents a detailed solution to an Applied Machine Learning assignment (COMPM090) from University College London, focusing on core concepts such as Automatic Differentiation (AutoDiff) and Reverse AutoDiff, including explanations of their algorithms, time complexities, and practical applications. The assignment explores the use of AutoDiff for gradient calculation, particularly in the context of minimizing a loss function. It also covers Long Short-Term Memory (LSTM) networks, detailing their architecture and advantages in handling long-term dependencies. Furthermore, the solution delves into autoencoders, comparing and contrasting them with Principal Component Analysis (PCA) and discussing their respective strengths and weaknesses in dimensionality reduction and feature representation. The document provides comprehensive answers to the assignment questions, offering valuable insights for students studying machine learning.

1. a) Automatic Differentiation (A.D.) is a technique to evaluate the derivatives of a
function defined by a computer program.
Answer:
Customized Differentiation, much equivalent to isolated differences, requires only the
main program P. Nevertheless, instead of executing P on different courses of action of
data sources, it creates another, expanded, program P', that enlists the interpretive
backups close by the primary program. This new program is known as the isolated
program. Precisely, each time the primary program holds some regard v, the isolated
program holds an additional regard dv, the differential of v. Also, each time the principal
program plays out some undertaking, the isolated program plays out additional exercises
dealing with the differential characteristics. For instance, if the first program, sooner or
later amid execution, executes the accompanying guidance on factors a, b, c, and exhibit
T:
a = b*T (10) + c
There are two ways to implement A.D:
 Overloading comprises in telling the compiler that every genuine number is
supplanted by a couple of genuine numbers, the second holding the differential. Each
rudimentary activity on genuine numbers is over-burden, for example inside supplanted
by another one, dealing with sets of reals, that registers the esteem and its differential.
The great position is that the principal program is in every way that really matters
unaltered, since everything is done at accumulate time. The drawback is that the
consequent program runs step by step in light of the way that it generally creates and
wrecks sets of veritable numbers. Additionally, it is hard to execute the "reverse mode"
with over-loading.
 Source change comprises in including into the program the new factors, exhibits,
and information structures that will hold the subsidiaries, and in including the new
guidelines that register these subordinates. The great position is that the consequent
program can be requested into a powerful code, and the "pivot mode" is possible. The
drawback is this is a gigantic change, that is unimaginable by hand on significant
undertakings. Instruments are relied upon to play out this change precisely and rapidly.
Our gathering considers this sort of contraptions. Our Tapenade engine is just a single
such Automatic Differentiation gadget that uses source change.
1. b) Explain what is meant by Reverse AutoDiff. Describe the algorithm and its time
complexity. Give an example to illustrate how the algorithm works.
Answer:
The usage straightforwardness of forward-mode AD accompanies a major drawback,
which ends up clear when we need to compute both ∂z/∂x∂z/∂x and ∂z/∂y∂z/∂y. In

forward-mode AD, doing as such requires seeding with dx = 1 and dy = 0, running the
program, at that point seeding with dx = 0 and dy = 1 and running the program once
more. In actuality, the expense of the strategy scales directly as O(n) where n is the
quantity of info factors. This would be all around expensive on the off chance that we
needed to compute the slope of an expansive convoluted capacity of numerous factors,
which happens shockingly regularly.
1c) Answer:
The usage effortlessness of forward-mode AD accompanies a major detriment, which
winds up clear when we need to figure both ∂z/∂x∂z/∂x and ∂z/∂y∂z/∂y. In forward-mode
AD, doing as such requires seeding with dx = 1 and dy = 0, running the program, at that
point seeding with dx = 0 and dy = 1 and running the program once more. As a result, the
expense of the strategy scales straightly as O(n) where n is the quantity of info factors.
This would be in all respects expensive in the event that we needed to ascertain the slope
of an extensive confounded capacity of numerous factors, which happens shockingly
frequently by and by.
How about we investigate the chain rule (C1) we used to determine forward-mode AD:
(C1)
To figure the slant using forward-mode AD, we expected to perform two substitutions:
one with t=xt=x and another with t=yt=y. This inferred we expected to run the entire
program twice.
Regardless, the chain rule is symmetric: it couldn't mind less what's in the "numerator" or
the "denominator". So, we should change the chain rule anyway flip around the backups:
In doing as such, we have modified the info yield jobs of the factors. A similar naming
show is utilized here: uu for some info variable and wiwi for every one of the yield
factors that rely upon uu. The yet-to-given variable is presently called ss to feature the
adjustment in position.

In this structure, the chain guideline could be connected over and over to each
information variable uu, similar to how in forward-mode AD we connected the chain rule
more than once to each yield variable ww to get condition (F1). Consequently, given
some tt, we expect a program that utilizes chain rule (C2) to have the capacity to process
both ∂s/∂x∂s/∂x and ∂s/∂y∂s/∂y in one go!
Up until now, this is only a hunch. How about we attempt it on the precedent issue (A).
In the event that you haven't done this previously, I recommend setting aside the effort
to really determine these conditions utilizing (C2). It tends to be very personality
bowing in light of the fact that everything appears "in reverse": rather than asking what
input factors a given yield variable relies upon, we need to ask what yield factors a
given information variable can influence. The most effortless approach to see this
outwardly is by illustration a reliance chart of the articulation:

The diagram demonstrates that
 the variable a straightforwardly relies upon x and y,
 the variable b straightforwardly relies upon x, and
 the variable z straightforwardly relies upon an and b.
Or on the other hand, proportionally:
 the variable b can straightforwardly influence z,
 the variable a can legitimately influence z,
 the variable y can legitimately influence an, and
 the variable x can legitimately influence an and b
 Going back to the conditions (R1), we see that on the off chance that we substitute
s=zs=z, we would get the angle in the last two conditions. In the program, this is
comparable to setting gz = 1 since gz is simply ∂s/∂z∂s/∂z. We never again need to run
the program twice! This is turn around mode programmed separation.
 There is an exchange off, obviously. In the event that we need to figure the
subsidiary of an alternate yield variable, at that point we would need to re-run the
program again with various seeds, so the expense of turn around mode AD is O(m)where
m is the quantity of yield factors. In the event that we had an alternate precedent, for
example,
{z=2x+sin(x)v=4x+cos(x)
{z=2x+sin (x)v=4x+cos (x)
 in invert mode AD we would need to run the program with gz = 1and gv = 0 (for
example s=zs=z) to get ∂z/∂x∂z/∂x, and afterward rerun the program with gz = 0 and gv =
1 (for example s=vs=v) to get ∂v/∂x∂v/∂x. Conversely, in forward-mode AD, we'd quite
recently set dx = 1 and get both ∂z/∂x∂z/∂x and ∂v/∂x∂v/∂x in one run.
 There is a progressively inconspicuous issue with switch mode AD, be that as it may:
we can't simply interleave the subsidiary counts with the assessments of the first
articulation any longer, since all the subordinate computations give off an impression of
being going backward to the first program. Besides, it's not clear how one would even
touch base now in utilizing a basic guideline-based calculation – is administrator over-

burdening even a legitimate methodology here? How would we put the "programmed"
once again into switch mode AD.
1d) Answer:
The execution straightforwardness of forward-mode AD accompanies a major
impediment, which winds up apparent when we need to ascertain both ∂z/∂x∂z/∂x and
∂z/∂y∂z/∂y. In forward-mode AD, doing as such requires seeding with dx = 1 and dy = 0,
running the program, at that point seeding with dx = 0 and dy = 1 and running the
program once more. As a result, the expense of the technique scales straightly as O(n)
where n is the quantity of information factors. This would be all around expensive on the
off chance that we needed to figure the inclination of a huge entangled capacity of
numerous factors, which happens shockingly frequently by and by.
We should investigate the chain rule (C1) we used to determine forward-mode AD:
1e) Answer:
Long Short-Term Memory systems – normally just called "LSTMs" – are an uncommon
sort of RNN, equipped for adapting long haul conditions. They were presented by
Hochreiter and Schmidhuber (1997), and were refined and advanced by numerous
individuals in following work.1 They work enormously well on a huge assortment of
issues, and are currently broadly utilized.
LSTMs are unequivocally intended to maintain a strategic distance from the long-haul
reliance issue. Recollecting data for significant lots of time is for all intents and purposes
their default conduct, not something they battle to learn!
All repetitive neural systems have the type of a chain of rehashing modules of neural
system. In standard RNNs, this rehashing module will have a straightforward structure, for

example, a solitary tanh layer.
The repeating module in a standard RNN contains a single layer.
LSTMs additionally have this chain like structure; however, the rehashing module has an
alternate structure. Rather than having a solitary neural system layer, there are four,
interfacing in an exceptionally extraordinary manner.
The repeating module in an LSTM contains four interacting layers.
Try not to stress over the subtleties of what's happening. We'll stroll through the LSTM
chart well-ordered later. Until further notice, allows simply endeavor to get settled with the
documentation we'll be utilizing.

2a) Answer:
In the above graph, each line conveys a whole vector, from the yield of one hub to the
contributions of others. The pink circles speak to pointwise activities, similar to vector
expansion, while the yellow boxes are scholarly neural system layers. Lines combining
indicate connection, while a line forking signify its substance being duplicated and the
duplicates going to various areas.
2b) Answer:
All things considered, an autoencoder is a unique sort of feed forward neural system
which encodes input x into concealed layer h and after that translates it once more from its
shrouded portrayal. The model is prepared to limit the misfortune b/w the info and yield
layer.
An autoencoder where dim(h) < dim(xi) is called an under complete autoencoder
Presently, suppose with the assistance of concealed layer h, you can recreate xhat
perfectly, h is lossless encoding of xi. It catches immeasurably significant qualities of xi.
The similarity with PCA is clear, h carries on like PCA's diminished measurements
framework from which the yield is recreated with some misfortune in esteem.
Subsequently, the encoder part demonstrates similarity to PCA.

How to achieve this equivalence? Can this equivalence be of any use?
Encoder part will be proportionate to PCA if straight encoder, direct decoder, square
blunder misfortune work with standardized sources of info are utilized. Which implies
PCA is confined to direct maps just while autoencoders are definitely not.
Due to these linearity constraints, we moved to encoders with sigmoid-like non-linear
functions which give more accuracy in the reconstruction of data. See an illustration
related to it.
2c) Answer:
We've dug into the ideas driving PCA and Autoencoders all through this article.
Tragically, there is no general predominant model. The choice between the PCA and
Autoencoder models must be done on a conditional premise. As a rule, PCA is superior —
it's quicker, increasingly interpretable and can decrease the dimensionality of your
information the same amount of as an Autoencoder can. In the event that you can utilize
PCA, you should. In any case, in case you're working with information that requires an
exceedingly non-straight component portrayal for satisfactory execution or perception,
PCA may miss the mark. For this situation, it might merit using the push to prepare
Autoencoders. Of course, regardless of whether the inactive highlights created by an
Autoencoder increment model execution, the obscureness of those highlights represents a
boundary to learning disclosure.
2d) Answer:
For the most part, you can think about autoencoders as an unsupervised learning system,
since you don't require express names to prepare the model on. All you have to prepare
an autoencoder is crude information.
Autoencoder

As you read in the presentation, an autoencoder is an unsupervised AI calculation that
accepts a picture as information and attempts to remake it utilizing a smaller number of
bits from the bottleneck otherwise called inert space. The picture is significantly
compacted at the bottleneck. The pressure in autoencoders is accomplished via preparing
the system for a timeframe and as it learns it endeavors to best speak to the information
picture at the bottleneck. The general picture pressure calculations like JPEG and JPEG
lossless pressure systems pack the pictures without the requirement for any sort of
preparing and do genuinely well in compacting the pictures.
Autoencoders are like dimensionality decrease methods like Principal Component
Analysis (PCA). They anticipate the information from a higher measurement to a lower
measurement utilizing direct change and attempt to protect the significant highlights of
the information while expelling the insignificant parts.
In any case, the significant contrast among autoencoders and PCA lies in the change part:
as you previously read, PCA utilizes direct change though autoencoders use non-straight
changes.
Since you have a touch of comprehension about autoencoders, we should now break this
term and attempt to get some instinct about it!
The above figure is a two-layer vanilla autoencoder with one concealed layer. In
profound learning phrasing, you will frequently see that the info layer is never considered

while tallying the all-out number of layers in an engineering. The all-out layers in a
design just contains the quantity of concealed layers and the output layer.
As appeared in the picture over, the information and yield layers have a similar number
of neurons.
We should take a model. You feed a picture with only five-pixel esteems into the
autoencoder which is compacted by the encoder into three-pixel esteems at the bottleneck
(center layer) or inactive space. Utilizing these three qualities, the decoder attempts to
remake the five-pixel esteems or rather the info picture which you encouraged as a
contribution to the system. Truly, there are progressively number of concealed layers in
the middle of the info and the yield.

Autoencoder can be broken in to three sections
• Encoder: this piece of the system packs or down samples the contribution to a smaller
number of bits. The space spoken to by these smaller number of bits is regularly called
the inactive space or bottleneck. The bottleneck is additionally called the "most extreme
purpose of pressure" since now the info is compacted the greatest. These compacted bits
that speak to the first info are as one called an "encoding" of the information.
• Decoder: this piece of the system attempts to reproduce the info utilizing just the
encoding of the information. At the point when the decoder can reproduce the info
precisely as it was sustained to the encoder, you can say that the encoder can create the
best encodings for the contribution with which the decoder can remake well!
There is assortment of autoencoders, for example, the convolutional autoencoder,
denoising autoencoder, variational autoencoder and scanty autoencoder. Be that as it
may, as you read in the presentation, you'll just concentrate on the convolutional and
denoising ones in this instructional exercise.

2e) Answer:
Up until this point, we have depicted the use of neural systems to regulated learning, in
which we have marked preparing models. Presently assume we have just a lot of
unlabeled preparing precedents {x (1), x (2), x (3),}, where x(i)∈, setting the objective
qualities to be equivalent to the information sources. I.e., it utilizes y(i)=x(i).
Here is an autoencoder:
The autoencoder endeavors to gain proficiency with a capacity hW, b(x)≈x. As it were, it
is endeavoring to gain proficiency with an estimation to the character work, to yield x^
that is like x. The character work appears an especially minor capacity to attempt adapt;
yet by setting requirements on the system, for example, by constraining the quantity of
shrouded units, we can find intriguing structure about the information. As a solid model,
assume the information sources x are the pixel power esteems from a 10×10 picture (100
pixels) so n=100, and there are s2=50 concealed units in layer L2. Note that we
additionally have y∈R100. Since there are just 50 shrouded units, the system is compelled
to become familiar with a "packed" portrayal of the info. I.e., given just the vector of
shrouded unit actuations a (2) ∈R50, it must endeavor to "'remake"' the 100-pixel input x.
On the off chance that the info was totally irregular—state, every xi originates from an IID
Gaussian free of different highlights—at that point this pressure errand would be