GATE DA

Multi-layer perceptrons and feed-forward neural networks

A feed-forward neural network is a stack of layers, each computing a weighted sum of the previous layer's outputs followed by a non-linear activation. GATE DA questions on it ask you to run the perceptron rule, count parameters, do a forward pass by hand, use activation derivatives, and take one backpropagation or gradient-descent step on a small network.

5 previous-year questions · 5 solved · 7 practice questions

The perceptron and linear separability

Convention used throughout. Vectors are bold and are column vectors ( $w$ , $x$ ). $w$ holds weights and $b$ is the bias. A single input is $x$ . For a whole data set the samples are the rows of $X$ . $ln$ and $lo g$ mean the natural logarithm. $η > 0$ is the learning rate. $∥ \cdot ∥$ is the Euclidean norm.

A perceptron computes a weighted sum of its inputs plus a bias, called the net input $z$ , and passes it through a step function. Geometrically, $w^{⊤} x + b = 0$ is a hyperplane (a line in 2-D). The perceptron answers 1 on one side of it and 0 on the other. The weight vector $w$ is perpendicular to the hyperplane and points towards the side where the output is 1. The bias shifts the hyperplane away from the origin: its distance from the origin is $∣ b ∣/ ∥ w ∥$ , and the signed distance of a point $x$ from it is $(w^{⊤} x + b) / ∥ w ∥$ .

PERCEPTRON OUTPUT (STEP ACTIVATION)

z = w^{⊤} x + b, \overset{y}{^} = {10 z \geq 0 z < 0

Targets are 0 or 1. The tie z = 0 is assigned to class 1 here. A question may use the opposite tie rule or targets of -1 and +1, so read the convention given.

PERCEPTRON LEARNING RULE

w \leftarrow w + η (t - \overset{y}{^}) x, b \leftarrow b + η (t - \overset{y}{^})

Applied sample by sample, where t is the target. If the sample is classified correctly, t - y is 0 and nothing changes. A missed class-1 sample pulls w towards x, and a missed class-0 sample pushes w away from x.

Convergence. For a finite data set and any $η > 0$ : if the two classes are linearly separable, the perceptron rule reaches a separating hyperplane after a finite number of updates, whatever the starting weights. If they are not separable, mistakes keep occurring and the weights never settle.

BOUND ON THE NUMBER OF PERCEPTRON MISTAKES

number of updates \leq (\frac{R}{γ})^{2}, R = i max ∥ x_{i} ∥, γ = i min t_{i} u^{⊤} x_{i} > 0

Assumptions: targets are -1 and +1, the bias is absorbed by appending a constant 1 to every x, the weights start at zero, and u is a unit-length vector that separates the data with margin gamma. A larger margin or smaller inputs mean fewer updates.

XOR. The points (0,0) and (1,1) belong to class 0 and (0,1) and (1,0) to class 1. No single line separates them, so no single perceptron computes XOR. AND, OR, NAND and NOR are all linearly separable. XOR can be computed by one hidden layer of two step units feeding one output unit, as (x1 OR x2) AND (x1 NAND x2). One choice of weights: $h_{1} = step (x_{1} + x_{2} - 0.5)$ (OR), $h_{2} = step (1.5 - x_{1} - x_{2})$ (NAND), $y = step (h_{1} + h_{2} - 1.5)$ (AND). Check: (0,0) gives $h = (0, 1)$ , $y = 0$ ; (0,1) and (1,0) give $h = (1, 1)$ , $y = 1$ ; (1,1) gives $h = (1, 0)$ , $y = 0$ .

Multi-layer perceptrons: structure and parameter counting

A feed-forward network (multi-layer perceptron, MLP) has an input layer, one or more hidden layers and an output layer. Every unit in a layer receives all outputs of the previous layer (a fully connected or dense layer), and information flows only forward, with no loops. Each non-input unit has its own weight for every incoming connection and one bias.

LAYER-WISE FORWARD PROPAGATION

z^{(l)} = W^{(l)} a^{(l - 1)} + b^{(l)}, a^{(l)} = f^{(l)} (z^{(l)}), a^{(0)} = x

W(l) has one row per unit of layer l and one column per unit of layer l-1. f is applied element-wise, except softmax, which acts on the whole vector.

FORWARD PROPAGATION FOR A WHOLE DATA SET (SAMPLES IN ROWS)

Z^{(l)} = A^{(l - 1)} (W^{(l)})^{⊤} + 1 (b^{(l)})^{⊤}, A^{(0)} = X

With N samples, A(l-1) is N by n_(l-1), Z(l) is N by n_l, and 1 is a column of N ones, so every sample gets the same bias. Each row of Z(l) is the transpose of z(l) for one sample.

NUMBER OF PARAMETERS IN A DENSE LAYER

n_{l - 1} n_{l} + n_{l} = n_{l} (n_{l - 1} + 1)

n_l is the number of units in layer l. The network total is the sum over all layers after the input. The first term counts weights, the second counts biases.

Procedure for a forward pass by hand

Write the input vector as a(0).
For each unit of the next layer compute z = (sum of weight times input) + bias.
Apply the activation function to each z to get that unit's output.
Treat these outputs as the inputs of the following layer and repeat until the output layer.
Apply the output activation (linear, sigmoid or softmax) and read off the answer.

WORKED EXAMPLE

How many trainable parameters does a fully connected network with layer sizes 4-6-3-2 (4 inputs, hidden layers of 6 and 3 units, 2 outputs) have, counting biases?

Solution

Layer 1 (4 to 6): 4*6 weights + 6 biases = 30.
Layer 2 (6 to 3): 6*3 + 3 = 21.
Layer 3 (3 to 2): 3*2 + 2 = 8.
Total = 30 + 21 + 8 = 59.

Answer: 59

WORKED EXAMPLE

A 2-2-1 network has hidden ReLU units and a linear output unit. Hidden unit 1 has weights (1, -1) and bias 0.5. Hidden unit 2 has weights (2, 1) and bias -1. The output unit has weights (2, 1) on the two hidden outputs and bias 0.5. Find the output for the input x = (1, 2).

Solution

Hidden 1: z = 1*1 + (-1)*2 + 0.5 = -0.5, so the ReLU output is 0.
Hidden 2: z = 2*1 + 1*2 - 1 = 3, so the ReLU output is 3.
Output: 2*0 + 1*3 + 0.5 = 3.5 (linear output, no activation).

Answer: 3.5

Activation functions and why they must be non-linear

If every layer used the identity activation, the whole network would be a composition of affine maps, which is itself a single affine map: $W^{(2)} (W^{(1)} x + b^{(1)}) + b^{(2)} = W^{'} x + b^{'}$ with $W^{'} = W^{(2)} W^{(1)}$ and $b^{'} = W^{(2)} b^{(1)} + b^{(2)}$ . Depth would add nothing, and such a network could not even represent XOR. A non-linear hidden activation is what lets an MLP represent non-linear decision boundaries.

Universal approximation. A feed-forward network with one hidden layer of finitely many units, a non-polynomial continuous activation such as sigmoid, tanh or ReLU, and a linear output unit can approximate any continuous function on a closed, bounded input region to any required accuracy, provided it has enough hidden units. The result says nothing about how many units are needed, or whether gradient descent will find suitable weights.

Common activation functions

Name	f(z)	f'(z)	Range	Notes
Step	1 if z >= 0, else 0	0 (undefined at 0)	{0, 1}	Not usable with gradient descent
Sigmoid	1 / (1 + e^(-z))	f(z)(1 - f(z))	(0, 1)	Largest slope is 0.25, at z = 0; saturates for large \|z\|
Tanh	(e^z - e^(-z)) / (e^z + e^(-z))	1 - f(z)^2	(-1, 1)	Zero-centred; largest slope is 1, at z = 0; tanh(z) = 2*sigmoid(2z) - 1
ReLU	max(0, z)	1 if z > 0, 0 if z < 0 (undefined at 0; taken as 0 or 1 by convention)	[0, infinity)	Slope is 0 for negative z, so a unit with z < 0 passes no gradient; a unit whose z is negative for every input stops learning (a dead unit)
Softmax	e^(z_i) / sum_j e^(z_j)	dp_i/dz_i = p_i(1 - p_i); dp_i/dz_j = -p_i p_j for i != j	(0, 1), sums to 1	Output layer for multi-class problems

SIGMOID AND ITS DERIVATIVE

σ (z) = \frac{1}{1 + e ^{- z}}, σ^{'} (z) = σ (z) (1 - σ (z))

Write the derivative in terms of the output y = sigma(z), so backpropagation needs no recomputation. Also 1 - sigma(z) = sigma(-z).

TANH AND ITS DERIVATIVE

tanh^{'} (z) = 1 - tanh^{2} (z)

Valid for all real z.

SOFTMAX

p_{i} = \frac{e ^{z_{i}}}{\sum _{j = 1}^{K} e ^{z_{j}}}

K output units. Outputs are positive and sum to 1. Adding the same constant to every z_i leaves p unchanged.

TWO-CLASS SOFTMAX IS A SIGMOID

K = 2 : p_{1} = \frac{e ^{z_{1}}}{e ^{z_{1}} + e ^{z_{2}}} = σ (z_{1} - z_{2})

Only the difference of the two logits matters, which is why one sigmoid unit is enough for binary classification.

EXAM TIP

Backpropagation multiplies each sigmoid layer's contribution by an activation slope of at most 0.25. Across many layers this product tends to make the gradient very small (vanishing gradient), unless the weights are large. Where a ReLU unit is active its slope is exactly 1, so the activation itself adds no shrinking factor.

Loss functions

Training minimises a loss that measures the mismatch between the network output $\overset{y}{^}$ and the target $t$ . Squared error is used for regression, and cross-entropy is used for classification. All logarithms below are natural.

SQUARED-ERROR LOSS (ONE SAMPLE)

L = \frac{1}{2} (\overset{y}{^} - t)^{2}, \frac{\partial L}{\partial y ^} = \overset{y}{^} - t

The factor 1/2 is a convention that cancels the 2 on differentiation. If a question defines L without it, the gradient is doubled.

BINARY CROSS-ENTROPY (ONE SAMPLE, SIGMOID OUTPUT)

L = - [t ln \overset{y}{^} + (1 - t) ln (1 - \overset{y}{^})], \frac{\partial L}{\partial z} = \overset{y}{^} - t

t is 0 or 1 and y = sigmoid(z). The gradient with respect to the output unit's net input z simplifies to y - t.

CATEGORICAL CROSS-ENTROPY WITH SOFTMAX

L = - i = 1 \sum K t_{i} ln p_{i}, \frac{\partial L}{\partial z _{i}} = p_{i} - t_{i}

t is a one-hot vector. With true class c, the loss is -ln p_c.

Matching output layer and loss

Task	Output activation	Loss	Output delta dL/dz
Regression	Linear (identity)	Squared error (with 1/2)	y - t
Binary classification	Sigmoid	Binary cross-entropy	y - t
Multi-class classification	Softmax	Categorical cross-entropy	p_i - t_i
Any, if a question pairs them	Sigmoid	Squared error (with 1/2)	(y - t) y (1 - y)

Backpropagation and gradient descent

Backpropagation applies the chain rule layer by layer, starting at the output. For each unit it computes the error signal $δ = \partial L / \partial z$ , which is then used to get the gradients for that unit's weights and bias. Gradient descent then moves every parameter against its gradient.

GRADIENT-DESCENT UPDATE

θ \leftarrow θ - η \frac{\partial L}{\partial θ}

theta is any weight or bias. All gradients are computed at the old parameter values, and then all parameters are updated together.

OUTPUT-LAYER ERROR SIGNAL FOR SQUARED ERROR

δ^{(L)} = (a^{(L)} - t) ⊙ f^{'} (z^{(L)})

Squared-error loss with factor 1/2, output activation f applied element-wise. For a single output unit this is (y - t) f'(z). For sigmoid output, f' = y(1 - y); for linear output, f' = 1. With cross-entropy and sigmoid or softmax, delta = y - t directly.

HIDDEN-LAYER ERROR SIGNAL

δ^{(l)} = ((W^{(l + 1)})^{⊤} δ^{(l + 1)}) ⊙ f^{'} (z^{(l)})

The symbol is the element-wise product. The error of a hidden unit is the weighted sum of the errors of the units it feeds, times its own activation slope. Use the weights as they were before this step's update.

WEIGHT AND BIAS GRADIENTS

\frac{\partial L}{\partial W ^{(l)}} = δ^{(l)} (a^{(l - 1)})^{⊤}, \frac{\partial L}{\partial b ^{(l)}} = δ^{(l)}

For a single weight: dL/dw = delta of the receiving unit times the output of the sending unit.

One training step

Forward pass: compute and store every z and a.
Compute the loss at the output.
Compute the output delta, then work backwards to get the delta of each earlier layer.
Compute the gradient of every weight and bias.
Update all parameters with the learning rate.

Batch, stochastic and mini-batch. Batch gradient descent uses the gradient of the loss over the whole training set for each update. Stochastic gradient descent (SGD) updates after each single sample. Mini-batch gradient descent updates after each small group of samples. One pass through the whole training set is an epoch. A learning rate that is too large can make the loss oscillate or diverge, and one that is too small makes progress slow.

GRADIENT OVER A BATCH OF N SAMPLES (MEAN LOSS)

θ \leftarrow θ - η \frac{1}{N} n = 1 \sum N \frac{\partial L _{n}}{\partial θ}

L_n is the loss on sample n. If a question defines the total loss as a sum rather than a mean, drop the 1/N.

L2 REGULARISATION (WEIGHT DECAY)

L_{reg} = L + \frac{λ}{2} ∥ w ∥^{2}, w \leftarrow (1 - η λ) w - η \frac{\partial L}{\partial w}

lambda >= 0 sets the strength. The penalty shrinks the weights towards zero at every step and is used against overfitting. Biases are usually not penalised.

WORKED EXAMPLE

A single sigmoid neuron has input x = 1, weight w = 0, bias b = 0 and target t = 1. The loss is L = (1/2)(y - t)^2 with y = sigmoid(wx + b). Find the weight after one gradient-descent step with eta = 2.

Solution

z = 0*1 + 0 = 0, so y = sigmoid(0) = 0.5.
dL/dw = (y - t) y(1 - y) x = (-0.5)(0.25)(1) = -0.125.
w_new = 0 - 2*(-0.125) = 0.25.

Answer: 0.25

WORKED EXAMPLE

A 1-1-1 network has one ReLU hidden unit h = ReLU(w1 x + b1) and a linear output y = w2 h + b2. Initially w1 = 1, b1 = 0, w2 = 2, b2 = 0. For x = 1, t = 1, L = (1/2)(y - t)^2 and eta = 0.1, find all four parameters after one gradient-descent step.

Solution

Forward: z1 = 1*1 + 0 = 1, h = 1; y = 2*1 + 0 = 2.
Output delta (linear output): delta2 = y - t = 1.
dL/dw2 = delta2 * h = 1; dL/db2 = delta2 = 1.
Hidden delta, using the OLD w2: delta1 = w2 delta2 ReLU'(z1) = 2 1 1 = 2.
dL/dw1 = delta1 * x = 2; dL/db1 = delta1 = 2.
Update: w2 = 2 - 0.1*1 = 1.9, b2 = 0 - 0.1*1 = -0.1, w1 = 1 - 0.1*2 = 0.8, b1 = 0 - 0.1*2 = -0.2.

Answer: w1 = 0.8, b1 = -0.2, w2 = 1.9, b2 = -0.1

EXAM TIP

Check the loss definition before using a gradient. With squared error and a sigmoid output the factor y(1 - y) is present, so the update is small when the output is saturated near 0 or 1. With cross-entropy that factor cancels and the output delta is just y - t.

Exam traps and quick checks

Quick reference

Question type	What to do
Can one perceptron do this function?	Check linear separability. AND, OR, NAND, NOR yes. XOR and XNOR no.
How many parameters?	Sum n_l(n_(l-1) + 1) over all layers after the input.
Output of a small network	Do the forward pass layer by layer and apply the activation after each layer's sum, including the biases.
What does a linear hidden layer add?	Nothing. Stacked linear layers equal one linear layer.
One update step on a sigmoid unit	dL/dw = (y - t) y (1 - y) x for L = (1/2)(y - t)^2, and (y - t) x for binary cross-entropy. For the bias, drop the x.

EXAM TIP

Perceptron updates use the error t - y, so with targets 0 and 1 a correct sample gives no change. If targets are -1 and +1 and the output is the sign of z, the usual update on a mistake is w <- w + eta*t*x and b <- b + eta*t. The general rule would give twice this, because t - y = 2t on a mistake, and the usual form absorbs the 2 into eta.

EXAM TIP

The perceptron rule needs only finite steps if the data are separable, but the number of steps and the final weights depend on the order of samples and the starting point. Do not assume a unique answer unless the order is specified.

EXAM TIP

If every weight in a layer starts at the same value (for example all zero), every unit in that layer computes the same output and receives the same gradient, so the units stay identical after every update. Random initialisation breaks this symmetry.

EXAM TIP

Softmax outputs are unchanged when the same constant is added to all logits. A question that gives logits can often be simplified by subtracting the largest one.

PREVIOUS-YEAR QUESTIONS

5 questions from the DA papers

Try each question first, then open the answer. We placed each question under a topic ourselves; the exam paper does not label topics.

See these papers in full: GATE DA 2026 · GATE DA 2025 · GATE DA 2024

GATE 2026 DA Q.56

GATE 2026 · DAQ.56 · 2 marks · NAT

Consider a fully-connected feed-forward multi-layer perceptron. It has 30 neurons in the input layer, followed by two hidden layers and an output layer. The first hidden layer has 4 neurons and the second hidden layer has 3 neurons. The output layer has only one neuron. Assume that no bias parameters are used.

The number of learnable parameters in the multi-layer perceptron is ______. (Answer in integer)

Answer and solution

GATE 2026 DA Q.56: answer and solution

See the question in the paper

Tap to open full size

GATE 2026 DA Q.56: answer and solution

GATE 2025 DA Q.22

GATE 2025 · DAQ.22 · 1 mark · MCQ

Consider designing a linear classifier $y = sign (f (x; w, b))$ , $f (x; w, b) = w^{⊤} x + b$ on a dataset $D = {(x_{1}, y_{1}), (x_{2}, y_{2}), \dots, (x_{N}, y_{N})}$ , $x_{i} \in R^{d}$ , $y_{i} \in {+ 1, - 1}$ , $i = 1, 2, \dots, N$ . Recall that the sign function outputs $+ 1$ if the argument is positive, and $- 1$ if the argument is non-positive. The parameters $w$ and $b$ are updated as per the following training algorithm: $w_{n e w} = w_{o l d} + y_{n} x_{n}$ , $b_{n e w} = b_{o l d} + y_{n}$ whenever $sign (f (x_{n}; w_{o l d}, b_{o l d})) \neq = y_{n}$ . In other words, whenever the classifier wrongly predicts a sample $(x_{n}, y_{n})$ from the dataset, $w_{o l d}$ gets updated to $w_{n e w}$ , and likewise $b_{o l d}$ gets updated to $b_{n e w}$ . Consider the case $(x_{n}, + 1)$ , $f (x_{n}; w_{o l d}, b_{o l d}) < 0$ . Then

(A)
$f (x_{n}; w_{n e w}, b_{n e w}) > f (x_{n}; w_{o l d}, b_{o l d})$
(B)
$f (x_{n}; w_{n e w}, b_{n e w}) < f (x_{n}; w_{o l d}, b_{o l d})$
(C)
$f (x_{n}; w_{n e w}, b_{n e w}) = f (x_{n}; w_{o l d}, b_{o l d})$
(D)
$y_{n} f (x_{n}; w_{o l d}, b_{o l d}) > 1$

Answer and solution

GATE 2025 DA Q.22: answer and solution

See the question in the paper

Tap to open full size

GATE 2025 DA Q.22: answer and solution

GATE 2025 DA Q.42

GATE 2025 · DAQ.42 · 2 marks · MCQ

Consider the neural network shown in the figure with

inputs: $u, v$
weights: $a, b, c, d, e, f$
output: $y$

$R$ denotes the ReLU function, $R (x) = max (0, x)$ .

Given $u = 2$ , $v = 3$ ,
$a = 1, b = 1, c = 1, d = - 1, e = 4, f = - 1$ ,

which one of the following is correct?

GATE 2025 DA question 42: Feed-forward network. — Figure from the GATE 2025 DA paper. It is part of the question.

(A)
∂y/∂a = 8, ∂y/∂f = 0
(B)
∂y/∂a = 1, ∂y/∂f = 0
(C)
∂y/∂a = 1, ∂y/∂f = −1
(D)
∂y/∂a = 2, ∂y/∂f = −1

Answer and solution

GATE 2025 DA Q.42: answer and solution

GATE 2025 DA Q.48

GATE 2025 · DAQ.48 · 2 marks · MSQ

Which of the following statements is/are correct about the rectified linear unit (ReLU) activation function defined as $ReLU (x) = max (x, 0)$ , where $x \in R$ ?

(A)
ReLU is continuous everywhere
(B)
ReLU is differentiable everywhere
(C)
ReLU is not differentiable at $x = 0$
(D)
$ReLU (x) = ReLU (a x)$ , for all $a \in R$

Answer and solution

GATE 2025 DA Q.48: answer and solution

See the question in the paper

Tap to open full size

GATE 2025 DA Q.48: answer and solution

GATE 2024 DA Q.43

GATE 2024 · DAQ.43 · 2 marks · MCQ

Consider the two neural networks (NNs) shown in Figures 1 and 2, with $R e LU$ activation ( $R e LU (z) = max {0, z}, \forall z \in R$ ). $R$ denotes the set of real numbers. The connections and their corresponding weights are shown in the Figures. The biases at every neuron are set to 0. For what values of $p, q, r$ in Figure 2 are the two NNs equivalent, when $x_{1}, x_{2}, x_{3}$ are positive?

GATE 2024 DA question 43: Figure 1: a feed-forward network with inputs x1, x2, x3, a first hidden layer of 3 neurons (top, middle, bottom), a second… — Figure from the GATE 2024 DA paper. It is part of the question.

(A)
$p = 36, q = 24, r = 24$
(B)
$p = 24, q = 24, r = 36$
(C)
$p = 18, q = 36, r = 24$
(D)
$p = 36, q = 36, r = 36$

Answer and solution

GATE 2024 DA Q.43: answer and solution

PRACTICE

7 questions to test yourself

PRACTICE 11 mark · MCQ

A fully connected feed-forward network has 10 inputs, one hidden layer of 5 units and a single output unit. Every non-input unit has a bias. How many trainable parameters does it have?

(A)
50
(B)
55
(C)
61
(D)
66

Answer and solution

Answer: (C)

Input to hidden: 10*5 weights + 5 biases = 55.
Hidden to output: 5*1 weights + 1 bias = 6.
Total = 55 + 6 = 61.

PRACTICE 21 mark · MSQ

A single perceptron with a step activation, weights, and a bias is used for two binary inputs x1 and x2 (values 0 or 1). Which of the following functions can it compute exactly?

(A)
AND
(B)
OR
(C)
XOR
(D)
NAND

Answer and solution

Answer: (A), (B), (D)

AND, OR and NAND each have a line that separates the 1-outputs from the 0-outputs (for example x1 + x2 - 1.5 >= 0 for AND, x1 + x2 - 0.5 >= 0 for OR, and 1.5 - x1 - x2 >= 0 for NAND).
XOR puts (0,1) and (1,0) on one class and (0,0) and (1,1) on the other, which no line can separate.
So the answers are AND, OR and NAND.

PRACTICE 32 marks · NAT

A 2-3-1 network has ReLU hidden units and a linear output unit. For the input x = (2, 1): hidden unit 1 has weights (1, -2) and bias 1; hidden unit 2 has weights (-1, 1) and bias 0; hidden unit 3 has weights (1, 1) and bias -1. The output unit has weights (1, 3, -1) on the hidden outputs and bias 0.5. Find the output value.

Answer and solution

Answer: $- 0.5$ (accepted range $\pm 0.01$ )

Hidden 1: z = 2 - 2 + 1 = 1, ReLU gives 1.
Hidden 2: z = -2 + 1 + 0 = -1, ReLU gives 0.
Hidden 3: z = 2 + 1 - 1 = 2, ReLU gives 2.
Output: 1*1 + 3*0 + (-1)*2 + 0.5 = -0.5.

PRACTICE 42 marks · NAT

A single sigmoid neuron has input x = 2, weight w = 0 and bias b = 0 (the bias is not updated in this question). The target is t = 1 and the loss is L = (1/2)(y - t)^2 where y = sigmoid(wx + b). One step of gradient descent with learning rate 4 is applied to w. Find the new value of w.

Answer and solution

Answer: $1$ (accepted range $\pm 0.01$ )

z = 0, so y = 0.5.
dL/dw = (y - t) y(1 - y) x = (-0.5)(0.25)(2) = -0.25.
w_new = 0 - 4*(-0.25) = 1.

PRACTICE 52 marks · MCQ

A 3-class classifier has a softmax output layer. For one training sample the output-layer net inputs are z = (0, ln 3, 0), and the true class is class 2 (the middle one). What is the categorical cross-entropy loss for this sample, using natural logarithms?

(A)
0.511
(B)
0.916
(C)
1.609
(D)
0.223

Answer and solution

Answer: (A)

e^z = (1, 3, 1), whose sum is 5.
p = (0.2, 0.6, 0.2), so p for the true class is 0.6.
L = -ln 0.6 = 0.511.
The other options are -ln 0.4 = 0.916, ln 5 = 1.609 and -ln 0.8 = 0.223, which do not match.

PRACTICE 62 marks · NAT

A 1-1-1 network has one sigmoid hidden unit h = sigmoid(w1 x + b1) and a linear output unit y = w2 h + b2. The current values are w1 = 0, b1 = 0, w2 = 4, b2 = 0. For the input x = 2 with target t = 1 and loss L = (1/2)(y - t)^2, find the partial derivative of L with respect to w1.

Answer and solution

Answer: $2$ (accepted range $\pm 0.01$ )

Forward: z1 = 0*2 + 0 = 0, so h = sigmoid(0) = 0.5; y = 4*0.5 + 0 = 2.
Output delta (linear output): delta2 = y - t = 2 - 1 = 1.
Hidden delta: delta1 = w2 delta2 h(1 - h) = 4 1 0.25 = 1.
dL/dw1 = delta1 x = 1 2 = 2.

PRACTICE 72 marks · MSQ

Which of the following statements about feed-forward neural networks are true?

(A)
A network of any depth in which every unit uses the identity activation computes an affine function of its input.
(B)
The derivative of the sigmoid function never exceeds 0.25.
(C)
The tanh activation takes values only in the interval (0, 1).
(D)
If all weights and biases of an MLP with sigmoid hidden units start at zero, the hidden units within a layer remain identical to each other after any number of gradient-descent steps.

Answer and solution

Answer: (A), (B), (D)

A: a composition of affine maps is affine (W2(W1 x + b1) + b2 = W'x + b'), so true.
B: sigma'(z) = sigma(z)(1 - sigma(z)) is a product p(1 - p) with p in (0, 1), maximised at p = 0.5, giving 0.25. True.
C: tanh takes values in (-1, 1), not (0, 1). False.
D: with equal starting weights, every hidden unit in a layer has the same input weights and the same outgoing weights, so they get the same outputs, the same deltas and the same updates; the symmetry is never broken. True.
So the answers are A, B and D.

OFFICIAL GATE 2027 SYLLABUS WORDING

Copied from the GATE 2027 syllabus; see all 2027 syllabi.

GATE DA

multi-layer perceptron, feed-forward neural network

Preparing in Kolkata? IAE Academy's course for GATE DA.