Matrix Calculus
1 minute read

Matrix Derivatives

1. Scalar-by-scalar

Gradient is also a scalar.

2. Scalar-by-vector

Gradient

Also a vector, which has the same size with input $x$.

Hessian

A matrix of the size $m \times m$. (m : dimension of $x$)

3. Vector-by-vector

Gradient


Resources

Recent Posts

Lazy learning vs Eager learning
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
Perplexity of Language Models
Inverted Indexing
Deep Contextualized Word Representations