Matrix Calculus

## Matrix Derivatives

### 1. Scalar-by-scalar

• input : $x \in \mathbb{R}$
• output : $f(x) \in \mathbb{R}$

### 2. Scalar-by-vector

• input : $x \in \mathbb{R}^m$
• output : $f \in \mathbb{R}$

Also a vector, which has the same size with input $x$.

#### Hessian

A matrix of the size $m \times m$. (m : dimension of $x$)

• useful to decide whether a optimization problem has a global optimum
• always symmetric

### 3. Vector-by-vector

• input : $x \in \mathbb{R}^m$
• output : $y(x) = Ax \in \mathbb{R}^n, \; A \in \mathbb{R}^{n \times m}$