## Framework

GAN is composed of two models:

• discriminative model: learns to determine whether a sample is from the model distribution or the data distribution
• generative model: generate samples which look like from data distribution

$p_g(x)$: generator's distribution
$p_z(z)$: prior distribution on input noize variable $z$
$G(z; \theta_g)$: differentiable function represented by MLP with parameter $\theta_g$
$D(x; \theta_d)$: differentiable function represented by MLP with parameter $\theta_d$ that outputs a single scalar

The goal of the training is to maximize the probability to assign correct label for $D$, and to minimize the probability to assign correct label for $G$. Therefore, the objection is

$\min_G \max_D V(D, G) = \mathbb{E}_{x\sim p_{data}(x)}[\log D(x)] + \mathbb{E}_{z\sim p_z(z)}[\log(1-D(G(z)))]$

In practice, it is better to train $G$ to maximize $\log D(G(z))$, because $\log(1-D(G(z)))$ easily saturates in early stage of training.

## Theoretical Analysis

### Proposition1

When $G$ is fixed, the optimial discriminator $D$ is
$D^{*}_{G}(x) = \frac{p_{data}(x)}{p_{data}(x)+p_g(x)}$

\begin{aligned} V(G, D) &= \int_x p_{data}(x) \log(D(x))dx + \int_z p_z(z) \log(1-D(G(z)))dz \\ \\ &= \int_x p_{data}(x) \log(D(x))dx + p_g(x) \log(1-D(x))dx \end{aligned}

The maximum of
$f(x) = a\log(x)+b\log(1-x)$
is attained when $x = \frac{a}{a+b}$ when the domain is limited to $[0, 1]$, because
$f'(x) = \frac{a}{x}-\frac{b}{1-x}$
and $f'(x)=0$ when $x = \frac{a}{a+b}$.

### Theorem 1

The global optimum of $C(G) = \max_D V(G, D)$ is achieved if and only if $p_g = p_{data}$

\begin{aligned} C(G) &= \max_D V(G, D) \\ &= \mathbb{E}_{x\sim p_{data}(x)}[\log D^*(x)] + \mathbb{E}_{z\sim p_z(z)}[\log(1-D^*(G(z)))] \\ \\ &= \mathbb{E}_{x\sim p_{data}(x)}\left[\log \frac{p_{data}(x)}{p_{data}(x)+p_g(x)}\right] + \mathbb{E}_{x\sim p_g}\left[\log \frac{p_{data}(x)}{p_{g}(x)+p_g(x)}\right] \\ \\ &= \mathbb{E}_{x\sim p_{data}(x)}\left[-\log 2 + \log \frac{p_{data}(x)}{(p_{data}(x)+p_g(x))/2}\right] + \mathbb{E}_{x\sim p_g}\left[-\log 2 + \log \frac{p_{data}(x)}{(p_{g}(x)+p_g(x))/2}\right] \\ \\ &= -\log 4 + \text{KL}\left(p_{data} \mid\mid \frac{p_{data}+p_g}{2}\right) + \text{KL}\left(p_g \mid\mid \frac{p_{data}+p_g}{2}\right) \\ \\ &= -\log 4 + 2 \cdot \text{JSD} (p_{data}\mid \mid p_g) \end{aligned}

This concludes the proof.

### Proposition 2

If $G$ and $D$ have enough capacity, and each step of Algorithm 1, the discriminator is allowed to reach its optimum given $G$, and $p_g$ is updated so as to improve the criterion $V(G, D)$, then $p_g$ converges to $p_{data}$

Since global minima can be attained by gradient descent and its global optima is attaied when $p_g = p_{data}$, the conclusion follows.