RKHS

evaluation operator

• (in the space where the inner product is defined) the evaluation operation is the inverse operation of the inner product, which recovers the intrinsic properties of an object from a distorted representation mapped to a basis direction of a specific space. this is performed by taking the inner product with the dual basis of the mapped basis.

  \[\delta(\cdot):=\langle\cdot,\psi\rangle\]

vector

• vector is a spatial object that has a direction and a magnitude in that direction, and has directionality as its essential property.

  \[\begin{aligned} \mathbf{v} &=\underbrace{\begin{bmatrix}\phi_{1}&\phi_{2}&\phi_{3}\end{bmatrix}}_{\text{basis}} \underbrace{\begin{bmatrix}\beta_{1}\\\beta_{2}\\\beta_{3}\end{bmatrix}}_{\text{coefficient}}\\ &= \beta_{1}\phi_{1}+\beta_{2}\phi_{2}+\beta_{3}\phi_{3} \end{aligned}\]

• vector space $(V,\langle\cdot,\cdot\rangle)$ can be spaned by basis:

  \[\begin{aligned} \mathbf{v} \in V &=\mathrm{span}\left\{\phi_{1},\phi_{2},\phi_{3}\right\} \end{aligned}\]

• vector evaluation is an operation based on the direction, that derives the magnitude in a given direction that is not dependent on the coordinate system.

  \[\delta_{i}[v]:=\langle v, \psi_{i} \rangle = \beta_{i}\]

function

• function is a mapping object that corresponds one value of the codomain $y$ to each point of the domain $X$, and has correspondence as its essential property.

  \[\begin{aligned} f(X) &=\underbrace{\begin{bmatrix}\phi_{1}(X)&\phi_{2}(X)&\phi_{3}(X)\end{bmatrix}}_{\text{basis}} \underbrace{\begin{bmatrix}\beta_{1}\\\beta_{2}\\\beta_{3}\end{bmatrix}}_{\text{coefficient}}\\ &= \beta_{1}\phi_{1}(X)+\beta_{2}\phi_{2}(X)+\beta_{3}\phi_{3}(X) \end{aligned}\]

• function space $(\mathcal{F},\langle\cdot,\cdot\rangle)$ can be spaned by basis:

  \[f(\cdot) \in \mathcal{F}=\mathrm{span}\left\{\phi_{1}(X),\phi_{2}(X),\phi_{3}(X)\right\}\]

• function evaluation is an operation based on the input value, that derives an output value for the corresponding input value that is not dependent on the expansion of the function by the basis.

  \[\delta_{X_{i}}[f(\cdot)]:=\langle f(\cdot), \psi(X_{i}) \rangle = f(X_{i})\]

dual space

• original space consisting of $N$ bases has a dual space consisting of $N$ dual basis.

  \[\begin{aligned} \mathcal{S} &=\mathrm{span}\left\{\phi_{1},\phi_{2},\cdots,\phi_{N}\right\}\\ \mathcal{S}^{*} &=\mathrm{span}\left\{\psi_{1},\psi_{2},\cdots,\psi_{N}\right\} \end{aligned}\]

• dual basis is defined via inverse of the Gram matrix of original basis.

  \[\begin{aligned} \psi_{j}:=\sum_{k=1}^{N}\left(G^{-1}\right)_{k,j}\phi_{k}, \quad G:=\left\{\langle\phi_{i},\phi_{j}\rangle\right\}_{(i,j)} \end{aligned}\]

• original basis and the dual basis react only to each other.

  \[\begin{aligned} \langle\phi_{i},\psi_{j}\rangle &= \left\langle\phi_{i},\sum_{k=1}^{N}{\left(G^{-1}\right)_{k,j}\phi_{k}}\right\rangle\\ &= \sum_{k=1}^{N}{\left(G^{-1}\right)_{k,j}\langle\phi_{i},\phi_{k}\rangle}\\ &= \sum_{k=1}^{N}{\left(G^{-1}\right)_{k,j}G_{i,k}}\\ &= \delta_{i,j} \end{aligned}\]

• dual basis is expressed as a linear combination of the original basis, which means that the original basis can span the dual space.

  \[\mathcal{F}^{*} \subseteq \mathrm{span}\left\{\phi_{1},\phi_{2},\cdots,\phi_{N}\right\} \quad \because \psi_{j}:=\sum_{k=1}^{N}{G_{k,j}\phi_{k}}\]

• If the Gram matrix is invertible, the original basis can also be expressed as a linear combination of the dual basis. This also means that the dual basis can span the original space.

  \[\mathcal{F} \subseteq \mathrm{span}\left\{\psi_{1},\psi_{2},\cdots,\psi_{N}\right\} \quad \because \phi_{j}:=\sum_{k=1}^{N}{G_{k,j}\psi_{k}}\]

• Since the original basis and the dual basis can be completely restored as linear combinations of each other, they span the same linear space.

  \[\mathcal{S} \subseteq \mathcal{S}^{*} \quad \mathrm{and} \quad \mathcal{S}^{*} \subseteq \mathcal{S}\\ \Downarrow\\ \mathcal{S}=\mathcal{S}^{*}\]

• therefore, the elements of the original space can be expressed as a dual basis.

  \[f=\sum_{i=1}^{N}{\beta_{i}\phi_{i}}=\sum_{i=1}^{N}{\gamma_{i}\psi_{i}}\]

RKHS

• parametric function is a function whose number and shape of basis are fixed (e.g. polynomial regression):

  \[\begin{aligned} f(X) &=\beta_{1}+\beta_{2}X+\beta_{3}X^{2}\\ &=\sum_{i=1}^{3}{\beta_{i}\phi_{i}(X)} \quad \mathrm{for} \quad \begin{cases}\phi_{1}(X)=1\\\phi_{2}(X)=X\\\phi_{3}(X)=X^{2}\end{cases} \end{aligned}\]

• parameteric function space $f \in \mathcal{F}$ is a finite-dimensional linear subspace consisting of a finite number of basis functions \(\{\phi_{i}\}_{i=1}^{N}\):

  \[\mathcal{F}=\mathrm{span}\left\{\phi_{1}(X),\phi_{2}(X),\cdots,\phi_{N}(X)\right\}\]

• nonparametric function is a function whose number and shape of basis are unfixed:

  \[f(X)=\sum_{i=1}^{\infty}{\beta_{i}\phi_{i}(X)}\]

• nonparametric function space $f \in \mathcal{F}$ is infinite-dimensional function space consisting of an infinite number of basis functions \(\{\phi_{i}\}_{i=1}^{\infty}\):

  \[\mathcal{F} =\mathrm{span}\left\{\phi_{1}(X),\phi_{2}(X),\cdots\right\}\]

• RKHS is a nonparametric function space that takes the dual basis as a reproducible kernel function:

  \[\psi(X):=k(\cdot,X)\]

• therefore, function @ RKHS can be expressed through reproducible kernel function:

  \[f(X)=\sum_{i=1}^{\infty}{\gamma_{i}k(X,X_{i})}\]