Gaussian Sketching yields a JL Lemma in RKHS
[edit]
Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, PMLR 108:39283937, 2020.
Abstract
The main contribution of the paper is to show that Gaussian sketching of a kernelGram matrix $\bm K$ yields an operator whose counterpart in an RKHS $\cal H$, is a \emph{random projection} operator—in the spirit of JohnsonLindenstrauss (JL) lemma. To be precise, given a random matrix $Z$ with i.i.d. Gaussian entries, we show that a sketch $Z\bm{K}$ corresponds to a particular random operator in (infinitedimensional) Hilbert space $\cal H$ that maps functions $f \in \cal H$ to a lowdimensional space $\bb R^d$, while preserving a weighted RKHS innerproduct of the form $⟨f, g \rangle_{\Sigma} \doteq ⟨f, \Sigma^3 g \rangle_{\cal H}$, where $\Sigma$ is the \emph{covariance} operator induced by the data distribution. In particular, under similar assumptions as in kernel PCA (KPCA), or kernel $k$means (K$k$means), wellseparated subsets of featurespace $\{K(\cdot, x): x \in \cal X\}$ remain wellseparated after such operation, which suggests similar benefits as in KPCA and/or K$k$means, albeit at the much cheaper cost of a random projection. In particular, our convergence rates suggest that, given a large dataset $\{X_i\}_{i=1}^N$ of size $N$, we can build the Gram matrix $\bm K$ on a much smaller subsample of size $n\ll N$, so that the sketch $Z\bm K$ is very cheap to obtain and subsequently apply as a projection operator on the original data $\{X_i\}_{i=1}^N$. We verify these insights empirically on synthetic data, and on realworld clustering applications.
Related Material


