When does a mixture of products contain a product of mixtures?

Guido F. Montúfar, Jason Morton

Research output: Contribution to journalArticle

10 Citations (Scopus)

Abstract

We derive relations between theoretical properties of restricted Boltzmann machines (RBMs), popular machine learning models which form the building blocks of deep learning models, and several natural notions from discrete mathematics and convex geometry. We give implications and equivalences relating RBM-representable probability distributions, perfectly reconstructible inputs, Hamming modes, zonotopes and zonosets, point configurations in hyperplane arrangements, linear threshold codes, and multicovering numbers of hypercubes. As a motivating application, we prove results on the relative representational power of mixtures of product distributions and products of mixtures of pairs of product distributions (RBMs) that formally justify widely held intuitions about distributed representations. In particular, we show that a mixture of products requiring an exponentially larger number of parameters is needed to represent the probability distributions which can be obtained as products of mixtures.

Original languageEnglish (US)
Pages (from-to)321-347
Number of pages27
JournalSIAM Journal on Discrete Mathematics
Volume29
Issue number1
DOIs
StatePublished - Jan 1 2015

Fingerprint

Boltzmann Machine
Product of Distributions
Probability Distribution
Convex Geometry
Hyperplane Arrangement
Discrete mathematics
Hypercube
Justify
Building Blocks
Machine Learning
Equivalence
Configuration
Model

All Science Journal Classification (ASJC) codes

  • Mathematics(all)

Cite this

@article{89b2853ed9db42e4a63a36b9c11f62c4,
title = "When does a mixture of products contain a product of mixtures?",
abstract = "We derive relations between theoretical properties of restricted Boltzmann machines (RBMs), popular machine learning models which form the building blocks of deep learning models, and several natural notions from discrete mathematics and convex geometry. We give implications and equivalences relating RBM-representable probability distributions, perfectly reconstructible inputs, Hamming modes, zonotopes and zonosets, point configurations in hyperplane arrangements, linear threshold codes, and multicovering numbers of hypercubes. As a motivating application, we prove results on the relative representational power of mixtures of product distributions and products of mixtures of pairs of product distributions (RBMs) that formally justify widely held intuitions about distributed representations. In particular, we show that a mixture of products requiring an exponentially larger number of parameters is needed to represent the probability distributions which can be obtained as products of mixtures.",
author = "Mont{\'u}far, {Guido F.} and Jason Morton",
year = "2015",
month = "1",
day = "1",
doi = "10.1137/140957081",
language = "English (US)",
volume = "29",
pages = "321--347",
journal = "SIAM Journal on Discrete Mathematics",
issn = "0895-4801",
publisher = "Society for Industrial and Applied Mathematics Publications",
number = "1",

}

When does a mixture of products contain a product of mixtures? / Montúfar, Guido F.; Morton, Jason.

In: SIAM Journal on Discrete Mathematics, Vol. 29, No. 1, 01.01.2015, p. 321-347.

Research output: Contribution to journalArticle

TY - JOUR

T1 - When does a mixture of products contain a product of mixtures?

AU - Montúfar, Guido F.

AU - Morton, Jason

PY - 2015/1/1

Y1 - 2015/1/1

N2 - We derive relations between theoretical properties of restricted Boltzmann machines (RBMs), popular machine learning models which form the building blocks of deep learning models, and several natural notions from discrete mathematics and convex geometry. We give implications and equivalences relating RBM-representable probability distributions, perfectly reconstructible inputs, Hamming modes, zonotopes and zonosets, point configurations in hyperplane arrangements, linear threshold codes, and multicovering numbers of hypercubes. As a motivating application, we prove results on the relative representational power of mixtures of product distributions and products of mixtures of pairs of product distributions (RBMs) that formally justify widely held intuitions about distributed representations. In particular, we show that a mixture of products requiring an exponentially larger number of parameters is needed to represent the probability distributions which can be obtained as products of mixtures.

AB - We derive relations between theoretical properties of restricted Boltzmann machines (RBMs), popular machine learning models which form the building blocks of deep learning models, and several natural notions from discrete mathematics and convex geometry. We give implications and equivalences relating RBM-representable probability distributions, perfectly reconstructible inputs, Hamming modes, zonotopes and zonosets, point configurations in hyperplane arrangements, linear threshold codes, and multicovering numbers of hypercubes. As a motivating application, we prove results on the relative representational power of mixtures of product distributions and products of mixtures of pairs of product distributions (RBMs) that formally justify widely held intuitions about distributed representations. In particular, we show that a mixture of products requiring an exponentially larger number of parameters is needed to represent the probability distributions which can be obtained as products of mixtures.

UR - http://www.scopus.com/inward/record.url?scp=84925348229&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84925348229&partnerID=8YFLogxK

U2 - 10.1137/140957081

DO - 10.1137/140957081

M3 - Article

AN - SCOPUS:84925348229

VL - 29

SP - 321

EP - 347

JO - SIAM Journal on Discrete Mathematics

JF - SIAM Journal on Discrete Mathematics

SN - 0895-4801

IS - 1

ER -