BoltzmannMachines.jl

This Julia package implements algorithms for training and evaluating several types of Boltzmann Machines (BMs):

• Learning of Restricted Boltzmann Machines (RBMs) using Contrastive Divergence (CD)
• Greedy layerwise pre-training of Deep Boltzmann Machines (DBMs)
• Learning procedure for general Boltzmann Machines using mean-field inference and stochastic approximation. Applicable to DBMs and used for fine-tuning the weights after the pre-training
• Exact calculation of the likelihood of BMs (only suitable for small models)
• Annealed Importance Sampling (AIS) for estimating the likelihood of larger BMs

The package is contained in the official Julia package registry and can be installed via

using Pkg
Pkg.add("BoltzmannMachines")

The package contains the following types of RBMs (subtypes of AbstractRBM):

DBMs are implemented as vectors of RBMs. BasicDBMs have only Bernoulli distributed nodes and therefore consist of a vector of BernoulliRBMs. DBMs with different types of visible units can be constructed by using the corresponding RBM type in the first layer. Actual MultimodalDBMs can be formed by using PartitionedRBMs, which is a type of AbstractRBM that is able to encapsulate non-connected RBMs of different types into an RBM-like layer.

All these types of DBMs can be trained using layerwise pre-training and fine-tuning employing the mean-field approximation. It is also possible to estimate or calculate the likelihood for these DBM types.

The following tables provide an overview of the functions of the package, together with a short description. You can find more detailed descriptions for each function using the Julia help mode (entered by typing ? at the beginning of the Julia command prompt).

Continuously valued data or ordinal data can be transformed into probabilities via intensities_encode and then fed to BernoulliRBMs, like it is usually done when handling grayscale or color intensities in images.

Categorical data can be binary encoded as input for a Softmax0BernoulliRBM via oneornone_encode.

The back transformations are available via the functions intensities_decode and oneornone_decode.

To fit MultimodalDBMs, the arguments for training its (partitioned) layers can be specified using structs of type TrainLayer and TrainPartitionedLayer (best see the examples for how to use these arguments in fitdbm or stackrbms).

The functions joindbms and joinrbms can be used to join the weights of two separately trained models.

The functions of the form monitor*! can be used for monitoring a property of the model during the learning process. The following words, corresponding to the denominated properties, may stand in place of *:

• freeenergy
• exactloglikelihood
• loglikelihood
• logproblowerbound
• reconstructionerror

The results of evaluations are stored in Monitor objects. The evaluations can be plotted by calling the function plotevaluation of the external plotting package BoltzmannMachinesPlots.

The monitoring mechanism is very flexible and allows the specification of callback functions that can be passed to the training functions fitrbm, stackrbms, traindbm!, and fitdbm. Monitoring can be streamlined with the functions monitored_fitrbm, monitored_stackrbms, monitored_traindbm! and monitored_fitdbm. These functions also allow user-defined monitoring functions that conform to the same argument schema as the above mentioned predefined monitoring functions.

To see how these functions can be used together, best take a look at the examples.

You can find example code here.

If you want to use the plotting functionality, you need to install the package BoltzmannMachinesPlots in addition.

The package has been used for an approach to uncover patterns in high-dimensional genetic data, described in the article

Hess M., Lenz S., Blätte T. J., Bullinger L., Binder H. Partitioned learning of deep Boltzmann machines for SNP data. Bioinformatics 2017 btx408. doi: https://doi.org/10.1093/bioinformatics/btx408.

The code for the analyses presented there is available in the article supplement.

[1] Salakhutdinov, R. (2015). Learning Deep Generative Models. Annual Review of Statistics and Its Application, 2, 361-385.

[2] Salakhutdinov, R. Hinton, G. (2012). An Efficient Learning Procedure for Deep Boltzmann Machines. Neural computation, 24(8), 1967-2006.

[3] Salakhutdinov. R. (2008). Learning and Evaluating Boltzmann Machines. Technical Report UTML TR 2008-002, Department of Computer Science, University of Toronto.

[4] Krizhevsky, A., Hinton, G. (2009). Learning Multiple Layers of Features from Tiny Images.

[5] Srivastava, N., Salakhutdinov R. (2014). Multimodal Learning with Deep Boltzmann Machines. Journal of Machine Learning Research, 15, 2949-2980.

[6] Cho, K., Ilin A., Raiko, T. (2011) Improved learning of Gaussian-Bernoulli restricted Boltzmann machines. Artificial Neural Networks and Machine Learning – ICANN 2011.