mlpack

mlpack is a free, open-source and header-only software library for machine learning and artificial intelligence written in C++, built on top of the Armadillo library and the ensmallen numerical optimization library. mlpack has an emphasis on scalability, speed, and ease-of-use. Its aim is to make machine learning possible for novice users by means of a simple, consistent API, while simultaneously exploiting C++ language features to provide maximum performance and maximum flexibility for expert users. mlpack has also a light deployment infrastructure with minimum dependencies, making it perfect for embedded systems and low resource devices. Its intended target users are scientists and engineers.

mlpack

Initial release	February 1, 2008 (2008-02-01)
Stable release	4.4.0 / 28 May 2024 (28 May 2024)
Repository	github.com/mlpack/mlpack
Written in	C++, Python, Julia, Go
Operating system	Cross-platform
Available in	English
Type	Software library Machine learning
License	Open source (BSD)
Website	mlpack.org

It is open-source software distributed under the BSD license, making it useful for developing both open source and proprietary software. Releases 1.0.11 and before were released under the LGPL license. The project is supported by the Georgia Institute of Technology and contributions from around the world.

Features

Classical machine learning algorithms

mlpack contains a wide range of algorithms that are used to solved real problems from classification and regression in the Supervised learning paradigm to clustering and dimension reduction algorithms. In the following, a non exhaustive list of algorithms and models that mlpack supports:

• Collaborative Filtering
• Decision stumps (one-level decision trees)
• Density Estimation Trees
• Euclidean minimum spanning trees
• Gaussian Mixture Models (GMMs)
• Hidden Markov Models (HMMs)
• Kernel density estimation (KDE)
• Kernel Principal Component Analysis (KPCA)
• K-Means Clustering
• Least-Angle Regression (LARS/LASSO)
• Linear Regression
• Bayesian Linear Regression
• Local Coordinate Coding
• Locality-Sensitive Hashing (LSH)
• Logistic regression
• Max-Kernel Search
• Naive Bayes Classifier
• Nearest neighbor search with dual-tree algorithms
• Neighbourhood Components Analysis (NCA)
• Non-negative Matrix Factorization (NMF)
• Principal Components Analysis (PCA)
• Independent component analysis (ICA)
• Rank-Approximate Nearest Neighbor (RANN)
• Simple Least-Squares Linear Regression (and Ridge Regression)
• Sparse Coding, Sparse dictionary learning
• Tree-based Neighbor Search (all-k-nearest-neighbors, all-k-furthest-neighbors), using either kd-trees or cover trees
• Tree-based Range Search

Class templates for GRU, LSTM structures are available, thus the library also supports Recurrent Neural Networks.

Bindings

There are bindings to R, Go, Julia, Python, and also to Command Line Interface (CLI) using terminal. Its binding system is extensible to other languages.

Reinforcement learning

mlpack contains several Reinforcement Learning (RL) algorithms implemented in C++ with a set of examples as well, these algorithms can be tuned per examples and combined with external simulators. Currently mlpack supports the following:

• Q-learning
• Deep Deterministic Policy Gradient
• Soft Actor-Critic
• Twin Delayed DDPG (TD3)

Design features

mlpack includes a range of design features that make it particularly well-suited for specialized applications, especially in the Edge AI and IoT domains. Its C++ codebase allows for seamless integration with sensors, facilitating direct data extraction and on-device preprocessing at the Edge. Below, we outline a specific set of design features that highlight mlpack's capabilities in these environments:

Low number of dependencies

mlpack is low dependencies library which makes it perfect for easy deployment of software. mlpack binaries can be linked statically and deployed to any system with minimal effort. The usage of Docker container is not necessary and even discouraged. This makes it suitable for low resource devices, as it requires only the ensmallen and Armadillo or Bandicoot depending on the type of hardware we are planning to deploy to. mlpack uses Cereal library for serialization of the models. Other dependencies are also header-only and part of the library itself.

Low binary footprint

In terms of binary size, mlpack methods have a significantly smaller footprint compared to other popular libraries. Below, we present a comparison of deployable binary sizes between mlpack, PyTorch, and scikit-learn. To ensure consistency, the same application, along with all its dependencies, was packaged within a single Docker container for this comparison.

Binary size comparison

	MNIST digit recognizer (CNN)	Language detection (Softmax regression)	Forest covertype classifier (decision tree)
scikit learn	N/A	327 MB	348 MB
Pytorch	1.04 GB	1.03 GB	N/A
mlpack	1.23 MB	1.03 MB	1.62 MB

Other libraries exist such as Tensorflow Lite, However, these libraries are usually specific for one method such as neural network inference or training.

Example

The following shows a simple example how to train a decision tree model using mlpack, and to use it for the classification. Of course you can ingest your own dataset using the Load function, but for now we are showing the API:

// Train a decision tree on random numeric data and predict labels on test data:

// All data and labels are uniform random; 10 dimensional data, 5 classes.
// Replace with a data::Load() call or similar for a real application.
arma::mat dataset(10, 1000, arma::fill::randu); // 1000 points.
arma::Row<size_t> labels =
    arma::randi<arma::Row<size_t>>(1000, arma::distr_param(0, 4));
arma::mat testDataset(10, 500, arma::fill::randu); // 500 test points.

mlpack::DecisionTree tree;               // Step 1: create model.
tree.Train(dataset, labels, 5);          // Step 2: train model.
arma::Row<size_t> predictions;
tree.Classify(testDataset, predictions); // Step 3: classify points.

// Print some information about the test predictions.
std::cout << arma::accu(predictions == 2) << " test points classified as class "
    << "2." << std::endl;

The above example demonstrate the simplicity behind the API design, which makes it similar to popular Python based machine learning kit (scikit-learn). Our objective is to simplify for the user the API and the main machine learning functions such as Classify and Predict. More complex examples are located in the examples repository, including documentations for the methods

Backend

Armadillo

Armadillo is the default linear algebra library that is used by mlpack, it provide matrix manipulation and operation necessary for machine learning algorithms. Armadillo is know for its efficiency and simplicity. it can also be used in header-only mode, and the only library we need to link against are either OpenBLAS, IntelMKL or LAPACK.

Bandicoot

Bandicoot is a C++ Linear Algebra library designed for scientific computing, it has the an identical API to Armadillo with objective to execute the computation on Graphics Processing Unit (GPU), the purpose of this library is to facilitate the transition between CPU and GPU by making a minor changes to the source code, (e.g. changing the namespace, and the linking library). mlpack currently supports partially Bandicoot with objective to provide neural network training on the GPU. The following examples shows two code blocks executing an identical operation. The first one is Armadillo code and it is running on the CPU, while the second one can runs on OpenCL supported GPU or NVIDIA GPU (with CUDA backend)

using namespace arma;
        
mat X, Y;
X.randu(10, 15);
Y.randu(10, 10);
mat Z = 2 * norm(Y) * (X * X.t() - Y);

using namespace coot;
        
mat X, Y;
X.randu(10, 15);
Y.randu(10, 10);
mat Z = 2 * norm(Y) * (X * X.t() - Y);

ensmallen

ensmallen is a high quality C++ library for non linear numerical optimizer, it uses Armadillo or bandicoot for linear algebra and it is used by mlpack to provide optimizer for training machine learning algorithms. Similar to mlpack, ensmallen is a header-only library and supports custom behavior using callbacks functions allowing the users to extend the functionalities for any optimizer. In addition ensmallen is published under the BSD license.

ensmallen contains a diverse range of optimizer classified based on the function type (differentiable, partially differentiable, categorical, constrained, etc). In the following we list a small set of optimizer that available in ensmallen. For the full list please check this documentation website.

• Limited memory Broyden–Fletcher–Goldfarb–Shanno (L-BFGS)
• GradientDescent
• FrankWolfe
• Covariance matrix adaptation evolution strategy (CMA-ES)
• AdaBelief
• AdaBound
• AdaDelta
• AdaGrad
• AdaSqrt
• Adam
• AdaMax
• AMSBound
• AMSGrad
• Big Batch SGD
• Eve
• FTML
• IQN
• Katyusha
• Lookahead
• Momentum SGD
• Nadam
• NadaMax
• NesterovMomentumSGD
• OptimisticAdam
• QHAdam
• QHSGD
• RMSProp
• SARAH/SARAH+
• Stochastic Gradient Descent SGD
• Stochastic Gradient Descent with Restarts (SGDR)
• Snapshot SGDR
• SMORMS3
• SPALeRA
• SWATS
• SVRG
• WNGrad

Support

mlpack is fiscally sponsored and supported by NumFOCUS, Consider making a tax-deductible donation to help the developers of the project. In addition mlpack team participates each year Google Summer of Code program and mentors several students.

See also

• Armadillo (C++ library)
• List of numerical analysis software
• List of numerical libraries
• Numerical linear algebra
• Scientific computing

External links

• Official website
• mlpack on GitHub