DETECT-2B’s output is a granular frame-by-frame analysis of the audio stream with predictions being made for each frame of audio to determine whether it is a spoof.

Mamba-SSM, or State Space Models, is an emerging architecture designed to enhance the capabilities of deepfake detection models by improving their sequence modeling capabilities. State Space Models aim to perform sequence modeling tasks like transformers but with greater efficiency, although more research is needed to determine if they can outperform transformers in terms of detection accuracy. Mamba-SSM is characterized by its unique approach to modeling temporal sequences and capturing intricate patterns within audio data.

At its core, Mamba-SSM leverages stochastic processes to model the state transitions within audio sequences. Traditional classifiers might look at static snapshots of data or rely on recurrent structures that process sequences in a somewhat deterministic manner. In contrast, Mamba-SSM introduces an element of stochasticity, allowing it to probabilistically transition between states based on observed data patterns. This stochastic approach provides several advantages:

1. Enhanced Temporal Dynamics: By treating audio sequences as stochastic processes, Mamba-SSM can better capture the temporal dynamics of audio signals. This is crucial in deepfake detection, where subtle temporal inconsistencies often differentiate real from fake audio.
2. Adaptive State Transitions: The stochastic nature of Mamba-SSM allows it to adaptively transition between states based on the observed audio features. This adaptability enables the classifier to more accurately track the evolution of audio signals over time, improving its ability to detect anomalies.
3. Robustness to Variability: Mamba-SSM’s probabilistic framework makes it inherently robust to variations and noise in the audio data. This robustness is particularly beneficial in real-world scenarios where audio quality can vary significantly.

The introduction of Mamba-SSM represents a significant advancement in the field of deepfake detection for several reasons:

1. Superior Detection of Subtle Artifacts

Deepfake audio often contains artifacts that are too subtle for traditional classifiers to detect reliably. These artifacts might manifest as slight variations in pitch, timing, or spectral properties that are not easily noticeable. Mamba-SSM’s stochastic state transitions allow it to capture these subtle differences with greater accuracy. By modeling the probabilistic relationships between audio frames, Mamba-SSM can identify inconsistencies that would otherwise go undetected.

2. Improved Generalization Across Languages and Accents

The ability of DETECT-2B to perform well across diverse languages is largely attributed to our extensive multi-lingual training data and the use of pre-trained models like Wav2Vec2. These components enable the system to learn language-agnostic features that are indicative of audio manipulation. While Mamba-SSM’s probabilistic modeling of temporal dynamics may contribute to its robustness, the primary factor in its cross-lingual performance is likely the diversity of the training data.

3. Integration with Self-Supervised Learning

Mamba-SSM is designed to integrate seamlessly with self-supervised pre-trained models like Wav2Vec2. Self-supervised learning models have already demonstrated exceptional performance in various audio tasks by learning rich representations from large amounts of unlabeled data. When combined with Mamba-SSM, these models gain an additional layer of refinement, focusing more precisely on the artifacts that indicate deepfake audio. This integration results in a synergistic effect, enhancing the overall performance of the deepfake detection system.

4. Scalability and Efficiency

Despite its advanced capabilities, Mamba-SSM is designed to be computationally efficient. Its probabilistic framework can be scaled to handle large volumes of audio data with improved speed compared to alternative approaches. While more research is needed to fully characterize the trade-offs between speed and accuracy, the efficiency of the underlying models enables us to create powerful ensembles that deliver strong performance. This efficiency is essential for deploying deepfake detection systems in real-time applications, where quick decisions are critical.

Of course, a model is only as good as the data it is trained and evaluated on. For DETECT-2B, we have curated an extensive and diverse dataset that includes a substantial amount of real and fake audio data generated using a variety of methods. The dataset covers a wide range of speakers across multiple languages to ensure robustness and generalization.

Crucially, we make sure to maintain a strict separation between the speakers used in the training and evaluation sets, to ensure that the model is truly learning to detect the artifacts of fake audio rather than overfitting to specific voices.

For model evaluation, we have put together a very large test set containing unseen speakers, deepfake generation methods, and languages. This includes data sourced from various academic datasets as well as internal data collected from diverse real-world sources. By testing on such a comprehensive dataset, we can be confident that DETECT-2B’s performance metrics reflect its ability to generalize to the types of deepfake audio it would encounter in the wild.

So how does DETECT-2B actually perform on this challenging test set? It achieves an impressive equal error rate (EER), which represents the point at which the false positive rate (real audio clips incorrectly classified as fake) and the false negative rate (fake audio clips incorrectly classified as real) are equal. DETECT-2B is able to correctly identify the vast majority of deepfake audio clips while maintaining a very low false positive rate. This is a substantial improvement over our original Detect model.

Looking at the aggregate results, DETECT-2B exhibits consistently high accuracy across a variety of languages, including those not seen during training. This indicates that the model is learning to pick up on language-agnostic cues of audio manipulation.

Similarly, when broken down by deepfake generation method, DETECT-2B performs well across the board, including on the latest synthetic audio approaches. Impressively, it achieves strong performance even on methods that were not represented in the training data, suggesting that the model is learning some fundamental features of synthetic audio rather than simply memorizing known fake audio patterns.

For customers looking to integrate DETECT-2B into their audio processing pipelines, we offer a simple yet flexible API. Audio clips can be submitted for analysis either individually or in batches. The API will first preprocess the audio to ensure consistent format and quality. It then analyzes the audio using the DETECT-2B ensemble model, employing various optimizations for maximum efficiency.

The raw fakeness scores for segments of the audio can be returned directly, or the API can aggregate them and apply the real/fake classification threshold to produce a single overall prediction for the clip. The threshold can be adjusted to trade off between false positives and false negatives depending on the use case.

Additionally, we offer a web-based dashboard interface for customers who prefer a more visual, user-friendly way to interact with DETECT-2B. The dashboard allows users to easily upload audio files, view analysis results, and adjust settings without needing to work with the API directly.

With the release of DETECT-2B, Resemble AI continues to push the boundaries of what is possible in deepfake detection. By leveraging state-of-the-art machine learning techniques and training on a vast, diverse dataset, DETECT-2B achieves unparalleled accuracy in identifying audio deepfakes.

But our work is far from over. As generative AI capabilities continue to advance, so must our detection capabilities. We have several exciting research directions planned to further improve DETECT-2B, focusing on areas such as representation learning, advanced model architectures, and data expansion.

We are committed to staying at the forefront of deepfake detection technology in order to help our customers navigate the increasingly complex landscape of AI-generated media. With tools like DETECT-2B in combination with our other innovative solutions, Resemble AI is providing the most comprehensive and effective suite for ensuring the integrity of audio content.

If you’re interested in learning more about DETECT-2B or integrating it into your own application, please reach out to us for further discussion. Together, we can work towards a future where AI is used responsibly and transparently for the benefit of all.