-
FuLG: 150B Romanian Corpus for Language Model Pretraining
Authors:
Vlad-Andrei Bădoiu,
Mihai-Valentin Dumitru,
Alexandru M. Gherghescu,
Alexandru Agache,
Costin Raiciu
Abstract:
Research in the field of language models is rapidly evolving, with many open models being released to the public. Openly available pretraining corpora usually focus on only a handful of languages, with many others either missing completely or extremely underrepresented. In this report, we introduce FuLG, a hundred-fifty-billion-token Romanian corpus extracted from CommonCrawl. We present our metho…
▽ More
Research in the field of language models is rapidly evolving, with many open models being released to the public. Openly available pretraining corpora usually focus on only a handful of languages, with many others either missing completely or extremely underrepresented. In this report, we introduce FuLG, a hundred-fifty-billion-token Romanian corpus extracted from CommonCrawl. We present our methodology for filtering FuLG and compare it via ablation studies against existing Romanian corpora.
△ Less
Submitted 18 July, 2024;
originally announced July 2024.
-
A Look Into Training Large Language Models on Next Generation Datacenters
Authors:
Alexandru M. Gherghescu,
Vlad-Andrei Bădoiu,
Alexandru Agache,
Mihai-Valentin Dumitru,
Iuliu Vasilescu,
Radu Mantu,
Costin Raiciu
Abstract:
Is it still worth doing computer networking research? What are relevant problems in this space given the supremacy of hyperscalers in deployed large networks? We take an unconventional approach to finding relevant research directions, by starting from Microsoft's plans to build a $100 billion datacenter for ML. Our goal is to understand what models could be trained in such a datacenter, as well as…
▽ More
Is it still worth doing computer networking research? What are relevant problems in this space given the supremacy of hyperscalers in deployed large networks? We take an unconventional approach to finding relevant research directions, by starting from Microsoft's plans to build a $100 billion datacenter for ML. Our goal is to understand what models could be trained in such a datacenter, as well as the high-level challenges one may encounter in doing so.
We first examine the constraints imposed by cooling and power requirements for our target datacenter and find that it is infeasible to build in a single location. We use LLM scaling laws to determine that we could train models of 50T or 100T. Finally, we examine how distributed training might work for these models, and what the networking requirements are. We conclude that building the datacenter and training such models is technically possible, but this requires a novel NIC-based multipath transport along with a redesign of the entire training stack, outlining a research agenda for our community in the near future.
△ Less
Submitted 1 July, 2024;
originally announced July 2024.
-
Prose-to-P4: Leveraging High Level Languages
Authors:
Mihai-Valentin Dumitru,
Vlad-Andrei Bădoiu,
Costin Raiciu
Abstract:
Languages such as P4 and NPL have enabled a wide and diverse range of networking applications that take advantage of programmable dataplanes. However, software development in these languages is difficult. To address this issue, high-level languages have been designed to offer programmers powerful abstractions that reduce the time, effort and domain-knowledge required for developing networking appl…
▽ More
Languages such as P4 and NPL have enabled a wide and diverse range of networking applications that take advantage of programmable dataplanes. However, software development in these languages is difficult. To address this issue, high-level languages have been designed to offer programmers powerful abstractions that reduce the time, effort and domain-knowledge required for developing networking applications. These languages are then translated by a compiler into P4/NPL code. Inspired by the recent success of Large Language Models (LLMs) in the task of code generation, we propose to raise the level of abstraction even higher, employing LLMs to translate prose into high-level networking code. We analyze the problem, focusing on the motivation and opportunities, as well as the challenges involved and sketch out a roadmap for the development of a system that can generate high-level dataplane code from natural language instructions. We present some promising preliminary results on generating Lucid code from natural language.
△ Less
Submitted 19 June, 2024;
originally announced June 2024.
-
A Data-Driven Gaussian Process Filter for Electrocardiogram Denoising
Authors:
Mircea Dumitru,
Qiao Li,
Erick Andres Perez Alday,
Ali Bahrami Rad,
Gari D. Clifford,
Reza Sameni
Abstract:
Objective: Gaussian Processes (GP)-based filters, which have been effectively used for various applications including electrocardiogram (ECG) filtering can be computationally demanding and the choice of their hyperparameters is typically ad hoc. Methods: We develop a data-driven GP filter to address both issues, using the notion of the ECG phase domain -- a time-warped representation of the ECG be…
▽ More
Objective: Gaussian Processes (GP)-based filters, which have been effectively used for various applications including electrocardiogram (ECG) filtering can be computationally demanding and the choice of their hyperparameters is typically ad hoc. Methods: We develop a data-driven GP filter to address both issues, using the notion of the ECG phase domain -- a time-warped representation of the ECG beats onto a fixed number of samples and aligned R-peaks, which is assumed to follow a Gaussian distribution. Under this assumption, the computation of the sample mean and covariance matrix is simplified, enabling an efficient implementation of the GP filter in a data-driven manner, with no ad hoc hyperparameters. The proposed filter is evaluated and compared with a state-of-the-art wavelet-based filter, on the PhysioNet QT Database. The performance is evaluated by measuring the signal-to-noise ratio (SNR) improvement of the filter at SNR levels ranging from -5 to 30dB, in 5dB steps, using additive noise. For a clinical evaluation, the error between the estimated QT-intervals of the original and filtered signals is measured and compared with the benchmark filter. Results: It is shown that the proposed GP filter outperforms the benchmark filter for all the tested noise levels. It also outperforms the state-of-the-art filter in terms of QT-interval estimation error bias and variance. Conclusion: The proposed GP filter is a versatile technique for preprocessing the ECG in clinical and research applications, is applicable to ECG of arbitrary lengths and sampling frequencies, and provides confidence intervals for its performance.
△ Less
Submitted 9 January, 2024; v1 submitted 6 January, 2023;
originally announced January 2023.
-
A Trust Framework for Government Use of Artificial Intelligence and Automated Decision Making
Authors:
Pia Andrews,
Tim de Sousa,
Bruce Haefele,
Matt Beard,
Marcus Wigan,
Abhinav Palia,
Kathy Reid,
Saket Narayan,
Morgan Dumitru,
Alex Morrison,
Geoff Mason,
Aurelie Jacquet
Abstract:
This paper identifies the current challenges of the mechanisation, digitisation and automation of public sector systems and processes, and proposes a modern and practical framework to ensure and assure ethical and high veracity Artificial Intelligence (AI) or Automated Decision Making (ADM) systems in public institutions. This framework is designed for the specific context of the public sector, in…
▽ More
This paper identifies the current challenges of the mechanisation, digitisation and automation of public sector systems and processes, and proposes a modern and practical framework to ensure and assure ethical and high veracity Artificial Intelligence (AI) or Automated Decision Making (ADM) systems in public institutions. This framework is designed for the specific context of the public sector, in the jurisdictional and constitutional context of Australia, but is extendable to other jurisdictions and private sectors. The goals of the framework are to: 1) earn public trust and grow public confidence in government systems; 2) to ensure the unique responsibilities and accountabilities (including to the public) of public institutions under Administrative Law are met effectively; and 3) to assure a positive human, societal and ethical impact from the adoption of such systems. The framework could be extended to assure positive environmental or other impacts, but this paper focuses on human/societal outcomes and public trust. This paper is meant to complement principles-based frameworks like Australia's Artificial Intelligence Ethics Framework and the EU Assessment List for Trustworthy AI. In many countries, COVID created a bubble of improved trust, a bubble which has arguably already popped, and in an era of unprecedented mistrust of public institutions (but even in times of high trust) it is not enough that a service is faster, or more cost-effective. This paper proposes recommendations for government systems (technology platforms, operations, culture, governance, engagement, etc.) that would help to improve public confidence and trust in public institutions, policies and services, whilst meeting the special obligations and responsibilities of the public sector.
△ Less
Submitted 22 August, 2022;
originally announced August 2022.
