list of efficient attention modules

Abstractive and Extractive Text summarization using Transformers.

Master thesis with code investigating methods for incorporating long-context reasoning in low-resource languages, without the need to pre-train from scratch. We investigated if multilingual models could inherit these properties by making it an Efficient Transformer (s.a. the Longformer architecture).

2020 AI研习社 金融用户评论分类

Longformer Encoder Decoder model for the legal domain, trained for long document abstractive summarization task.

using transformers to do text classification.

Convert pretrained RoBerta models to various long-document transformer models

[제 13회 투빅스 컨퍼런스] YoYAK - Yes or Yes, Attention with gap-sentence for Korean long sequence

This GitHub repository implements a novel approach for detecting Initial Public Offering (IPO) underpricing using pre-trained Transformers. The models, extended to handle large S-1 filings, leverage both textual information and financial indicators, outperforming traditional machine learning methods.

Industrial Text Scoring using Multimodal Deep Natural Language Processing 🚀 | Code for IEA AIE 2022 paper

Fine-tuned Longformer for Summarization of Machine Learning Articles

Kaggle NLP competition - Top 2% solution (36/2060)

This project applies the Longformer model to sentiment analysis using the IMDB movie review dataset. The Longformer model, introduced in "Longformer: The Long-Document Transformer," tackles long document processing with sliding-window and global attention mechanisms. The implementation leverages PyTorch, following the paper's architecture

A summarization website that can generate summaries from either YouTube videos or PDF files.

Project as part of COMP34812: Natural Language Understanding

A hyperpartisan news article classification system using BERT-based techniques. The goal was to leverage state-of-the-art transformer models like BERT, ROBERTa, and Longformer to accurately classify news articles as hyperpartisan or non-hyperpartisan.

This project was developed for a Kaggle competition focused on detecting Personally Identifiable Information (PII) in student writing. The primary objective was to build a robust model capable of identifying PII with high recall. The DeBERTa v3 transformer model was chosen for this task after comparing its performance with other transformer models.

Focus - Understanding contextual retrievability.

Longformer model trained on HotpotQA