Fine-tuning Transformer-based Natural Language Generation Algorithms for USDA Grains Reports for Farmers, Producers, and Small Businesses
Presentation Type
Poster
Student
Yes
Track
Other
Abstract
The Transformer architecture for Natural Language Generation (NLG) was nothing short of a revolution for natural language processing. Self- and multi-head attention models have proven their efficacy in a variety of textual tasks, including classification, translation, summarization, and generation. Fine-tuning Transformer-based models on summarization tasks has proven successful with small textual datasets. In this research, we focus on fine-tuning pre-trained transformer-based NLG algorithms for USDA reports on grains to produce high-quality summaries, highlights, narratives, and Q&As, to enable farmers, producers, and small businesses in making informed decisions on production, investments, expansions, and risk management. Based on high-performing transformer-based algorithms pre-trained on large textual datasets such as collections of news articles, we develop, train, and fine-tune NLG algorithms with implementation of the Transformer model architecture on publicly available USDA grain publications and reports with characteristics that require specific considerations in model building and training, such as topic-specific (e.g, corn or soybeans), numerical data-intensive (e.g, supply and demand), and temporally sequential (e.g, weekly progress updates). A multi-stage end-to-end NLG system will be developed in which learning evolves from unsupervised, semi-supervised, to supervised, and the final stage of supervised learning would utilize human-expert-produced summaries and highlights. While the research contributes to the field of applying transformer-based models into specific domains, the practical goal is to automate the processes to efficiently inform intended audiences with high-quality content.
Start Date
2-8-2022 1:00 PM
End Date
2-8-2022 2:00 PM
Fine-tuning Transformer-based Natural Language Generation Algorithms for USDA Grains Reports for Farmers, Producers, and Small Businesses
The Transformer architecture for Natural Language Generation (NLG) was nothing short of a revolution for natural language processing. Self- and multi-head attention models have proven their efficacy in a variety of textual tasks, including classification, translation, summarization, and generation. Fine-tuning Transformer-based models on summarization tasks has proven successful with small textual datasets. In this research, we focus on fine-tuning pre-trained transformer-based NLG algorithms for USDA reports on grains to produce high-quality summaries, highlights, narratives, and Q&As, to enable farmers, producers, and small businesses in making informed decisions on production, investments, expansions, and risk management. Based on high-performing transformer-based algorithms pre-trained on large textual datasets such as collections of news articles, we develop, train, and fine-tune NLG algorithms with implementation of the Transformer model architecture on publicly available USDA grain publications and reports with characteristics that require specific considerations in model building and training, such as topic-specific (e.g, corn or soybeans), numerical data-intensive (e.g, supply and demand), and temporally sequential (e.g, weekly progress updates). A multi-stage end-to-end NLG system will be developed in which learning evolves from unsupervised, semi-supervised, to supervised, and the final stage of supervised learning would utilize human-expert-produced summaries and highlights. While the research contributes to the field of applying transformer-based models into specific domains, the practical goal is to automate the processes to efficiently inform intended audiences with high-quality content.