TechRxiv
A_Comprehensive_Exploration_of_Pre_training_Language_Models.pdf (133.92 kB)

A Comprehensive Exploration of Pre-training Language Models

Download (133.92 kB)
preprint
posted on 26.06.2021, 09:44 by Tong Guo
Recently, the development of pre-trained language models has brought natural language processing (NLP) tasks to the new state-of-the-art. In this paper we explore the efficiency of various pre-trained language models. We pre-train a list of transformer-based models with the same amount of text and the same training steps. The experimental results shows that the most improvement upon the origin BERT is adding the RNN-layer to capture more contextual information for the transformer-encoder layers.

History

Email Address of Submitting Author

779222056@qq.com

Submitting Author's Institution

_

Submitting Author's Country

China

Usage metrics

Licence

Exports