Focused Contrastive Training for Test-based Constituency Analysis

Focused Contrastive Training for Test-based Constituency Analysis

Abstract

We propose a scheme for self-training of grammaticality models for constituency analysis based on linguistic tests. A pre-trained language model is fine-tuned by contrastive estimation of grammatical sentences from a corpus, and ungrammatical sentences that were perturbed by a syntactic test, a transformation that is motivated by constituency theory. We show that consistent gains can be achieved if only certain positive instances are chosen for training, depending on whether they could be the result of a test transformation. This way, the positives, and negatives exhibit similar characteristics, which makes the objective more challenging for the language model, and also allows for additional markup that indicates the position of the test application within the sentence.

Grafik Top
Authors
  • Roth, Benjamin
  • Çano, Erion
Grafik Top
Shortfacts
Category
Paper in Conference Proceedings or in Workshop Proceedings (Poster)
Event Title
NeurIPS 2021 Workshop: Self-Supervised Learning - Theory and Practice
Divisions
Data Mining and Machine Learning
Subjects
Kuenstliche Intelligenz
Sprachverarbeitung
Event Location
Sydney, Australia
Event Type
Workshop
Event Dates
Dec 14, 2021
Date
14 December 2021
Export
Grafik Top