Structured Regularizer for Neural Higher-Order Sequence Models

Structured Regularizer for Neural Higher-Order Sequence Models

Abstract

We introduce both neural higher-order linear-chain condi-tional random fields (NHO-LC-CRFs) and a new structured regularizerfor these sequence models. We show that this regularizer can be derivedas lower bound from a mixture of models sharing parts of each other, e.g.neural sub-networks, and relate it to ensemble learning. Furthermore, itcan be expressed explicitly as regularization term in the training objec-tive. We exemplify its effectiveness by exploring the introduced NHO-LC-CRFs for sequence labeling. Higher-order LC-CRFs with linear fac-tors are well-established for that task, but they lack the ability to modelnon-linear dependencies. These non-linear dependencies,however, can beefficiently modeled by neural higher-order input-dependentfactors. Onenovelty in this work is to map sub-sequences of inputs to sub-sequencesof outputs using distinct multilayer perceptron sub-networks. This map-ping is important in many tasks, in particular, for phoneme classifica-tion where the phoneme representation strongly depends on the contextphonemes. Experimental results for phoneme classification with NHO-LC-CRFs confirm this fact and we achieve state-of-the-art phoneme er-ror rate of 16.7% on TIMIT using the new structured regularizer. Thisis an absolute improvement of 1.3% over just usingl2regularization.

Grafik Top
Authors
  • Ratajczak, Martin
  • Tschiatschek, Sebastian
  • Pernkopf, Franz
Grafik Top
Shortfacts
Category
Paper in Conference Proceedings or in Workshop Proceedings (Paper)
Event Title
European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD)
Divisions
Data Mining and Machine Learning
Event Location
Porto, Portugal
Event Type
Conference
Event Dates
07.-11.09.2015
Series Name
Machine Learning and Knowledge Discovery in Databases. ECML PKDD 2015. Lecture Notes in Computer Science
ISSN/ISBN
978-3-319-23527-1
Page Range
pp. 168-183
Date
7 September 2015
Official URL
https://www.tschiatschek.net/files/ratajczak15Stru...
Export
Grafik Top