We introduce both neural higher-order linear-chain condi-tional random fields (NHO-LC-CRFs) and a new structured regularizerfor these sequence models. We show that this regularizer can be derivedas lower bound from a mixture of models sharing parts of each other, e.g.neural sub-networks, and relate it to ensemble learning. Furthermore, itcan be expressed explicitly as regularization term in the training objec-tive. We exemplify its effectiveness by exploring the introduced NHO-LC-CRFs for sequence labeling. Higher-order LC-CRFs with linear fac-tors are well-established for that task, but they lack the ability to modelnon-linear dependencies. These non-linear dependencies,however, can beefficiently modeled by neural higher-order input-dependentfactors. Onenovelty in this work is to map sub-sequences of inputs to sub-sequencesof outputs using distinct multilayer perceptron sub-networks. This map-ping is important in many tasks, in particular, for phoneme classifica-tion where the phoneme representation strongly depends on the contextphonemes. Experimental results for phoneme classification with NHO-LC-CRFs confirm this fact and we achieve state-of-the-art phoneme er-ror rate of 16.7% on TIMIT using the new structured regularizer. Thisis an absolute improvement of 1.3% over just usingl2regularization.

Top