Contextual HyperNetworks for Novel Feature Adaptation

Contextual HyperNetworks for Novel Feature Adaptation

Abstract

While deep learning has obtained state-of-the-art results in many applications, the adaptation of neural network architectures to incorporate new output features remains a challenge, as neural networks are commonly trained to produce a fixed output dimension. This issue is particularly severe in online learning settings, where new output features, such as items in a recommender system, are added continually with few or no associated observations. As such, methods for adapting neural networks to novel features which are both time and data-efficient are desired. To address this, we propose the Contextual HyperNetwork (CHN), an auxiliary model which generates parameters for extending the base model to a new feature, by utilizing both existing data as well as any observations and/or metadata associated with the new feature. At prediction time, the CHN requires only a single forward pass through a neural network, yielding a significant speed-up when compared to re-training and fine-tuning approaches. To assess the performance of CHNs, we use a CHN to augment a partial variational autoencoder (P-VAE), a deep generative model which can impute the values of missing features in sparsely-observed data. We show that this system obtains improved few-shot learning performance for novel features over existing imputation and meta-learning baselines across recommender systems, e-learning, and healthcare tasks.

Grafik Top
Authors
  • Lamb, Angus
  • Saveliev, Evgeny
  • Li, Yingzhen
  • Tschiatschek, Sebastian
  • Longden, Camilla
  • Woodhead, Simon
  • Hernández-Lobato, José Miguel
  • Turner, Richard E.
  • Cameron, Pashmina
  • Zhang, Cheng
Grafik Top
Shortfacts
Category
Journal Paper
Divisions
Data Mining and Machine Learning
Journal or Publication Title
CoRR arXiv
Volume
abs/21
Date
12 April 2021
Official URL
https://arxiv.org/abs/2104.05860
Export
Grafik Top