We address the problem of image collection summarization by learning mixtures ofsubmodular functions. Submodularity is useful for this problem since it naturallyrepresents characteristics such as fidelity and diversity, desirable for any summary.Several previously proposed image summarization scoring methodologies, in fact,instinctively arrived at submodularity. We provide classes of submodular compo-nent functions (including some which are instantiated via a deep neural network)over which mixtures may be learnt. We formulate the learning of such mixtures as asupervised problem via large-margin structured prediction. As a loss function, andfor automatic summary scoring, we introduce a novel summary evaluation methodcalled V-ROUGE, and test both submodular and non-submodular optimization(using the submodular-supermodular procedure) to learn a mixture of submodularfunctions. Interestingly, using non-submodular optimization to learn submodularfunctions provides the best results. We also provide a new data set consisting of14 real-world image collections along with many human-generated ground truthsummaries collected using Amazon Mechanical Turk. We compare our methodwith previous work on this problem and show that our learning approach outper-forms all competitors on this new data set. This paper provides, to our knowledge,the first systematic approach for quantifying the problem of image collection sum-marization, along with a new data set of image collections and human summaries.

Top