We're doing it live: A multi-method empirical study on continuous experimentation

We're doing it live: A multi-method empirical study on continuous experimentation

Abstract

Context:Continuous experimentation guides development activities based on data collected on a subset of online userson a new experimental version of the software. It includes practices such as canary releases, gradual rollouts, darklaunches, or A/B testing.Objective:Unfortunately, our knowledge of continuous experimentation is currently primarily based on well-known andoutspoken industrial leaders. To assess the actual state of practice in continuous experimentation, we conducted amixed-method empirical study.Method:In our empirical study consisting of four steps, we interviewed 31 developers or release engineers, and performeda survey that attracted 187 complete responses. We analyzed the resulting data using statistical analysis and open cod-ing.Results:Our results lead to several conclusions: (1) from a software architecture perspective, continuous experimen-tation is especially enabled by architectures that foster independently deployable services, such as microservices-basedarchitectures; (2) from a developer perspective, experiments require extensive monitoring and analytics to discover run-time problems, consequently leading to developer on call policies and influencing the role and skill sets required bydevelopers; and (3) from a process perspective, many organizations conduct experiments based on intuition rather thanclear guidelines and robust statistics.Conclusion:Our findings show that more principled and structured approaches for release decision making are needed,striving for highly automated, systematic, and data- and hypothesis-driven deployment and experimentation.

Grafik Top
Authors
  • Schermann, Gerald
  • Cito, Jürgen
  • Leitner, Philipp
  • Zdun, Uwe
  • Gall, C. Harald
Grafik Top
Shortfacts
Category
Journal Paper
Divisions
Software Architecture
Subjects
Software Engineering
Journal or Publication Title
Information and Software Technology
ISSN
0950-5849
Page Range
pp. 41-57
Volume
99
Date
July 2018
Export
Grafik Top