Context:Continuous experimentation guides development activities based on data collected on a subset of online userson a new experimental version of the software. It includes practices such as canary releases, gradual rollouts, darklaunches, or A/B testing.Objective:Unfortunately, our knowledge of continuous experimentation is currently primarily based on well-known andoutspoken industrial leaders. To assess the actual state of practice in continuous experimentation, we conducted amixed-method empirical study.Method:In our empirical study consisting of four steps, we interviewed 31 developers or release engineers, and performeda survey that attracted 187 complete responses. We analyzed the resulting data using statistical analysis and open cod-ing.Results:Our results lead to several conclusions: (1) from a software architecture perspective, continuous experimen-tation is especially enabled by architectures that foster independently deployable services, such as microservices-basedarchitectures; (2) from a developer perspective, experiments require extensive monitoring and analytics to discover run-time problems, consequently leading to developer on call policies and influencing the role and skill sets required bydevelopers; and (3) from a process perspective, many organizations conduct experiments based on intuition rather thanclear guidelines and robust statistics.Conclusion:Our findings show that more principled and structured approaches for release decision making are needed,striving for highly automated, systematic, and data- and hypothesis-driven deployment and experimentation.

Top