Where do I get started with data-driven engineering? How can the 3 I’s of data-driven engineering help me get off to a running start? How can I avoid the common pitfalls of data-driven engineering?
What are the 3 I’s?
The 3 I’s of data-driven engineering are insights, indicators and investments.
What are insights?
Insights are observations we derive from data generated by the software under test. They are the equivalent of observations we make about the product in the world of software testing. In software testing, we observe the software behavior, and we summarize these observations as feedback to the team. Insights are no different. We are still observing the software behavior. However, we are focusing on different ways of observing software behavior (e.g. logs, telemetry, CPU stats). In my mind, getting insights about our software is software testing.
What are indicators?
Indicators tell us that something could potentially be wrong. They indicate to us that an investigation and analysis should be done.
What are investments?
Investments are what we do based on our insights and indicators.
Why are the 3 I’s important? Why should I do this?
With the 3 I’s under your belt, you now have a framework to attack an engineering or testing problem driven by data, but why do this at all? Why not just put up a dashboard of indicators, call it a day and let the engineering team see the issues so that they fix them?
In an ideal world, all issues would be addressed as soon as they are surfaced. Hidden inside insights are 2 more I’s: investigations and interpretation. Even with the best indicators, data-driven engineers do investigations to interpret the indicators. They may dig into related data, try to correlate this with other indicators or trace through the source code. Based on the insights collected from investigating and interpreting the results, data-driven engineers push for investments or make the investments themselves.
Couldn’t all of this be automated?
Of the 3 I’s, indicators are well-suited for automation. Putting up a dashboard of indicators on a real-time basis is a job best done by a computer. However, the role of designing the indicator, deriving insights from investigations, and interpreting those insights into actionable investments is best suited for… you guessed it, a data-driven engineer.
Of all the 3 I’s (5 if you include investigations and interpretation), displaying indicators for the team can be easily automated, but other activities are uniquely human.
How can I get started?
Apply the 3 I’s!
Software testers are continuously questioning the product… Does it break when stress it this way? Does it allow me access to something I have no permission for? What if change the order of how I do certain steps?
Data-driven engineers are continuously questioning the product, too! What are the top server errors? What are the top client crashes? How many users do these errors impact? How many users have stopped using the product? How many new users are using it? Why are we seeing these trends?
Start with something you want to know about the product, and try to answer it with the data available. If it’s not available, then your investigation just yielded an insight! Take that insight and translate it into an actionable investment e.g. instrument the code so that we have the data to answer our question.
What are common mistakes?
You put up an indicator, because you have it available. Insights and indicators are tied together. The indicator must mean something to the business. For example, CPU usage is a nice well-defined indicator, but what does it mean to the user? Do we have data that says that increased CPU usage results in customers eventually giving up on the software? What’s the threshold for when this really becomes a problem?
You leave out the insights and investments. Your indicators, data and findings are required for data-driven engineering. However, your insights and investments are even more important. What opinion did you form based on all the indicators and data? What insights did you derive from your investigations? What should we invest in based on the interpretations? Link your findings to the business, and all will be right in the world.
This post originally appeared on rayli.net
Like this article? Subscribe to our weekly newsletter to never miss out!
