[0]: – Goal-Oriented Testing for DevOps: How to Deliver Quality Changes Faster
[1]: – April 21, 2017
[2]: – Ben Halpern — Looking to scale your testing efforts? Consider goal-oriented testing. Learn how to build reliable test suites that allow you to deliver code faster without sacrificing quality.
[3]: To build quality software quickly, you need to automate testing, run tests continuously, and increasingly turn testing over to developers. We have a number of testing tools and techniques for doing that: unit testing frameworks, BDD tools, static code analysis, contract testing, integration testing tools, mutation testing, property-based testing, and more.
[4]: But even with all this automation, tooling, and expertise, teams often struggle with some common problems:
[5]: – Further automation is hard. Teams often find that adding new automated test cases is difficult.
[6]: – Existing tests change too much. They frequently need significant amounts of refactoring every time the software changes.
[7]: – Test reliability is low. Even when tests find bugs, they can break frequently, either failing when there’s no bug or passing when there is.
[8]: – Test suites are slow. Runners or tools may not allow tests to be executed in parallel. Many of the tests may not be exercising the system in real-world scenarios.
[9]: – Test suites are slow to build. Tests may fail intermittently when executing even small changes.
[10]: – Test suites are flaky. The results of tests are inconsistent across runs, depending on timing or other factors.
[11]: – Test suites are large. There are often far more tests than necessary to achieve quality goals.
[12]: – Big problems produce no feedback. Tests often capture too little information about where a bug occurred or what sort of bug it is.
[13]: Automated testing is a prerequisite for continuous integration and continuous delivery (CI/CD), for monitoring production systems, and for any sort of automation in the build process — but we don’t see prevalence of these patterns correlate directly to quality or deployment frequency.
[14]: Instead, better test suites have more to do with goals and skills than tooling and process.
[15]: The Problem With Existing Approaches
[16]: There are several fundamental deficiencies all these approaches have in common:
[17]: It’s manual: Test cases for manual test scripts are created by people who aren’t developers (or the product owners) in a documentation format that requires larger teams. These tests are often errors in requirements specification, marketing material, or user guides.
[18]: It’s reactive: While a test suite may be executed as part of an ongoing release cycle, it is fundamentally operating in a “post-mortem” mode where bugs are found after they’ve already entered production and require manual analysis and debugging.
[19]: It’s brittle: Many teams write tests that fail whenever the software itself changes. This creates a long-term technical debt where tests have to be rewritten every time there is new functionality, leading to tests that are wasted effort and ignored.
[20]: It’s slow: Executing test suites is often very slow (days or weeks) because they need to exhaustively test every corner case or any random scenario that should be prevented.
[21]: This last category is shared even by many modern testing strategies: unit tests, contract testing, property-based testing, mutation testing, static code analysis, and more rely on defining test cases and input values that are arbitrary and frequently invalid.
[22]: The Goal-Oriented Testing Approach
[23]: The goal-oriented testing approach addresses these issues by adhering to four core principles:
[24]: – Test automation should be built around features: Features represent a carefully selected set of important requirements for how an application should behave for its users.
[25]: – Tests should only use valid input: Use valid input values with specific behavior coverage rather than arbitrary combinations of inputs, so features can be verified quickly.
[26]: – Test inputs should be generated from real-world user data: Use actual user data to explore valid behavior of your system.
[27]: – Test automation should exercise entire environments: From code through front-end interfaces to database storage protocols and production hardware such as cluster nodes, test entire systems in realistic usage scenarios.
[28]: Test Automation Should Be Built Around Features
[29]: At CAST AI, we believe that the best way to test features is to execute specific use cases that demonstrate their behavior. Each use case consists of specific steps that drive the feature in a predictable way, transitioning from one state to another.
[30]: Here’s an example:
[31]: A user wants to create a Kubernetes cluster and add a new node pool of instance types that they want to support:
[32]: – They navigate to the cluster homepage
[33]: – They click “Add Cluster”
[34]: – They input a name for their cluster
[35]: – They select US-West for their region
[36]: – They add a new node pool
[37]: – They select “m4.xlarge” for their instances
[38]: – They click “Create Cluster”
[39]: Once written down, use cases can be easily converted into automated testing scripts. We’ve chosen webdriverio as our test framework because it provides an easy-to-learn JavaScript API for driving web applications via Selenium.
[40]: Here’s that same example as a use case defined using webdriverio Expect-style assertions:
[41]: // examples/features/client.feature incudes the feature definition of this test example test: () => { return client // click on add cluster button .click(‘@addClusterButton’) // fill in cluster name with first clustername from fixtures .setValue(‘@clusterName’, clusternames[0]) // select region .waitForElementPresent(‘@regionSelect’) .selectByVisibleText(‘@regionSelect’, ‘US West (US-WEST1)’) // add nodepool .pause(2000) // scroll down to find Node Pool button .moveToElement(‘.agistani-nodepools .ant-collapse .ant-collapse-header’) .pause(2000) .click(‘@addPoolButton’) // select node pool type .pause(2000) // code adapted from supporting functions in StepDef steps.js .locateNodesPoolElByCode(‘m4.xlarge’) .click() .waitForExist(‘@poolSelectionbox’) .click(‘@poolSelectionbox’) // create node pool .click(‘@poolSubmitButton’) },
[42]: We’ve created a complete test runner in an accompanying repository (at asteps-gh-pages repo).
[43]: It is important to build the use cases as carefully as possible and follow the underlying requirements — too general tests may not prove enough to cover your features.
[44]: Tests Should Only Use Valid Input
[45]: However good your testing scripts may be, they should only be used with valid input data. It can take weeks or months to write a production-quality set of tests using arbitrary data values, and then many runs of “mystery shopper”-style users to validate these tests against real-world input.
[46]: Feature testing based on use cases avoids all these problems by directly validating all features with valid input: you’re writing test cases based on how the application is supposed to be used — there is no set of input values where any feature should not work.
[47]: Test Inputs Should Be Generated From Real-World User Data
[48]: Seed your test inputs from real-world user data — this will validate your application’s business logic requirements as well as its features, ensuring that your use cases exercise exactly what your users will do.
[49]: If your application itself doesn’t collect real-world data from your users (because it’s internal or because it stores no sensitive information), consider generating test inputs based on production data from similar applications — it’s likely that features are used in very similar ways alongside each other in each application. This data can be transformed using existing production fixtures or FaaSless functions located alongside each feature.
[50]: If you don’t have anything similar available to run against mock inputs derived from production data from public services or open datasets with the appropriate licensing for your use case.
[51]: In the CAST AI use case example above, we use fixture-based or open-source Docker registry image data to verify that we can pull images from Google container registries into our feature under test before attempting use cases that rely on them.
[52]: Test Automation Should Exercise Entire Environments
[53]: While it’s long been common practice to coordinate virtual services that interact with your application under test via mocks or fake resources, it’s more important than ever to test across private clouds or entire public cloud instances that you may use for hosting your application — especially in Kubernetes clusters or container-based environments!
[54]: There is often very little difference between running containers on a simple EC2 instance versus a full Kubernetes cluster — you cannot efficiently test Kubernetes clusters by running your packages locally, especially when testing clusters hosted within AWS or other cloud providers.
[55]: In addition to using true cloud environments for running your containers and serving up live web interfaces or APIs, run your applications in live cloud environments — it’s important that you validate that your infrastructure correctly provisions everything required by your application!
[56]: In our CAST AI Kubernetes-focused examples running in Google Cloud Platform, our applications can get access to their production container registries using credentials based on Google Cloud KMS through Google Secret Manager — any strong data protection solution will work if you bind it to your production infrastructure!
[57]: Committing To Improving Test Automation From Day One
[58]: We’ve found that by pursuing these four principles we’ve been able to dramatically improve our testing throughput and maintainability — we spent less time fixing test failures, had significantly lower breakage rates across our CI/CD pipeline, and have reduced our need for developer training on how to write tests.
[59]: While many tools and approaches can help you design and execute better test suites more quickly than before — try any amount of static code analysis or mutation testing can be useful — the most significant improvements will come from changing how you design your test suites rather than how you execute them.
[60]: That means assessing your business requirements and your development team’s workflow closely before starting any engineering effort. It also means being willing to invest in training developers how to design usable test suites rather than simply writing them — while tooling can eliminate many development hurdles for junior developers, you should ensure that developers can design effective test suites through practice exercises before attempting large-scale engineering projects 70+ Seconds In Pre-production: The First Trimester Of DevelopmentWarnings: Python 3.7 Drops Fixtures From Python Tests In Current Release Cybersource API Asks Webhooks To Access At Least 5% Of Total Requests To Trigger Threshold
[61]: Related Courses
[62]: Faster code with faster checks
[63]: How good enough is good enough? Carl Meyer discusses what it means to break the build – as soon as possible.
[64]: Advising API clients on environmental disaster recovery by using deltas between live environments
[65]: Learn how Storefront data storage engineers advise API clients on restoring eCommerce inventory databases using JSON Patch files.

expertise score: 4
strictly academic: Y
STEM related: Y
code relevance: Y
excerpts:
– ID: 1
type: guidelines
start line: 22
end line: 27
textbook relevance: 3
chain of reasoning:
description: Explanation of core principles underpinning goal-oriented testing
approach.
principles used:
– direct causality
level: 2
steps: Test automation around features -> Use valid inputs -> Use real-world user
data -> Exercise entire environments
num steps: 4
textbook relevance: 3
– ID: 2
type: guidelines
start line: 28
end line: 29
textbook relevance: 3
chain of reasoning:
description: Justification for using use cases in feature testing.
principles used:
– direct causality
level: 2
steps: Use cases represent important requirements -> Execute specific use cases
-> Best way to test features
num steps: 3
textbook relevance: 3
– ID: 3
type: factual
facual information:
content: Implementation of goal-oriented testing with webdriverio using specific
steps to create a Kubernetes cluster and add a new node pool.
level: 3
is internal: Y
start line: 31
end line: 38
textbook relevance: 3
– ID: 4
type: guidelines
chain of reasoning:
description: Explanation of why using real-world user data improves application
validation.
principles used:
– direct causality
level: 2
steps: Seed test inputs from real-world user data -> Validate business logic requirements
-> Use cases exercise exactly what users will do
num steps: 3
start line: 47
end line: 48
textbook relevance: 3

*** Excerpt data for ID: 1 ***

*** Conversation ***
## Suggestions for complexity

1. How could the principle of generating test inputs from real-world user data be affected by seasonality in usage patterns?
2. In what ways might valid input generation conflict with boundary testing practices, especially in systems designed for high fault tolerance?
3. Can you explain how exercising entire environments might reveal integration issues that isolated unit tests would miss? Provide an example scenario.
4. How does one balance between real-world user data and synthetic data when designing test automation systems in highly regulated industries?
5. Discuss the potential pitfalls in defining feature-centric test cases and how they can be mitigated.

## Conversation

How can seasonality affect the generation of test inputs from real-world user data?
Seasonality can significantly impact the generation of test inputs from real-world data because user behavior may vary dramatically at different times of the year. For example, an e-commerce website might see higher traffic during holiday seasons compared to other times of the year. If your test inputs are based solely on off-peak usage data, they might not capture potential performance bottlenecks or functional issues that arise under high load conditions typical during peak seasons.

To mitigate this issue, you might want to incorporate historical data from both peak and off-peak periods into your test input generation process. Additionally, you can implement load testing techniques that simulate peak season usage patterns based on past data trends.

Would you like me to delve into specific methodologies for integrating seasonal data into test automation frameworks?*** Excerpt data for ID: 2 ***

*** Conversation ***
## Suggestions for complexity

1. **Edge Cases and Failures**: How do use cases handle unexpected edge cases or failures during the execution stages? Are there built-in mechanisms within these use case frameworks to deal with exceptions?

2. **Performance Metrics**: How do you ensure performance metrics are maintained while executing specific use cases, especially under heavy load conditions?

3. **Comparative Effectiveness**: In what scenarios would traditional unit testing be more effective compared to use-case-based feature testing?

4. **Integration with CI/CD Pipelines**: How seamlessly can use-case-based feature testing integrate with existing CI/CD pipelines, and what challenges might arise during this integration?

5. **Advanced Feature Interactions**: In complex applications with multiple interacting features, how do use-case-based feature tests ensure comprehensive coverage? Are there strategies in place for prioritizing which interactions to test?

## Conversation

When discussing automated testing frameworks, how do you deal with edge cases and unexpected failures within specific use cases? Are there mechanisms built into these frameworks to handle exceptions?