SlideShare uma empresa Scribd logo
1 de 23
Baixar para ler offline
Case Study: How CA Went From 40 Days to Three Days
Building Crystal-Clear Test Cases While Improving Test
Coverage
Stephen Tyler
DO3T022S
VP, Software Engineering
CA Technologies
DEVOPS – CONTINUOUS DELIVERY
2 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
© 2017 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2017 presentation is intended for informational purposes only and does not form any type
of warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.
For Informational Purposes Only
Terms of This Presentation
3 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Abstract
Here at CA Technologies, our development teams share many of the same challenges
producing quality software as our customers. This session will cover how we:
Introduced collaborative modeling between our product owners and engineers to drive
extreme clarity around requirements during backlog refinement, before a story was
picked up for a sprint.
Built out a requirements model to generate a set of test cases that can fully cover the
acceptance criteria for your stories.
Gained the result—reducing the time it took to build out regression test cases from 40
days to three—while improving test coverage with less test cases
Stephen
Tyler
CA Technologies
VP, Software
Engineering
4 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Agenda
WHY: OUR CHALLENGES AND WHY WE THOUGHT MBT MIGHT HELP
WHAT: OUR HYPOTHESES AND METHODS
RESULTS: WHAT WORKED, LESSONS LEARNED, NEXT STEPS
1
2
3
5 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Challenges and Aspirations
 Incomplete/ambiguous requirements
 Inadequate functional coverage
 Too many defects leading to rework
ACCEPTANCE TESTING
FOR NEW STORIES
 Lack of visibility to functional
coverage/test efficiency
 Test Suite maintenance
 Automation cost
REGRESSION TESTING
Clear, complete,
unambiguous
requirements
What We Hoped MBT Would Get Us
Confidence in
functional
coverage
Fewer defects
escaping the
sprint
Increased
effectiveness
and efficiency
Ease of
Maintenance
Reduced
automation cost
6 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Acceptance Testing Example 1
Configure Connection and Resources Dialog
7 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
In the MBT tool we used, the functionality is
modeled as a flowchart/graph as shown
here.
Each node of the graph is an object on
the UI to which data is entered and
metadata associated to the node defines
what data ins and outs it supports.
Each path through the graph is a test
case driven by particular data entries or
decisions at various nodes along the
path.
In this model there are 23 distinct paths. That
is 23 test cases to fully cover the
functionality as modelled.
Acceptance Testing Example 1
The Model
8 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
All 23 possible paths
through the model
Path 15 of 23 visually highlighted on the graph and
textually represented (human readable test steps)
Model-generated Protractor
script to execute path 15
Generating Test Cases and Automated Tests
9 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Automating the Automation
Export the “Automation layer” for each path through the
model.
The script snippets for the relevant actions on each node
in the exported paths are concatenated with appropriate
variable substitution to produce an executable script
corresponding to that path in the model.
Attach Automation “actions” to the
outputs of each node in the model.
Each action is defined with
parameterized text snippets.
10 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Acceptance Testing Example 2
Service Configuration Sub-dialog
We will focus on the “client-facing certificate configuration”
requirement, implemented in the sub-dialog shown here.
11 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Initial flowchart shown here describes the basic
functional requirement (not the UI) – i.e. what should
happen with various combinations of inputs to the fields,
with constraints that define which “paths” through the
“graph” should lead to successful vs unsuccessful
validation.
However, there were also UI behavior requirements where
not just combination of entries, but also order of entry was
significant i.e. the validate button should only be enabled
(therefore clickable at all) when the fields for the other data
items were in certain states.
This required us to change our approach to modeling...
Acceptance Testing Example 2
Model
12 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
This is the same flow refactored to be suitable for
the generation of UI tests. It’s a little more
complex and required thinking creatively about
how to model the requirement and a bit of a
learning curve for our engineers.
More challenging to model but worth it:
• No escaped defects relating to this story were found
in the subsequent pre-release testing, and so far none
have been logged by our users that have adopted the
release this was shipped in.
• Automated regression tests based on the model and
running in CI have proven effective in detecting
regression defects and preventing defect escape from
subsequent sprints.
Acceptance Testing Example 2
Tailoring for UI Order Significance and Automation
13 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Example 3 - TDM Shredder Regression Testing
TDM Shredder - Synthetic Data Generation (e.g. for XML)
Register XSD or
Sample XML if no
XSD
Create Tables in
Staging DB
Register Derived
Tables with TDM
Repository
Import Data from
Sample XML to Derived
Tables in Staging DB
Template-ize for data
generation
Publish Synthetic
Data to Staging DB
Export from Staging
DB
to XML files
Import Data to a
TDM Data Pool from
Staging DB
Staging DB
(for Derived Tables)
TDM
Repository
TDM Shredder TDM DataMaker
Sample
XML
XSD
Generated
XML Files
TDM Shredder Lines of Code: 130,000. i.e. Medium-high complexity sub-component of a product. Lots of complexity in the types of input file
and specs supported, especially with XML. The testing complexity is in the data inputs, as much as in the UI behaviour.
14 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Example 3 - TDM Shredder Model
The main flow.
2 of the 6 sub-flows.
1 main Flow , 6 sub flows, 85 total nodes.
15 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Example 3 - TDM Shredder Regression Testing
Test Configurations – Data-driven Test Steps
Decision table nodes can be driven with many specific
examples of each type, each one is a separate test case.
For each variety of file, different inputs can
be passed using decision table inputs.
Very efficient way to model / represent large number of test cases that differ only in the nature of the test data.
16 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Example 3 - TDM Shredder Regression Testing
Test Case Authoring With and Without MBT
Manual Test Authoring
# of TC
Man Days to
Create
Code
Coverage
Functional Coverage
318 40 55% ~65% (Guess)
Model-based Test Authoring
# of TC
Man Days to
Create
Code Coverage Functional Coverage
450 * 3 63% 67% (Known)
* Test Cases: The model has 17,000 distinct paths (test cases). Far too many to execute so we used the modelling tool’s path selection
capability to select a manageable number of paths smartly, minimizing duplication and preserving best possible functional coverage. We used
an “All in/out edges” algorithm to select 450 test cases (comparable number to the manually written suite).
MBT enabled us to create a better regression suite with a lot less time and effort than doing it
manually. This applies to initial creation and maintenance over time.
Furthermore with the MBT approach we know what the functional coverage is, and we know what we
are choosing to not test and why.
Coverage: The 450 paths selected gave 67% functional coverage and 8% better code coverage than the manually authored test cases. Had
we executed all 17000 we would have 100% functional coverage.
17 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Example 3 - TDM Shredder Regression Testing
Test Automation
Automated 68 test cases out of the 318 manually authored tests cases. Required 30 man days.
Chose one highly complex end to end path (43 nodes) and added automation logic to enable
us to have the model produce a working Protractor script. Required 6 man days.
 Involved around half of the total 85 nodes in the model, so we created a lot of “node code” that would be
re-used heavily if we continued to automate other paths.
 Required just 3 man days to get the next 13 paths done due to increasing re-use of automation logic
already there in each node of the model. As we cover more and more nodes and add automation to more
paths, the marginal cost of automating each additional test case drops rapidly.
From this we estimate that:
 The time required to complete the initial coding of the automation in the tool would be similar to what it
took to manually author the existing automated tests
 The time and cost involved in maintaining the automated test suite when the functionality changes will
be much lower with the model-based approach.
18 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Key Learnings and Insights
The most compelling benefits were in the areas of requirements clarification and designing acceptance test cases for
new stories.
Modelling of entire components to create high functional coverage regression test suites is possible and highly valuable but it is
a lot more work and requires access to info/people that may be very hard for large components or systems that are older and not
well understood.
Getting started is easy but there are significant learning curves around how to model, especially when trying to model UI /
event-driven behaviour where order is significant. This was fairly easy for us to work through as we have highly technical testers
embedded in our dev teams. YMMV.
Huge potential for automated test suite maintenance to become an order of magnitude simpler if automation scripts become fully
model-generated. Our experience with this though was that debugging was a challenge, especially trying to generate raw
automation script (e.g. protractor or Selenium script) directly from the model. Scripting at a higher level of abstraction e.g. against
application specific test automation frameworks may help.
“We modeled a flow for a recent new feature. While we had spent over a week
with the architects and UX designer to define the story and particularly the
acceptance criteria, it took only 5 minutes to realize our requirements were
incomplete. Not only that, but I quickly realized we had not even discussed
edge cases and interactions that were immediately visible from the
requirement modeling and analysis.”
Anand Kameswaran
Product Owner
“The combination of the clarity requirements that had been forced through the
process of modelling, together with the ability to create test cases from the
model to cover every possible path, raised my confidence to a 9/10 that the
testing done was adequate for story acceptance and to prevent defect escape.
I’d say for the prior approach it was more like 3/10.”
Robert Eaglestone
Principal QA Engineer
19 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Challenges and Aspirations
 Incomplete/ambiguous requirements
 Inadequate functional coverage
 Too many defects/rework
ACCEPTANCE TESTING
FOR NEW STORIES
 Lack of visibility to functional
coverage/test efficiency
 Automated Test Suite maintenance
 Automation cost
REGRESSION TESTING
Clear,
complete,
unambiguous
requirements
What We Hoped MBT Would Get Us
Confidence in
functional
coverage
Fewer defects
escaping the
sprint
Increased
effectiveness
and efficiency
Ease of
Maintenance
Reduced
automation cost
20 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Questions?
21 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Drive
Quality and
Testing
Efficiency
Test at the
Speed of
Agile
Must See Demos
Automate
Your
Ecosystem
End-to-end
Theater 3
308
Theater 3
311
Theater 3
312
22 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
Stay connected at communities.ca.com
Thank you.
23 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS
DevOps:
Continuous Delivery
For more information on DevOps: Continuous Delivery,
please visit: http://cainc.to/CAW17-CD

Mais conteúdo relacionado

Mais procurados

Automating Disaster Recovery for Faultless Service Delivery
Automating Disaster Recovery for Faultless Service DeliveryAutomating Disaster Recovery for Faultless Service Delivery
Automating Disaster Recovery for Faultless Service DeliveryCA Technologies
 
Tech Talk: Leverage the combined power of CA Unified Infrastructure Managemen...
Tech Talk: Leverage the combined power of CA Unified Infrastructure Managemen...Tech Talk: Leverage the combined power of CA Unified Infrastructure Managemen...
Tech Talk: Leverage the combined power of CA Unified Infrastructure Managemen...CA Technologies
 
Gartner EA Architecting for DevOps and Hybrid Cloud
Gartner EA Architecting for DevOps and Hybrid CloudGartner EA Architecting for DevOps and Hybrid Cloud
Gartner EA Architecting for DevOps and Hybrid CloudRosalind Radcliffe
 
Improve Network Latency and Hold Service Providers to SLAs
Improve Network Latency and Hold Service Providers to SLAsImprove Network Latency and Hold Service Providers to SLAs
Improve Network Latency and Hold Service Providers to SLAsCA Technologies
 
Desjardins Group Leverages CA Workload Automation as It Begins Its DevOps Jou...
Desjardins Group Leverages CA Workload Automation as It Begins Its DevOps Jou...Desjardins Group Leverages CA Workload Automation as It Begins Its DevOps Jou...
Desjardins Group Leverages CA Workload Automation as It Begins Its DevOps Jou...CA Technologies
 
DevTest Portfolio Overview
DevTest Portfolio OverviewDevTest Portfolio Overview
DevTest Portfolio OverviewCA Technologies
 
Predicting When Your Applications Will Go Off the Rails! Managing DB2 Appli...
Predicting When Your Applications Will Go Off the Rails!  Managing DB2 Appli...Predicting When Your Applications Will Go Off the Rails!  Managing DB2 Appli...
Predicting When Your Applications Will Go Off the Rails! Managing DB2 Appli...CA Technologies
 
Best Practices for Optimizing Mainframe Application Performance with CA Cross...
Best Practices for Optimizing Mainframe Application Performance with CA Cross...Best Practices for Optimizing Mainframe Application Performance with CA Cross...
Best Practices for Optimizing Mainframe Application Performance with CA Cross...CA Technologies
 
How National Australia Bank (NAB) used CA APM during performance testing to i...
How National Australia Bank (NAB) used CA APM during performance testing to i...How National Australia Bank (NAB) used CA APM during performance testing to i...
How National Australia Bank (NAB) used CA APM during performance testing to i...CA Technologies
 
Hands-On Lab: Experience How to Leverage CA Spectrum 10.1 Support for Wirele...
Hands-On Lab: Experience How to Leverage CA Spectrum 10.1 Support for Wirele...Hands-On Lab: Experience How to Leverage CA Spectrum 10.1 Support for Wirele...
Hands-On Lab: Experience How to Leverage CA Spectrum 10.1 Support for Wirele...CA Technologies
 
Tech Talk: Introduction to SDN/NFV Assurance (CA Virtual Network Assurance)
Tech Talk: Introduction to SDN/NFV Assurance (CA Virtual Network Assurance)Tech Talk: Introduction to SDN/NFV Assurance (CA Virtual Network Assurance)
Tech Talk: Introduction to SDN/NFV Assurance (CA Virtual Network Assurance)CA Technologies
 
Con-way Case Study: Optimizing Application Integration Software Development L...
Con-way Case Study: Optimizing Application Integration Software Development L...Con-way Case Study: Optimizing Application Integration Software Development L...
Con-way Case Study: Optimizing Application Integration Software Development L...CA Technologies
 
Pre-Con Education: Building Basic ITSM Workflows in CA Service Management
Pre-Con Education: Building Basic ITSM Workflows in CA Service ManagementPre-Con Education: Building Basic ITSM Workflows in CA Service Management
Pre-Con Education: Building Basic ITSM Workflows in CA Service ManagementCA Technologies
 
High Scalability Network Monitoring for Communications Service Providers
High Scalability Network Monitoring for Communications Service ProvidersHigh Scalability Network Monitoring for Communications Service Providers
High Scalability Network Monitoring for Communications Service ProvidersCA Technologies
 
Integrating SAP into DevOps Pipelines: Why and How
Integrating SAP into DevOps Pipelines: Why and HowIntegrating SAP into DevOps Pipelines: Why and How
Integrating SAP into DevOps Pipelines: Why and HowDevOps.com
 
Maximizing Your CA IDMS™ Investment for the New Application Economy (Part 1)
Maximizing Your CA IDMS™ Investment for the New Application Economy (Part 1)Maximizing Your CA IDMS™ Investment for the New Application Economy (Part 1)
Maximizing Your CA IDMS™ Investment for the New Application Economy (Part 1)CA Technologies
 
How to Test the New Fashioned Way
How to Test the New Fashioned WayHow to Test the New Fashioned Way
How to Test the New Fashioned WayCA Technologies
 
Who’s Minding the SSO Store?
Who’s Minding the SSO Store? Who’s Minding the SSO Store?
Who’s Minding the SSO Store? CA Technologies
 
CA Performance Management Deep Dive
CA Performance Management Deep DiveCA Performance Management Deep Dive
CA Performance Management Deep DiveCA Technologies
 
How to Increase User Accountability by Eliminating the Default User in Unix S...
How to Increase User Accountability by Eliminating the Default User in Unix S...How to Increase User Accountability by Eliminating the Default User in Unix S...
How to Increase User Accountability by Eliminating the Default User in Unix S...CA Technologies
 

Mais procurados (20)

Automating Disaster Recovery for Faultless Service Delivery
Automating Disaster Recovery for Faultless Service DeliveryAutomating Disaster Recovery for Faultless Service Delivery
Automating Disaster Recovery for Faultless Service Delivery
 
Tech Talk: Leverage the combined power of CA Unified Infrastructure Managemen...
Tech Talk: Leverage the combined power of CA Unified Infrastructure Managemen...Tech Talk: Leverage the combined power of CA Unified Infrastructure Managemen...
Tech Talk: Leverage the combined power of CA Unified Infrastructure Managemen...
 
Gartner EA Architecting for DevOps and Hybrid Cloud
Gartner EA Architecting for DevOps and Hybrid CloudGartner EA Architecting for DevOps and Hybrid Cloud
Gartner EA Architecting for DevOps and Hybrid Cloud
 
Improve Network Latency and Hold Service Providers to SLAs
Improve Network Latency and Hold Service Providers to SLAsImprove Network Latency and Hold Service Providers to SLAs
Improve Network Latency and Hold Service Providers to SLAs
 
Desjardins Group Leverages CA Workload Automation as It Begins Its DevOps Jou...
Desjardins Group Leverages CA Workload Automation as It Begins Its DevOps Jou...Desjardins Group Leverages CA Workload Automation as It Begins Its DevOps Jou...
Desjardins Group Leverages CA Workload Automation as It Begins Its DevOps Jou...
 
DevTest Portfolio Overview
DevTest Portfolio OverviewDevTest Portfolio Overview
DevTest Portfolio Overview
 
Predicting When Your Applications Will Go Off the Rails! Managing DB2 Appli...
Predicting When Your Applications Will Go Off the Rails!  Managing DB2 Appli...Predicting When Your Applications Will Go Off the Rails!  Managing DB2 Appli...
Predicting When Your Applications Will Go Off the Rails! Managing DB2 Appli...
 
Best Practices for Optimizing Mainframe Application Performance with CA Cross...
Best Practices for Optimizing Mainframe Application Performance with CA Cross...Best Practices for Optimizing Mainframe Application Performance with CA Cross...
Best Practices for Optimizing Mainframe Application Performance with CA Cross...
 
How National Australia Bank (NAB) used CA APM during performance testing to i...
How National Australia Bank (NAB) used CA APM during performance testing to i...How National Australia Bank (NAB) used CA APM during performance testing to i...
How National Australia Bank (NAB) used CA APM during performance testing to i...
 
Hands-On Lab: Experience How to Leverage CA Spectrum 10.1 Support for Wirele...
Hands-On Lab: Experience How to Leverage CA Spectrum 10.1 Support for Wirele...Hands-On Lab: Experience How to Leverage CA Spectrum 10.1 Support for Wirele...
Hands-On Lab: Experience How to Leverage CA Spectrum 10.1 Support for Wirele...
 
Tech Talk: Introduction to SDN/NFV Assurance (CA Virtual Network Assurance)
Tech Talk: Introduction to SDN/NFV Assurance (CA Virtual Network Assurance)Tech Talk: Introduction to SDN/NFV Assurance (CA Virtual Network Assurance)
Tech Talk: Introduction to SDN/NFV Assurance (CA Virtual Network Assurance)
 
Con-way Case Study: Optimizing Application Integration Software Development L...
Con-way Case Study: Optimizing Application Integration Software Development L...Con-way Case Study: Optimizing Application Integration Software Development L...
Con-way Case Study: Optimizing Application Integration Software Development L...
 
Pre-Con Education: Building Basic ITSM Workflows in CA Service Management
Pre-Con Education: Building Basic ITSM Workflows in CA Service ManagementPre-Con Education: Building Basic ITSM Workflows in CA Service Management
Pre-Con Education: Building Basic ITSM Workflows in CA Service Management
 
High Scalability Network Monitoring for Communications Service Providers
High Scalability Network Monitoring for Communications Service ProvidersHigh Scalability Network Monitoring for Communications Service Providers
High Scalability Network Monitoring for Communications Service Providers
 
Integrating SAP into DevOps Pipelines: Why and How
Integrating SAP into DevOps Pipelines: Why and HowIntegrating SAP into DevOps Pipelines: Why and How
Integrating SAP into DevOps Pipelines: Why and How
 
Maximizing Your CA IDMS™ Investment for the New Application Economy (Part 1)
Maximizing Your CA IDMS™ Investment for the New Application Economy (Part 1)Maximizing Your CA IDMS™ Investment for the New Application Economy (Part 1)
Maximizing Your CA IDMS™ Investment for the New Application Economy (Part 1)
 
How to Test the New Fashioned Way
How to Test the New Fashioned WayHow to Test the New Fashioned Way
How to Test the New Fashioned Way
 
Who’s Minding the SSO Store?
Who’s Minding the SSO Store? Who’s Minding the SSO Store?
Who’s Minding the SSO Store?
 
CA Performance Management Deep Dive
CA Performance Management Deep DiveCA Performance Management Deep Dive
CA Performance Management Deep Dive
 
How to Increase User Accountability by Eliminating the Default User in Unix S...
How to Increase User Accountability by Eliminating the Default User in Unix S...How to Increase User Accountability by Eliminating the Default User in Unix S...
How to Increase User Accountability by Eliminating the Default User in Unix S...
 

Semelhante a Case Study: How CA Went From 40 Days to Three Days Building Crystal-Clear Test Cases While Improving Test Coverage

Pre-Con Ed: There has to be a Better Way to Fast Test Coverage!
Pre-Con Ed: There has to be a Better Way to Fast Test Coverage!Pre-Con Ed: There has to be a Better Way to Fast Test Coverage!
Pre-Con Ed: There has to be a Better Way to Fast Test Coverage!CA Technologies
 
Improvement in Error Resilience in BIST using hamming code
Improvement in Error Resilience in BIST using hamming codeImprovement in Error Resilience in BIST using hamming code
Improvement in Error Resilience in BIST using hamming codeIJMTST Journal
 
Past Experiences and Future Challenges using Automatic Performance Modelling ...
Past Experiences and Future Challenges using Automatic Performance Modelling ...Past Experiences and Future Challenges using Automatic Performance Modelling ...
Past Experiences and Future Challenges using Automatic Performance Modelling ...Paul Brebner
 
Advanced Verification Methodology for Complex System on Chip Verification
Advanced Verification Methodology for Complex System on Chip VerificationAdvanced Verification Methodology for Complex System on Chip Verification
Advanced Verification Methodology for Complex System on Chip VerificationVLSICS Design
 
Service Virtualization: What, Who, When, and How
Service Virtualization: What, Who, When, and HowService Virtualization: What, Who, When, and How
Service Virtualization: What, Who, When, and HowTechWell
 
Performance modeling and simulation for accumulo applications
Performance modeling and simulation for accumulo applicationsPerformance modeling and simulation for accumulo applications
Performance modeling and simulation for accumulo applicationsAccumulo Summit
 
1 SDEV 460 – Homework 4 Input Validation and Busine
1  SDEV 460 – Homework 4 Input Validation and Busine1  SDEV 460 – Homework 4 Input Validation and Busine
1 SDEV 460 – Homework 4 Input Validation and BusineVannaJoy20
 
Implement Test Harness For Streaming Data Pipelines
Implement Test Harness For Streaming Data PipelinesImplement Test Harness For Streaming Data Pipelines
Implement Test Harness For Streaming Data PipelinesKnoldus Inc.
 
CTD for client signoff and early requirements validation - 11th IEEE SW Eng...
CTD for client signoff and early requirements validation   - 11th IEEE SW Eng...CTD for client signoff and early requirements validation   - 11th IEEE SW Eng...
CTD for client signoff and early requirements validation - 11th IEEE SW Eng...Saritha Route
 
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...Andreas Grabner
 
Evolution of Test Automation
Evolution of Test AutomationEvolution of Test Automation
Evolution of Test AutomationDharmik Rajput
 
The Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs PublicThe Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs PublicDavid Solivan
 
MLOps and Reproducible ML on AWS with Kubeflow and SageMaker
MLOps and Reproducible ML on AWS with Kubeflow and SageMakerMLOps and Reproducible ML on AWS with Kubeflow and SageMaker
MLOps and Reproducible ML on AWS with Kubeflow and SageMakerProvectus
 
Service Virtualization - Next Gen Testing Conference Singapore 2013
Service Virtualization - Next Gen Testing Conference Singapore 2013Service Virtualization - Next Gen Testing Conference Singapore 2013
Service Virtualization - Next Gen Testing Conference Singapore 2013Min Fang
 
Microservices Testing Strategies: The Good, the Bad, and the Reality
Microservices Testing Strategies: The Good, the Bad, and the RealityMicroservices Testing Strategies: The Good, the Bad, and the Reality
Microservices Testing Strategies: The Good, the Bad, and the RealityTechWell
 
CA Service Virtualization vs mocks stubs
CA Service Virtualization vs mocks stubsCA Service Virtualization vs mocks stubs
CA Service Virtualization vs mocks stubsAlan Baptista
 
5 Years Of Building SaaS On AWS
5 Years Of Building SaaS On AWS5 Years Of Building SaaS On AWS
5 Years Of Building SaaS On AWSChristian Beedgen
 
Automatic Performance Modelling from Application Performance Management (APM)...
Automatic Performance Modelling from Application Performance Management (APM)...Automatic Performance Modelling from Application Performance Management (APM)...
Automatic Performance Modelling from Application Performance Management (APM)...Paul Brebner
 
Pragmatic Architecture in .NET
Pragmatic Architecture in .NETPragmatic Architecture in .NET
Pragmatic Architecture in .NEThousecor
 
What are Software Testing Methodologies | Software Testing Techniques | Edureka
What are Software Testing Methodologies | Software Testing Techniques | EdurekaWhat are Software Testing Methodologies | Software Testing Techniques | Edureka
What are Software Testing Methodologies | Software Testing Techniques | EdurekaEdureka!
 

Semelhante a Case Study: How CA Went From 40 Days to Three Days Building Crystal-Clear Test Cases While Improving Test Coverage (20)

Pre-Con Ed: There has to be a Better Way to Fast Test Coverage!
Pre-Con Ed: There has to be a Better Way to Fast Test Coverage!Pre-Con Ed: There has to be a Better Way to Fast Test Coverage!
Pre-Con Ed: There has to be a Better Way to Fast Test Coverage!
 
Improvement in Error Resilience in BIST using hamming code
Improvement in Error Resilience in BIST using hamming codeImprovement in Error Resilience in BIST using hamming code
Improvement in Error Resilience in BIST using hamming code
 
Past Experiences and Future Challenges using Automatic Performance Modelling ...
Past Experiences and Future Challenges using Automatic Performance Modelling ...Past Experiences and Future Challenges using Automatic Performance Modelling ...
Past Experiences and Future Challenges using Automatic Performance Modelling ...
 
Advanced Verification Methodology for Complex System on Chip Verification
Advanced Verification Methodology for Complex System on Chip VerificationAdvanced Verification Methodology for Complex System on Chip Verification
Advanced Verification Methodology for Complex System on Chip Verification
 
Service Virtualization: What, Who, When, and How
Service Virtualization: What, Who, When, and HowService Virtualization: What, Who, When, and How
Service Virtualization: What, Who, When, and How
 
Performance modeling and simulation for accumulo applications
Performance modeling and simulation for accumulo applicationsPerformance modeling and simulation for accumulo applications
Performance modeling and simulation for accumulo applications
 
1 SDEV 460 – Homework 4 Input Validation and Busine
1  SDEV 460 – Homework 4 Input Validation and Busine1  SDEV 460 – Homework 4 Input Validation and Busine
1 SDEV 460 – Homework 4 Input Validation and Busine
 
Implement Test Harness For Streaming Data Pipelines
Implement Test Harness For Streaming Data PipelinesImplement Test Harness For Streaming Data Pipelines
Implement Test Harness For Streaming Data Pipelines
 
CTD for client signoff and early requirements validation - 11th IEEE SW Eng...
CTD for client signoff and early requirements validation   - 11th IEEE SW Eng...CTD for client signoff and early requirements validation   - 11th IEEE SW Eng...
CTD for client signoff and early requirements validation - 11th IEEE SW Eng...
 
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
Deploy Faster Without Failing Faster - Metrics-Driven - Dynatrace User Groups...
 
Evolution of Test Automation
Evolution of Test AutomationEvolution of Test Automation
Evolution of Test Automation
 
The Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs PublicThe Magic Of Application Lifecycle Management In Vs Public
The Magic Of Application Lifecycle Management In Vs Public
 
MLOps and Reproducible ML on AWS with Kubeflow and SageMaker
MLOps and Reproducible ML on AWS with Kubeflow and SageMakerMLOps and Reproducible ML on AWS with Kubeflow and SageMaker
MLOps and Reproducible ML on AWS with Kubeflow and SageMaker
 
Service Virtualization - Next Gen Testing Conference Singapore 2013
Service Virtualization - Next Gen Testing Conference Singapore 2013Service Virtualization - Next Gen Testing Conference Singapore 2013
Service Virtualization - Next Gen Testing Conference Singapore 2013
 
Microservices Testing Strategies: The Good, the Bad, and the Reality
Microservices Testing Strategies: The Good, the Bad, and the RealityMicroservices Testing Strategies: The Good, the Bad, and the Reality
Microservices Testing Strategies: The Good, the Bad, and the Reality
 
CA Service Virtualization vs mocks stubs
CA Service Virtualization vs mocks stubsCA Service Virtualization vs mocks stubs
CA Service Virtualization vs mocks stubs
 
5 Years Of Building SaaS On AWS
5 Years Of Building SaaS On AWS5 Years Of Building SaaS On AWS
5 Years Of Building SaaS On AWS
 
Automatic Performance Modelling from Application Performance Management (APM)...
Automatic Performance Modelling from Application Performance Management (APM)...Automatic Performance Modelling from Application Performance Management (APM)...
Automatic Performance Modelling from Application Performance Management (APM)...
 
Pragmatic Architecture in .NET
Pragmatic Architecture in .NETPragmatic Architecture in .NET
Pragmatic Architecture in .NET
 
What are Software Testing Methodologies | Software Testing Techniques | Edureka
What are Software Testing Methodologies | Software Testing Techniques | EdurekaWhat are Software Testing Methodologies | Software Testing Techniques | Edureka
What are Software Testing Methodologies | Software Testing Techniques | Edureka
 

Mais de CA Technologies

Mainframe as a Service: Sample a Buffet of IBM z/OS® Platform Excellence
Mainframe as a Service: Sample a Buffet of IBM z/OS® Platform ExcellenceMainframe as a Service: Sample a Buffet of IBM z/OS® Platform Excellence
Mainframe as a Service: Sample a Buffet of IBM z/OS® Platform ExcellenceCA Technologies
 
Case Study: How The Home Depot Built Quality Into Software Development
Case Study: How The Home Depot Built Quality Into Software DevelopmentCase Study: How The Home Depot Built Quality Into Software Development
Case Study: How The Home Depot Built Quality Into Software DevelopmentCA Technologies
 
Pre-Con Ed: Privileged Identity Governance: Are You Certifying Privileged Use...
Pre-Con Ed: Privileged Identity Governance: Are You Certifying Privileged Use...Pre-Con Ed: Privileged Identity Governance: Are You Certifying Privileged Use...
Pre-Con Ed: Privileged Identity Governance: Are You Certifying Privileged Use...CA Technologies
 
Case Study: Privileged Access in a World on Time
Case Study: Privileged Access in a World on TimeCase Study: Privileged Access in a World on Time
Case Study: Privileged Access in a World on TimeCA Technologies
 
Case Study: How SGN Used Attack Path Mapping to Control Privileged Access in ...
Case Study: How SGN Used Attack Path Mapping to Control Privileged Access in ...Case Study: How SGN Used Attack Path Mapping to Control Privileged Access in ...
Case Study: How SGN Used Attack Path Mapping to Control Privileged Access in ...CA Technologies
 
Case Study: Putting Citizens at The Center of Digital Government
Case Study: Putting Citizens at The Center of Digital GovernmentCase Study: Putting Citizens at The Center of Digital Government
Case Study: Putting Citizens at The Center of Digital GovernmentCA Technologies
 
Making Security Work—Implementing a Transformational Security Program
Making Security Work—Implementing a Transformational Security ProgramMaking Security Work—Implementing a Transformational Security Program
Making Security Work—Implementing a Transformational Security ProgramCA Technologies
 
Keynote: Making Security a Competitive Advantage
Keynote: Making Security a Competitive AdvantageKeynote: Making Security a Competitive Advantage
Keynote: Making Security a Competitive AdvantageCA Technologies
 
Emerging Managed Services Opportunities in Identity and Access Management
Emerging Managed Services Opportunities in Identity and Access ManagementEmerging Managed Services Opportunities in Identity and Access Management
Emerging Managed Services Opportunities in Identity and Access ManagementCA Technologies
 
The Unmet Demand for Premium Cloud Monitoring Services—and How Service Provid...
The Unmet Demand for Premium Cloud Monitoring Services—and How Service Provid...The Unmet Demand for Premium Cloud Monitoring Services—and How Service Provid...
The Unmet Demand for Premium Cloud Monitoring Services—and How Service Provid...CA Technologies
 
Leveraging Monitoring Governance: How Service Providers Can Boost Operational...
Leveraging Monitoring Governance: How Service Providers Can Boost Operational...Leveraging Monitoring Governance: How Service Providers Can Boost Operational...
Leveraging Monitoring Governance: How Service Providers Can Boost Operational...CA Technologies
 
The Next Big Service Provider Opportunity—Beyond Infrastructure: Architecting...
The Next Big Service Provider Opportunity—Beyond Infrastructure: Architecting...The Next Big Service Provider Opportunity—Beyond Infrastructure: Architecting...
The Next Big Service Provider Opportunity—Beyond Infrastructure: Architecting...CA Technologies
 
Application Experience Analytics Services: The Strategic Digital Transformati...
Application Experience Analytics Services: The Strategic Digital Transformati...Application Experience Analytics Services: The Strategic Digital Transformati...
Application Experience Analytics Services: The Strategic Digital Transformati...CA Technologies
 
Application Experience Analytics Services: The Strategic Digital Transformati...
Application Experience Analytics Services: The Strategic Digital Transformati...Application Experience Analytics Services: The Strategic Digital Transformati...
Application Experience Analytics Services: The Strategic Digital Transformati...CA Technologies
 
Strategic Direction Session: Deliver Next-Gen IT Ops with CA Mainframe Operat...
Strategic Direction Session: Deliver Next-Gen IT Ops with CA Mainframe Operat...Strategic Direction Session: Deliver Next-Gen IT Ops with CA Mainframe Operat...
Strategic Direction Session: Deliver Next-Gen IT Ops with CA Mainframe Operat...CA Technologies
 
Strategic Direction Session: Enhancing Data Privacy with Data-Centric Securit...
Strategic Direction Session: Enhancing Data Privacy with Data-Centric Securit...Strategic Direction Session: Enhancing Data Privacy with Data-Centric Securit...
Strategic Direction Session: Enhancing Data Privacy with Data-Centric Securit...CA Technologies
 
Blockchain: Strategies for Moving From Hype to Realities of Deployment
Blockchain: Strategies for Moving From Hype to Realities of DeploymentBlockchain: Strategies for Moving From Hype to Realities of Deployment
Blockchain: Strategies for Moving From Hype to Realities of DeploymentCA Technologies
 
Establish Digital Trust as the Currency of Digital Enterprise
Establish Digital Trust as the Currency of Digital EnterpriseEstablish Digital Trust as the Currency of Digital Enterprise
Establish Digital Trust as the Currency of Digital EnterpriseCA Technologies
 
How Components Increase Speed and Risk
How Components Increase Speed and RiskHow Components Increase Speed and Risk
How Components Increase Speed and RiskCA Technologies
 
Securing Your Enterprise Continuous Delivery Pipelines with CA Automation Sol...
Securing Your Enterprise Continuous Delivery Pipelines with CA Automation Sol...Securing Your Enterprise Continuous Delivery Pipelines with CA Automation Sol...
Securing Your Enterprise Continuous Delivery Pipelines with CA Automation Sol...CA Technologies
 

Mais de CA Technologies (20)

Mainframe as a Service: Sample a Buffet of IBM z/OS® Platform Excellence
Mainframe as a Service: Sample a Buffet of IBM z/OS® Platform ExcellenceMainframe as a Service: Sample a Buffet of IBM z/OS® Platform Excellence
Mainframe as a Service: Sample a Buffet of IBM z/OS® Platform Excellence
 
Case Study: How The Home Depot Built Quality Into Software Development
Case Study: How The Home Depot Built Quality Into Software DevelopmentCase Study: How The Home Depot Built Quality Into Software Development
Case Study: How The Home Depot Built Quality Into Software Development
 
Pre-Con Ed: Privileged Identity Governance: Are You Certifying Privileged Use...
Pre-Con Ed: Privileged Identity Governance: Are You Certifying Privileged Use...Pre-Con Ed: Privileged Identity Governance: Are You Certifying Privileged Use...
Pre-Con Ed: Privileged Identity Governance: Are You Certifying Privileged Use...
 
Case Study: Privileged Access in a World on Time
Case Study: Privileged Access in a World on TimeCase Study: Privileged Access in a World on Time
Case Study: Privileged Access in a World on Time
 
Case Study: How SGN Used Attack Path Mapping to Control Privileged Access in ...
Case Study: How SGN Used Attack Path Mapping to Control Privileged Access in ...Case Study: How SGN Used Attack Path Mapping to Control Privileged Access in ...
Case Study: How SGN Used Attack Path Mapping to Control Privileged Access in ...
 
Case Study: Putting Citizens at The Center of Digital Government
Case Study: Putting Citizens at The Center of Digital GovernmentCase Study: Putting Citizens at The Center of Digital Government
Case Study: Putting Citizens at The Center of Digital Government
 
Making Security Work—Implementing a Transformational Security Program
Making Security Work—Implementing a Transformational Security ProgramMaking Security Work—Implementing a Transformational Security Program
Making Security Work—Implementing a Transformational Security Program
 
Keynote: Making Security a Competitive Advantage
Keynote: Making Security a Competitive AdvantageKeynote: Making Security a Competitive Advantage
Keynote: Making Security a Competitive Advantage
 
Emerging Managed Services Opportunities in Identity and Access Management
Emerging Managed Services Opportunities in Identity and Access ManagementEmerging Managed Services Opportunities in Identity and Access Management
Emerging Managed Services Opportunities in Identity and Access Management
 
The Unmet Demand for Premium Cloud Monitoring Services—and How Service Provid...
The Unmet Demand for Premium Cloud Monitoring Services—and How Service Provid...The Unmet Demand for Premium Cloud Monitoring Services—and How Service Provid...
The Unmet Demand for Premium Cloud Monitoring Services—and How Service Provid...
 
Leveraging Monitoring Governance: How Service Providers Can Boost Operational...
Leveraging Monitoring Governance: How Service Providers Can Boost Operational...Leveraging Monitoring Governance: How Service Providers Can Boost Operational...
Leveraging Monitoring Governance: How Service Providers Can Boost Operational...
 
The Next Big Service Provider Opportunity—Beyond Infrastructure: Architecting...
The Next Big Service Provider Opportunity—Beyond Infrastructure: Architecting...The Next Big Service Provider Opportunity—Beyond Infrastructure: Architecting...
The Next Big Service Provider Opportunity—Beyond Infrastructure: Architecting...
 
Application Experience Analytics Services: The Strategic Digital Transformati...
Application Experience Analytics Services: The Strategic Digital Transformati...Application Experience Analytics Services: The Strategic Digital Transformati...
Application Experience Analytics Services: The Strategic Digital Transformati...
 
Application Experience Analytics Services: The Strategic Digital Transformati...
Application Experience Analytics Services: The Strategic Digital Transformati...Application Experience Analytics Services: The Strategic Digital Transformati...
Application Experience Analytics Services: The Strategic Digital Transformati...
 
Strategic Direction Session: Deliver Next-Gen IT Ops with CA Mainframe Operat...
Strategic Direction Session: Deliver Next-Gen IT Ops with CA Mainframe Operat...Strategic Direction Session: Deliver Next-Gen IT Ops with CA Mainframe Operat...
Strategic Direction Session: Deliver Next-Gen IT Ops with CA Mainframe Operat...
 
Strategic Direction Session: Enhancing Data Privacy with Data-Centric Securit...
Strategic Direction Session: Enhancing Data Privacy with Data-Centric Securit...Strategic Direction Session: Enhancing Data Privacy with Data-Centric Securit...
Strategic Direction Session: Enhancing Data Privacy with Data-Centric Securit...
 
Blockchain: Strategies for Moving From Hype to Realities of Deployment
Blockchain: Strategies for Moving From Hype to Realities of DeploymentBlockchain: Strategies for Moving From Hype to Realities of Deployment
Blockchain: Strategies for Moving From Hype to Realities of Deployment
 
Establish Digital Trust as the Currency of Digital Enterprise
Establish Digital Trust as the Currency of Digital EnterpriseEstablish Digital Trust as the Currency of Digital Enterprise
Establish Digital Trust as the Currency of Digital Enterprise
 
How Components Increase Speed and Risk
How Components Increase Speed and RiskHow Components Increase Speed and Risk
How Components Increase Speed and Risk
 
Securing Your Enterprise Continuous Delivery Pipelines with CA Automation Sol...
Securing Your Enterprise Continuous Delivery Pipelines with CA Automation Sol...Securing Your Enterprise Continuous Delivery Pipelines with CA Automation Sol...
Securing Your Enterprise Continuous Delivery Pipelines with CA Automation Sol...
 

Último

Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfinfogdgmi
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServiceRenan Moreira de Oliveira
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIUdaiappa Ramachandran
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UbiTrack UK
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesMd Hossain Ali
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxYounusS2
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfDianaGray10
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioChristian Posta
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopBachir Benyammi
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsSeth Reyes
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintMahmoud Rabie
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationIES VE
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfAnna Loughnan Colquhoun
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxGDSC PJATK
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxMatsuo Lab
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...Aggregage
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding TeamAdam Moalla
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPathCommunity
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdfJamie (Taka) Wang
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8DianaGray10
 

Último (20)

Videogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdfVideogame localization & technology_ how to enhance the power of translation.pdf
Videogame localization & technology_ how to enhance the power of translation.pdf
 
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer ServicePicPay - GenAI Finance Assistant - ChatGPT for Customer Service
PicPay - GenAI Finance Assistant - ChatGPT for Customer Service
 
RAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AIRAG Patterns and Vector Search in Generative AI
RAG Patterns and Vector Search in Generative AI
 
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
UWB Technology for Enhanced Indoor and Outdoor Positioning in Physiological M...
 
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just MinutesAI Fame Rush Review – Virtual Influencer Creation In Just Minutes
AI Fame Rush Review – Virtual Influencer Creation In Just Minutes
 
Babel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptxBabel Compiler - Transforming JavaScript for All Browsers.pptx
Babel Compiler - Transforming JavaScript for All Browsers.pptx
 
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdfUiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
UiPath Solutions Management Preview - Northern CA Chapter - March 22.pdf
 
Comparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and IstioComparing Sidecar-less Service Mesh from Cilium and Istio
Comparing Sidecar-less Service Mesh from Cilium and Istio
 
NIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 WorkshopNIST Cybersecurity Framework (CSF) 2.0 Workshop
NIST Cybersecurity Framework (CSF) 2.0 Workshop
 
Computer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and HazardsComputer 10: Lesson 10 - Online Crimes and Hazards
Computer 10: Lesson 10 - Online Crimes and Hazards
 
Empowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership BlueprintEmpowering Africa's Next Generation: The AI Leadership Blueprint
Empowering Africa's Next Generation: The AI Leadership Blueprint
 
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve DecarbonizationUsing IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
Using IESVE for Loads, Sizing and Heat Pump Modeling to Achieve Decarbonization
 
Spring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdfSpring24-Release Overview - Wellingtion User Group-1.pdf
Spring24-Release Overview - Wellingtion User Group-1.pdf
 
Cybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptxCybersecurity Workshop #1.pptx
Cybersecurity Workshop #1.pptx
 
Introduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptxIntroduction to Matsuo Laboratory (ENG).pptx
Introduction to Matsuo Laboratory (ENG).pptx
 
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
The Data Metaverse: Unpacking the Roles, Use Cases, and Tech Trends in Data a...
 
9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team9 Steps For Building Winning Founding Team
9 Steps For Building Winning Founding Team
 
UiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation DevelopersUiPath Community: AI for UiPath Automation Developers
UiPath Community: AI for UiPath Automation Developers
 
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
20200723_insight_release_plan_v6.pdf20200723_insight_release_plan_v6.pdf
 
UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8UiPath Studio Web workshop series - Day 8
UiPath Studio Web workshop series - Day 8
 

Case Study: How CA Went From 40 Days to Three Days Building Crystal-Clear Test Cases While Improving Test Coverage

  • 1. Case Study: How CA Went From 40 Days to Three Days Building Crystal-Clear Test Cases While Improving Test Coverage Stephen Tyler DO3T022S VP, Software Engineering CA Technologies DEVOPS – CONTINUOUS DELIVERY
  • 2. 2 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS © 2017 CA. All rights reserved. All trademarks referenced herein belong to their respective companies. The content provided in this CA World 2017 presentation is intended for informational purposes only and does not form any type of warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA. For Informational Purposes Only Terms of This Presentation
  • 3. 3 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Abstract Here at CA Technologies, our development teams share many of the same challenges producing quality software as our customers. This session will cover how we: Introduced collaborative modeling between our product owners and engineers to drive extreme clarity around requirements during backlog refinement, before a story was picked up for a sprint. Built out a requirements model to generate a set of test cases that can fully cover the acceptance criteria for your stories. Gained the result—reducing the time it took to build out regression test cases from 40 days to three—while improving test coverage with less test cases Stephen Tyler CA Technologies VP, Software Engineering
  • 4. 4 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Agenda WHY: OUR CHALLENGES AND WHY WE THOUGHT MBT MIGHT HELP WHAT: OUR HYPOTHESES AND METHODS RESULTS: WHAT WORKED, LESSONS LEARNED, NEXT STEPS 1 2 3
  • 5. 5 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Challenges and Aspirations  Incomplete/ambiguous requirements  Inadequate functional coverage  Too many defects leading to rework ACCEPTANCE TESTING FOR NEW STORIES  Lack of visibility to functional coverage/test efficiency  Test Suite maintenance  Automation cost REGRESSION TESTING Clear, complete, unambiguous requirements What We Hoped MBT Would Get Us Confidence in functional coverage Fewer defects escaping the sprint Increased effectiveness and efficiency Ease of Maintenance Reduced automation cost
  • 6. 6 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Acceptance Testing Example 1 Configure Connection and Resources Dialog
  • 7. 7 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS In the MBT tool we used, the functionality is modeled as a flowchart/graph as shown here. Each node of the graph is an object on the UI to which data is entered and metadata associated to the node defines what data ins and outs it supports. Each path through the graph is a test case driven by particular data entries or decisions at various nodes along the path. In this model there are 23 distinct paths. That is 23 test cases to fully cover the functionality as modelled. Acceptance Testing Example 1 The Model
  • 8. 8 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS All 23 possible paths through the model Path 15 of 23 visually highlighted on the graph and textually represented (human readable test steps) Model-generated Protractor script to execute path 15 Generating Test Cases and Automated Tests
  • 9. 9 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Automating the Automation Export the “Automation layer” for each path through the model. The script snippets for the relevant actions on each node in the exported paths are concatenated with appropriate variable substitution to produce an executable script corresponding to that path in the model. Attach Automation “actions” to the outputs of each node in the model. Each action is defined with parameterized text snippets.
  • 10. 10 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Acceptance Testing Example 2 Service Configuration Sub-dialog We will focus on the “client-facing certificate configuration” requirement, implemented in the sub-dialog shown here.
  • 11. 11 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Initial flowchart shown here describes the basic functional requirement (not the UI) – i.e. what should happen with various combinations of inputs to the fields, with constraints that define which “paths” through the “graph” should lead to successful vs unsuccessful validation. However, there were also UI behavior requirements where not just combination of entries, but also order of entry was significant i.e. the validate button should only be enabled (therefore clickable at all) when the fields for the other data items were in certain states. This required us to change our approach to modeling... Acceptance Testing Example 2 Model
  • 12. 12 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS This is the same flow refactored to be suitable for the generation of UI tests. It’s a little more complex and required thinking creatively about how to model the requirement and a bit of a learning curve for our engineers. More challenging to model but worth it: • No escaped defects relating to this story were found in the subsequent pre-release testing, and so far none have been logged by our users that have adopted the release this was shipped in. • Automated regression tests based on the model and running in CI have proven effective in detecting regression defects and preventing defect escape from subsequent sprints. Acceptance Testing Example 2 Tailoring for UI Order Significance and Automation
  • 13. 13 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Example 3 - TDM Shredder Regression Testing TDM Shredder - Synthetic Data Generation (e.g. for XML) Register XSD or Sample XML if no XSD Create Tables in Staging DB Register Derived Tables with TDM Repository Import Data from Sample XML to Derived Tables in Staging DB Template-ize for data generation Publish Synthetic Data to Staging DB Export from Staging DB to XML files Import Data to a TDM Data Pool from Staging DB Staging DB (for Derived Tables) TDM Repository TDM Shredder TDM DataMaker Sample XML XSD Generated XML Files TDM Shredder Lines of Code: 130,000. i.e. Medium-high complexity sub-component of a product. Lots of complexity in the types of input file and specs supported, especially with XML. The testing complexity is in the data inputs, as much as in the UI behaviour.
  • 14. 14 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Example 3 - TDM Shredder Model The main flow. 2 of the 6 sub-flows. 1 main Flow , 6 sub flows, 85 total nodes.
  • 15. 15 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Example 3 - TDM Shredder Regression Testing Test Configurations – Data-driven Test Steps Decision table nodes can be driven with many specific examples of each type, each one is a separate test case. For each variety of file, different inputs can be passed using decision table inputs. Very efficient way to model / represent large number of test cases that differ only in the nature of the test data.
  • 16. 16 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Example 3 - TDM Shredder Regression Testing Test Case Authoring With and Without MBT Manual Test Authoring # of TC Man Days to Create Code Coverage Functional Coverage 318 40 55% ~65% (Guess) Model-based Test Authoring # of TC Man Days to Create Code Coverage Functional Coverage 450 * 3 63% 67% (Known) * Test Cases: The model has 17,000 distinct paths (test cases). Far too many to execute so we used the modelling tool’s path selection capability to select a manageable number of paths smartly, minimizing duplication and preserving best possible functional coverage. We used an “All in/out edges” algorithm to select 450 test cases (comparable number to the manually written suite). MBT enabled us to create a better regression suite with a lot less time and effort than doing it manually. This applies to initial creation and maintenance over time. Furthermore with the MBT approach we know what the functional coverage is, and we know what we are choosing to not test and why. Coverage: The 450 paths selected gave 67% functional coverage and 8% better code coverage than the manually authored test cases. Had we executed all 17000 we would have 100% functional coverage.
  • 17. 17 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Example 3 - TDM Shredder Regression Testing Test Automation Automated 68 test cases out of the 318 manually authored tests cases. Required 30 man days. Chose one highly complex end to end path (43 nodes) and added automation logic to enable us to have the model produce a working Protractor script. Required 6 man days.  Involved around half of the total 85 nodes in the model, so we created a lot of “node code” that would be re-used heavily if we continued to automate other paths.  Required just 3 man days to get the next 13 paths done due to increasing re-use of automation logic already there in each node of the model. As we cover more and more nodes and add automation to more paths, the marginal cost of automating each additional test case drops rapidly. From this we estimate that:  The time required to complete the initial coding of the automation in the tool would be similar to what it took to manually author the existing automated tests  The time and cost involved in maintaining the automated test suite when the functionality changes will be much lower with the model-based approach.
  • 18. 18 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Key Learnings and Insights The most compelling benefits were in the areas of requirements clarification and designing acceptance test cases for new stories. Modelling of entire components to create high functional coverage regression test suites is possible and highly valuable but it is a lot more work and requires access to info/people that may be very hard for large components or systems that are older and not well understood. Getting started is easy but there are significant learning curves around how to model, especially when trying to model UI / event-driven behaviour where order is significant. This was fairly easy for us to work through as we have highly technical testers embedded in our dev teams. YMMV. Huge potential for automated test suite maintenance to become an order of magnitude simpler if automation scripts become fully model-generated. Our experience with this though was that debugging was a challenge, especially trying to generate raw automation script (e.g. protractor or Selenium script) directly from the model. Scripting at a higher level of abstraction e.g. against application specific test automation frameworks may help. “We modeled a flow for a recent new feature. While we had spent over a week with the architects and UX designer to define the story and particularly the acceptance criteria, it took only 5 minutes to realize our requirements were incomplete. Not only that, but I quickly realized we had not even discussed edge cases and interactions that were immediately visible from the requirement modeling and analysis.” Anand Kameswaran Product Owner “The combination of the clarity requirements that had been forced through the process of modelling, together with the ability to create test cases from the model to cover every possible path, raised my confidence to a 9/10 that the testing done was adequate for story acceptance and to prevent defect escape. I’d say for the prior approach it was more like 3/10.” Robert Eaglestone Principal QA Engineer
  • 19. 19 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Challenges and Aspirations  Incomplete/ambiguous requirements  Inadequate functional coverage  Too many defects/rework ACCEPTANCE TESTING FOR NEW STORIES  Lack of visibility to functional coverage/test efficiency  Automated Test Suite maintenance  Automation cost REGRESSION TESTING Clear, complete, unambiguous requirements What We Hoped MBT Would Get Us Confidence in functional coverage Fewer defects escaping the sprint Increased effectiveness and efficiency Ease of Maintenance Reduced automation cost
  • 20. 20 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Questions?
  • 21. 21 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Drive Quality and Testing Efficiency Test at the Speed of Agile Must See Demos Automate Your Ecosystem End-to-end Theater 3 308 Theater 3 311 Theater 3 312
  • 22. 22 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS Stay connected at communities.ca.com Thank you.
  • 23. 23 COPYRIGHT © 2017 CA. ALL RIGHTS RESERVED#CAWORLD #NOBARRIERS DevOps: Continuous Delivery For more information on DevOps: Continuous Delivery, please visit: http://cainc.to/CAW17-CD