How to Solve Problems of Testing in Agile Environments

World®
’16
How to Solve Problems of Testing
in Agile Environments
Jonathon Wright – Director of Digital Assurance – CA Technologies
Huw Price – VP of Application Delivery – CA Technologies
DO5X29E
DEVOPS

2 © 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD
For Informational Purposes Only
Terms of this Presentation
© 2016 CA. All rights reserved. All trademarks referenced herein belong to their respective companies.
The content provided in this CA World 2016 presentation is intended for informational purposes only and does not form any type of
warranty. The information provided by a CA partner and/or CA customer has not been reviewed for accuracy by CA.

Abstract
Companies adopting an agile framework for software development know that one area still
contributing to slowdowns is testing. In all its forms—regression, system, performance, etc.—
testing is not optimized in the software development lifecycle. The importance of test data
management—having the right data delivered to the right place, at the right time—for testing
purposes is often overlooked.
Poor test case design and the inefficient provisioning of poor quality data mean that test teams
find themselves unable to keep up with the rate of change while fully testing systems. Quality is
then compromised in favour of delivering software on time and within budget. Organizations
that want to implement continuous delivery must rethink their testing and development
processes rather than simply reshuffle them, and reconsider their approach to test data
management and test case design. By using an end-to-end approach to testing, driven by
requirements, organizations can do shift-left testing, mitigate risk and minimize defect creation,
thereby delivering quality software faster, and for less cost.
Huw Price
CA Technologies
Vice President of
Application Delivery
CA Technologies
Director of Digital
Assurance
Jonathon
Wright

Agenda
INTRODUCTION
DIGITAL EVOLUTION
COGNITIVE ADAPTIVE ADOPTION
SOLUTIONS THINKING
DIGITAL ASSURANCE
CONTINUOUS ADAPTIVE TESTING
1
2
3
4
5
6

Shortened Release
Cycles
Demand for High
Quality
Balancing Today’s Demands
Customer needs Moving towards Agile development
Competitive pressures
Digital transformation initiatives
Most testing is manual, not automated
Better test data needed
Impacts
Customer
Experience

Businesses no longer have the luxury of time.
Rising customer expectations, competitive threats,
and increased consumer choice have turned rapid
delivery and iteration of software applications into
a competitive differentiator.
Optimizing the delivery value chain has become
the new means to better business outcomes.
TechRadar™: Continuous Software Delivery, Q2 2015, Forrester
Research, May 1, 2015
“
”
2014 Vanson Bourne study commissioned by CA
UNPRECEDENTED
94%of executives face increased
pressure to release apps
more quickly
“
The Journey to Digital Transformation

Cognitive Adoption – ‘Evolution, Over Revolution’
1 – “The Battle for Competitive Advantage in the App Economy”, Oxford Economics, 2015
VALUE-DRIVEN
TIME
1960s 1990s TODAY
of enterprises believe that they need value-driven
algorithmic business models to respond to disruption
in the next 3 years.
78%
Legacy is your Legacy
(API)
Digitalization of your Core
(SOA)
Digital (Cognitive Adoption)

78% of Executives in most B2C industries say their business will be disrupted by digital
technologies in the next 12 months. 20% say their business will be massively disrupted.
- Forrester State of Digital Business - Big Bang Disruptions - Larry Downes - RallyON Keynote

Companies Look to DevOps for the Answer
Vanson Bourne – DevOps survey 2014
Benefits of DevOps
Percent improvements in business areas
Collaboration
App quality
Customers
New services
Time-to-market
Revenue
Cost savings
Deploy frequency
Plan to adopt DevOps
Next 3 to 5years
15%88
PERCENT
18% 18%
19% 19% 19%
21% 21%

Our Focus…
Dramatically Improve Testing in Digital
Better. Value. Smarter.
*National Institute of Standards & Technology
$60B $2.5BPer Year of IT Budget
for QA and Testing*
Per Year on
Testing Tools
How About Focusing on Minimum Viable Quality?

Limited test resources and
lack of Automation
of all testing is
still manual70%
Source: Bloor Research, 2014
Test Data
for setup & staging
of time spent
looking for50%
Source: Bloor Research, 2014
Agile Requirements Designer, TDM and RA
Build of various dev & testing
Environments
of critical dependencies
are unavailable when
dev and test teams
need them
56%
Source: voke Market Snapshot TM Report: Service
Virtualization – January 2015
Is Testing still the Bottleneck?
Digital Testing

DESIGNOPS
Continuous Delivery - Sub-Value Streams
1
Continuous
Delivery
2
Continuous
Testing
3
Continuous
Staging
4
Continuous
Release

idea
DEVELOPMENT OPERATIONS
‘Shift-Right’ Digital Ecology
‘Shift-Up’ Digital Assurance
‘Shift-Left’ Digital Evolution
CUSTOMER
EXPERIENCE TESTING
DesignDevTestOps
Agility across all 4 increases speed, reliability and efficiency
DESIGN
‘Shift-Down’ Digital Archaeology
Continuous
Adoption
Cognitive
Adaptive
Adoption
Continuous
Testing
Continuous
Improvement
Continuous
Delivery
Continuous
Intelligence
Continuous
Learning
Continuous
Innovation
Cognitive
Adaptive
Testing
Cognitive
Adaptive
Improvement
Cognitive
Adaptive
Delivery
Cognitive
Adaptive
Intelligence
Cognitive
Adaptive
Learning
Cognitive
Adaptive
Innovation

WATERFALL / V-MODELDAD, LeSS & SAFE (v4)LEAN STARTUP / UX
Incubate Scale
FAST
FEEDBACK
BUSINESS INTELLIGENCEREAL TIMEPREDICTIVEPERSCRIPTIVE
Enhance
SLOW
FEEDBACKWHAT IS HAPPENING NOW?
WHAT SHOULD WE DO?EXPERIMENT ASSURE

Digital Evolution feat. DesignOps
‘Shift Left’ Becoming an Digitally Disruptive Business
PORTFOLIO
MANAGEMENT
Easily and securely manage
from single location &
decrease creation time
by 96%
LIFECYCLE
VIRTUALIZATION
Simultaneously access
and virtual services
accelerating app
development by 97%
SCALED AGILE
MANAGEMENT
30–50% improvements
in both cost and delivery
duration compared to
traditional projects
SHIFT
LEFT
Innovation …. Faster

Rapid Evolution
Solution & Design Thinking, Lean Digital, Customer & User Experience (DX, CX & UX)
* Dave Landis, LitheSpeed.com, LeanUX principles and Design Thinking
© 2016 CA. ALL RIGHTS RESERVED.@CAWORLD #CAWORLD

Build
Deliver
MonitorMeasure
Learn
Design
Make
Check
Think
Proactive
over
Reactive
Governance
over
Ownership
Maturity
over
Mastery
Predictive
over
Continuous
Evolution
over
Revolution
Engineering
over Agility
Intelligence
over Things

INNOVATION (INNOVATION, GROWTH & MATURITY)
FAST
INNOVATION
SLOW
INNOVATION
DECOMPOSITIONLIBERATIONIMMUNIZATION
Incubate Scale Enhance
WHAT IS HAPPENING NOW?
Ecologist
DevScrum
Master
Therapist Researcher
Designer
DevOps
Engineer
Disciplined
Agilist
Maker
Creative
Infrastructure
Engineer
SDET
BA
Tester
Product
Owner PM
Ops
Support
FAST
FEEDBACK
SLOW
FEEDBACK
WHAT HAPPENED ?WHAT’S HAPPENING NOW? WHAT WILL HAPPEN ? WHAT COULD HAPPEN ?

Use Case - Agile Architect Designer
Wireframe
(DEMO)
Digital Evolution

Ideas
Beliefs,
Values,
Culture
Heuristics
Behaviors
Value
Agile
Lean
Copyright © 2016 nuCognitive LLC. All Rights Reserved.
Methods
and Skills
Solution Thinking
Exploring Solutions Thinking (Business, Technology & Usage)

Solution Thinking
Ray Arell
http://www.solution-thinking.com

LIFECYCLE
VIRTUALIZATION
Simulate missing systems,
to save up to 640 Hrs.
DIGITAL ENGINEERING
& AUTOMATION
Test less, cover more,
automate 100% of tests.
DATA VISUALIZATION
& VIRTUALIZATION
Reduce time spent
waiting on data by 95%.
SHIFT
UP
VALUE-DRIVEN
Multi-Modal Delivery
(Incubate to Lead, Scale for Growth and Enhance the Core)
Digital Assurance
‘Shift-Up’ Becoming an Value-Driven Business

COGNITIVE ADAPTIVE ADOPTION
Manual
Continuous
Autonomics
Lean
Engineering
Connected
Intelligence
BUSINESS VALUE
LEVEL 1 LEVEL 2 LEVEL 3 LEVEL 4 LEVEL 5
?
Optimized
Organization
for DesignOps
Continuous Insight
& Learning
Value Stream
Mapping
Consistency &
Collaboration
Heroes &
Heavy Lifting
DIGITAL MATURITY
§ Insight-Driven (Predictive / Prescriptive)
§ NoOps organized (IoT-Ops/CX-Ops/Sec-Ops)
§ Self-healing end-to-end autonomic orchestration
§ Microcontainerization & Microservices enabled
§ Open Innovation & Connected Intelligence
§ Quantum Teleportation (Shift X)
Where are you?
Minimum Viable Quality
§ Silos, manual handovers, waterfall process
§ One release/year
§ Monolithic apps
§ Long term project/resource planning
§ Error prone dev/test/release processes
Minimum Viable Product
Minimum Viable Delivery
Organization
Change
Methods
and Skills
Insight
Beliefs,
Values,
Culture
Value Behaviors Heuristics

Connected Intelligence
Release
Management
Environment
Management
Application
Management
Network
Management
Infrastructure
Management
Repository
Management
Data
Management
Test Data LearnProbe
CA App Experience Analytics
Cognitive Adaptive Insight (CAI)

REAL TIME, BIG DATA ANALYTICS EMBEDDED THROUGHOUT OUR PORTFOLIO
AGILE MANAGEMENT AGILE OPERATIONS DEVELOPER PRODUCTS
CONTINUOUS DELIVERY MAINFRAME SECURITY
API analyticsRelease management dashboard Access incident responseReal-time app performance
Cognitive Adaptive Insight (CAI)

Design QA/TESTDEV PRODUCTIONPRE-PROD
Specification
CI/Build Functional
testing UAT Integration
testing
Performance
engineering
Deploy to
pre-prod
Code
commit SCM
Design Spec
Requirements
Business
Users
TEST
Data
TEST
Stub
TEST
Data
User BA
Product
Manager
Customer
Users
Customer
Experience
Customer
Experience
Customer
Experience
Customer
Experience
Design
Business
Value
Business
Value
Business
Value
Improvement
Innovation
Intelligence
Assessment
Insight
Learning
Maintenance
Operations
Delivery
Testing
Support
Customer
Digital Assurance
Value Insight
Learn Learn
Monitor Monitor TEST
Stub
REAL
User
s
TEST
Data
REAL
Data
TEST
Stub
USER
Case
Business
idea
Value-Driven Delivery

Digital Assurance
Pocketbook
Paul Gerrard
NEW exclusive CA sponsored
Pocketbook authored by Paul
Gerrard, eminent Test-
Transformation consultant, thought
leader, teacher and international
conference speaker.
Pre-order your FREE copy today
from http://bit.ly/digital-assurance

Cognitive Adaptive Testing (CA-T)
Value-Driven Delivery (VDD)
Lean & Agile
Teams
Developers
& Testers
Release
Management
Product
Owner
Designer
Product
Manager
Daily
Reviews
Roadmap
Vision
Backlog
Sprint
Backlogs
Value
Driven
PLAN
Shippable
Products
SHIFT
LEFT
DESIGNOPS
SHIFT
RIGHT
OPERATE
Feedback
Loops

Step 1 – Continuous Adaptive Testing
In-Sprint everything
Sprint 1 Sprint 2 Sprint 3 Sprint N
Sprint 2 Sprint 3 Sprint N
Sprint 3 Sprint N
User Stories
Dev and
Manual
Testing
Automated
Regression
Testing
3 weeks 3 weeks 3 weeks 3 weeks
Sprint 1
Sprint 1 Sprint 2
Release Release Release Release
Highlights
§ No Sprint-lag effect
§ Release in 3 weeks
§ In-Sprint
Automation

So How to Achieve Speed, Quality and Coverage?
Pick an existing application and model requirements for 3 Sprints
Write
Code
and Test
Auto
Regression
Release
Model
Code
and Test
Auto
Regression
Release
Model
Code
and Test
Auto
Regression
Release
Model
Code
and Test
Auto
Regression
Release
User
Stories
Dev and
Exploratory
Manual
Testing
Automated
Regression
Testing
3 weeks 2 weeks 2 weeks 2 weeks
Automated
Release
Key Points
§ Sprint Acceleration
§ Do not go
Big-Bang
§ Start small, see
results, then scale
§ Scale gradually in
the enterprise

Step 2 – Lean Engineering
Pick an existing application and model requirements for 3 Sprints
Model
Code
and Test
Auto
Regression
Release
User Stories
Dev and
Exploratory
Manual Testing
Automated
Regression
Testing
2 weeks
Automated
Release
Sprint 1 Read User Story and
create basic Model
Automatically
generate basic
manual tests
Export them back
into your repository
(ALM, Rally, JIRA,
etc.)
Run test cycle as
usual
Sprint 2
Read User Story and
Create Model with
Negative and
Alternate Scenarios
Automatically
generate manual
tests with optimal
coverage
Export them back
into your repository
(ALM, Rally, JIRA,
etc.)
Run test cycle as
usual
Sprint 3
User Story is directly
Modeled. Model is
single source or
truth
Automatically
generate manual
and auto tests with
optimal coverage
Automatically
perform impact
analysis on any
changes
Run test cycle in
accelerated mode

Scaling Modeling Across the Enterprise
Example of a Roadmap across BUs
Business
Unit
Project M1 M2 M3 M4 M5 M6 M7 M8 M9 M10 M11 M12
BU 01
Project 1
Transition to Modeling completed after 3 Sprints and 1 extra month of
results/benefits analysis, review and final BU and PMO approval
Project 2 Modeling adoption completed in 2 Sprints
Project 3 Modeling adoption completed in 1 Sprint
BU 02
Project 1
Transition to Modeling completed, results/benefits
analyzed, reviewed and approved by end of Sprint 3
Project 2
Modeling adoption
completed still in 3 Sprints
Project 3

Scope
Mapping CD Value Streams to Organization
Level
Program /
Portfolio
(Cross Apps)
Project /
Product
(Cross Teams)
Sprint
(Single Team)
Value Stream
Continuous
Staging
and Release
Continuous
Testing
Continuous
Development
and Integration
Epic
Feature
01
US
01
US
02
US
03
Feature
02
US
01
US
02
US
03
Feature
03
US
01
US
02
US
03

Persona-Based Journey to Continuous Quality
Jedi Rank
Youngling
Padawan
Knight
Master
Business
Analyst
Writes Requirement
Creates Basic Model
Creates Model with
Negative and Alternate
Scenarios
Creates Complete,
Complex Models
Developer
Writes Code from
Requirements
Writes Tests First, then
Writes Code to Pass the
Test (TDD). Uses
Continuous Integration
Uses Service Virtualization
and Achieved
Continuous Testing
Looks at Everything as
Code (EAC) and Codes
Infra and Environment
Configurations. Achieved
Continuous Delivery
Data
Scientist
Reads Req.
Creates Lots of Data
Creates Only Needed Data
Based on Model-Specific
Paths
Auto-Generates Static
Data Based on the Model
Links TDM to the Model
for Automated
Provisioning of Dynamic
Synthetic Data
API
Engineer
Reads Req.
Manually Creates static
API Stubs
Creates Only Needed R/R
Pairs Based on Model-
Specific Paths
Auto-Generates R/R Pairs
Based on
the Model
Links Virtual Services to
the Model for Automated
Virtualization of APIs
Tester
Reads Req.
Creates Manual TC
Auto-Generates Manual
TCs from the Model with
Basic Coverage
Auto-Generates Manual
TCs from the Model with
Optimal Coverage
Links Test Automation
Keywords to the Model
for Automated Test Script
Generation
Operations
Engineer
Creates an Extensive
Runbook in a Word
Document and Works
Separately From the
Project Team
Drops the Word Doc and
Uses Tools to Manage
App and Environment
Config. Still Not Part of
the Project Team and
Does Manual Deployment
Enables Proper
Monitoring creates
Automated Build and
Deployment
Environments for Dev
Teams. Part of the Team
Focuses on Continuous
Delivery Improvements
while Developers Become
Responsible for the
Application in Production

Cognitive Adaptive Testing
Existing Manual
Test Cases
Dynamic Optimized –
Tests
Agile
Requirements
Designer
Unstructured
Partial coverage
Test bloat
Low effectiveness
Analog – difficult to maintain
Cognitive Adaptive -
Tests based on Insight
from Connected
Intelligence
New
Requirements
and Business
Processes
Fewest test steps (reduce by > 30%)
Greater coverage
Higher effectiveness
Fully digitized – easy to maintain
Continuous optimization
Highest effectiveness
Reduction > 50%
Release OperationsDesign Development

Modeling Maturity
AMBIVALENT
LEVEL 1
INITIALIZING
LEVEL 2
AWARE
LEVEL 3
MINDFUL
LEVEL 4
EXPERT
LEVEL 5
§ Test data is copied
from production
§ Extensive use of End
to End testing
§ Test data is created
manually by testers
§ PII in test data,
risking breach of
security
Challenges of Level 1 stay
with following
improvements:
§ Some databases are
subsetted for testing
purposed to speed
testing and reduce
storage costs
§ PII is masked in SOME
test databases
§ PII removed from test
data bases
Challenges of Level 2
reduce with following
definitive steps:
§ Test data is created
using
a tool
§ PII removed from test
data bases subsets
§ Test data is explicitly
drawn out in test plans
§ TDM is recognized
as a skill
Level 3 processes get
matured:
§ Test data can be
reserved
§ Test Data coverage is
understood and known
§ The right tests are run,
resulting in complete
testing
§ Test Data on Demand is
widespread.
§ Centralized TDM team
Everything in Level 4 +
§ Enterprise level
Modeling policy and
strategy fully adopted
by the organization
§ Common Enterprise
Model Repository is
leveraged used by all
teams
§ Models are the single
source of truth for
Biz/Dev/Test/Ops
§ Continuous
Development
Framework in place
Most IT
Organizations

> COGNITIVE ADAPTIVE <
RUNDEPLOYTESTBUILDPLAN
INTAKE
Requirements
User Stories
Release Plan
MODEL
Import User Stories to
Automatically Create,
Visualize and Optimize Tests
Initiate Functional, Non-
Functional and Security as
Code
FUNCTIONAL
§ Subset/Mask Test Data
§ Create/Reserve Test Data
§ Test Automation Library
§ Ensure Mobile Experience
INSIGHT-DRIVEN
CONFIG/DEPLOY
§ Provision Entire Stack
§ Confirm Configurations
§ Approve Changes
§ Successfully Deploy
§ Internal or External Cloud
INTEGRATION
Remove Constraints with
Virtual Services
§ Mobile, Web, App Server,
Middleware, Backend, MF
§ 3rd Party Systems / API’s
CODE
Develop and Commit Code,
Scan Code, Version Control,
Continuous Integration.
Complete Build and Initiate
Release
AUTOMATION AUTOMATION AUTOMATION
ü Speed/Time-to-Market
ü Quality/Availability
AUTOMATION
ü Cost/Financial Mix
ü Risk/Compliance
ü Requirements
ü Environments
ü Data
ü Automation
Customer
Experience
NON-FUNCTIONAL
§ Performance Engineering
§ Simulate Backend Load
§ Test Outlier Conditions
§ Ensure Mobile Experience
§ Security / Penetration
Common Goals “Big Picture” Issues
DESIGN SHIFT-LEFT SHIFT-RIGHT OPS
Node.Probe
Node.Test
TDD
BDD
ATDD
BPMN
Nimbus
Visio
Node.Data
Node.Learn
MEASURE/FEEDBACK
§ Customer Experience
§ Business Service View
§ Application View
§ Infrastructure View
§ Dynamic Capacity
§ Feedback loop

DESIGNOPS
VALUE-DRIVEN DELIVERY
Cognitive Adaptive Adoption
1
Cognitive
Adaptive
Learning
2
Cognitive
Adaptive
Testing
3
Cognitive
Adaptive
Technology
4
Cognitive
Adaptive
Delivery

Cognitive Adaptive Adoption (CA-A)
The future is Value-Driven Delivery
CA Project & Portfolio
Management
Unify long term strategy,
investment and portfolio
planning.
CA Agile Central
Collaboratively plan,
prioritize and track work
across the enterprise.
Develop & Test
Synchronize efforts to
dramatically speed app
development & increase
quality
Manage & Monitor
Make a great customer
experience a competitive
advantage
Release & Deploy
Control the release process,
to continuously advance
application quality, improve
the customer experience
and reduce costs.
Minimum
Viable
Delivery
Minimum
Viable
Experience
Minimum
Viable
Quality
(Minimum Viable-Value-Validated)
Executives
BUSINESS INITIATIVE
Complexity-
Informed
Organizational
Change
The Digital Ecology
Minimum
Viable
Product
CUSTOMER
THINK
MAKE

Q & A

What next?

Recommended Sessions
SESSION # TITLE DATE/TIME
DO5X29S
Model-Based Testing for Achieving Maximum CD Pipeline
Acceleration
11/16/2016 at 3:00 pm
DO5X47S 'Shift-Right' - Rapid Evolution with DesignOps 11/16/2016 at 3.45 pm
DO5X40S Case Study: Why Rabobank Was Inspired 11/17/2016 at 3.45 pm

Must See Demos
Use Model
Based Testing
CA ARD
DevOps Theatre 5
DevOps 5 Theatre
Achieve Max
Test Coverage
CA ARD
DevOps Theatre 5
Improve Data
Compliance
CA Test Data
Manager
DevOps Theatre 5
Deliver Test
Data Faster
CA Test Data
Manager
DevOps Theatre 5

Stay connected at communities.ca.com
Thank you.

Director of Digital Assurance
jonathon.wright@ca.com
Jonathon Wright
@Jonathon_Wright
SlideShare.net/Jonathon_Wright
LinkedIn.com/in/Automation
https://www.ca.com/us/trials/ca-agile-requirements-
designer.register.html

How to Solve Problems of Testing in Agile Environments

Recomendados

Recomendados

Mais conteúdo relacionado

Mais procurados

Mais procurados (20)

Destaque

Destaque (10)

Semelhante a How to Solve Problems of Testing in Agile Environments

Semelhante a How to Solve Problems of Testing in Agile Environments (20)

Mais de CA Technologies

Mais de CA Technologies (20)

Último

Último (20)

How to Solve Problems of Testing in Agile Environments