Tuesday, August 3, 2021

Risk Based Automation driven Testing for Agile





                                 Risk Based Automation driven Testing for Agile





Introduction:

Risk Based Automation driven Testing in Agile is a new testing methodology that can revolutionize the world of application and services testing. As organizations try to cope with a dynamic, demanding marketplace and the pressure to deliver high quality product in small time frame is ever-increasing. When it comes to technology, organizations are realizing the importance of software testing and the role it plays in developing quality code with speed and agility. Today we face several Quality engineering challenges – time constraints in test development and test cycles, executing large volumes of test cases, testing diverse legacy applications so now it is critical that we choose the right approach for test automation.

This paper highlights the new approach of test development and execution using continuous integration and deployment keeping in mind the impact and probability of business scenarios failures.



Risk Based Automation driven Test Strategy

Let us focus on the key test automation development practices


  • Test-Driven Development
  • Create failed automated test during feature development.
  • Run the test methods to make test pass after feature development.
  • Schedule the automation suites to run daily
  • Monitor and Refactor tests due to code change.
  • Continuous Integration : Automated build and test processes are initiated as code committed to the version control system
  • Database Design and Development: Based on continuous incremental definition and regular migration of test data




Risk-Based Testing is a universal approach used to determine the right and optimal set of test cases that should be executed while assuring maximum coverage. We would like to propose this approach in test automation execution and getting full confidence before releasing features of the product.
Almost all the organization have a thought of 3:1 ratio of Quality Engineer ratio as compared to Dev Engineer even though they have had moved to agile and until individual engineer mindset in a scrum totally change to one team scrum mindset and come to a common thought process that quality is everyone responsibility at each and every level things are not going to change. Risk based Test Automation driven testing approach will help the engineer working in agile as no software can be tested exhaustively, it must be tested selectively  but on what basis selection should be done in order to maximize test effectiveness .

What is Risk-Based Testing?

Risk-based testing is an approach to decide what to test and test the right thing by using risk as a way to consider the cost of not testing.  To prioritize what to test and to test the most important areas becomes important. By identifying and prioritizing risks, we can make sure that the limited time and resources are used to test the most important things.


Different Approaches:


  •   Bach: Heuristic Risk-Based Testing
  •   Besson: A Strategy for Risk-Based Testing
  •   Chen: An approach for risk-based regression testing
  •   Gerrard: Risk-Based E-Business Testing
  •   Schaefer: Prioritizing Test against Deadlines
  •   StÃ¥lhane: Finding failure modes
  •   HazOp: Used to prioritize functions



Everyone above has focused on prioritizing the tests and given importance of validating in the order of risky tests but I like the way how Schaefer did Prioritizing Test against Deadlines.

Schaefer: Prioritizing Test against Deadlines

Schaefer focus on prioritizing what to test, by finding the most important and worst parts of the product. The most important parts of the system are found using factors like cost of failure, most visible and most used parts of the product. The worst parts of the system are found using defect generators like complexity, changed areas, new technology, new solutions, new methods, new tools, number of people involved, where there was time pressure and local factors. Weights are assigned for each relevant cost factor and defect generator. For each part of the system, values are assigned for the factors and the defect generators. Higher values mean that the area is more important or worse. These values are multiplied by the weights and added together. The highest values give the most risky parts and should be prioritized.



How to Apply this Approach in Automation?

Those times are gone when Quality engineers automate the tests after feature validation for regression and smoke testing and when the product are stable. Now In agile world test automation comes first and automated tests drive the testing and it would be part of every sprint deliverables.  Identify the test methods which you are planning to automate considering above factors (Business Criticality, Visibility to users, Complexity and frequency of change),prioritize it as per your convention either P0, P1, P2…. or provide some ranking and run these automated tests on daily basis and make it  part of Continuous Integration and analyze the results.



 How to apply this approach if you don’t know anything about the product?

This approach is applicable only when you know the Product well. If you don’t know the product then you have to estimate the tasks in such way that you can spend time in learning the feature of the product, Automate it and run the automated tests all in the same sprint or Learning the features in one sprint and Automation and testing in another sprint.


Continuous Integration and Deployment

Continuous Integration (CI) is a practice which evolved out of the need to avoid defects introduced into a system due to the integration problems associated with infrequent monolithic source branch check ins. Deployment Pipeline - the next evolution of continuous integration
The deployment pipeline has simply taken the next logical step by extending the scope of Continuous Integration concept to the entire delivery process.


How to achieve Continuous Deployment?


  •  Automate, automate  and automate
  •  Frequent check-ins
  •  Rapid Feedback
  •  Quick reaction times to feedback - traceability



The Goal of CI is to provide rapid feedback and Feedback should be
 

  •  Fast and Early
  •  Visible and accessible
  •  Actionable


Teams should develop rules for handling various types of feedback and be held accountable to those rules.
 

  • Broken build => Do not check in, stop and correct
  • Lock the perforce – no checkins
  • Drop in code Coverage : Do not introduce new changes until test code coverage exceeds threshold!
  • BATS/Regressions failing :  Fix the code/test during expected TTR




Benefits of this approach:

• Gain confidence level  and Precision of Effectiveness
• Help in prioritizing testing efforts
• Lower the risk in the shortest period of time in order to release versions more aggressively.
• Prioritize deliveries based on readiness, predict successful deliveries well in advance
• De-scope partial requirements rather than complete release based on accuracy of status.
• Deliver agreed requirements without fear of production failures and QA assurances.
• Focus completing testing on time or before time if the need be through prioritization



Conclusion:

The idea is to pinpoint the essential function of the change, determine the scope for testing in and around that change, analyze the associated risks, then plan and prioritise your testing accordingly. So, you start by making sure the change is fit for purpose and, just as importantly, does no harm. This is non-negotiable, and you can’t sign off without giving this assurance. That should be followed by smart, correctly prioritized testing of the areas which are next most likely to cause problems, as determined by your intimate knowledge of the system under test.  The real key of risk based automation testing properly is strong knowledge of your systems and architecture, and a good understanding of the paths your users will take. You should be familiar with all the touch points around the functions, how they interact with each another as well as other applications in your overall architecture, and all the paths the user will take into, through, and out of your area. This will allow you to easily identify and isolate the important areas for any given change, and allow you to test just around those areas.