Software Reliability Testing: Analysis of Principles, Metrics and Plan

This report provides a comprehensive overview of software reliability testing, starting with the core principles of reliability in software engineering, including failure causes, wear out, and redundancy. It discusses the challenges in reliability testing, such as unidentified bugs and the need for reliability growth prediction. The report details methods for measuring reliability, including mathematical modeling and various reliability measurement processes like inter-rater reliability and test-retest reliability. It outlines the steps to develop a test plan, emphasizing product analysis, test strategy design, and resource planning. Furthermore, it includes a discussion on test cases, providing examples and highlighting the importance of clear descriptions and expected results. The report concludes by differentiating normal verification testing from reliability testing, underscoring the importance of proper management and planning in reliability testing to ensure the development of robust and failure-free software.

Running head: SOFTWARE RELIABILITY TESTING
SOFTWARE RELIABILITY TESTING
Name of the Student
Name of the University
Author Note

SOFTWARE RELIABILITY TESTING 1
Abstract:
The purpose of this study to discuss about the reliability in software engineering. Presently a
very important role is being played by the computers I n our daily life and high quality
software is always needed there Although, to test a software completely is not possible; but
proper testing and sufficient testing will improve the reliability of a software to great extent.

2SOFTWARE RELIABILITY TESTING
Table of Contents
Introduction:...................................................................................................................2
1 Principles of reliability in software engineering:.....................................................3
2 Issues and challenges:..............................................................................................4
3 Measuring reliability:...............................................................................................5
4 Reliability Measuring Process:................................................................................6
5 Steps to Develop a Test plan:..................................................................................8
6 Test case:................................................................................................................12
7 Example test plan:..................................................................................................15
Scenario:...................................................................................................................15
8 Determine reliability of system:............................................................................16
9 Normal testing for verification vs. reliability testing:............................................17
Conclusion:..................................................................................................................17
References:...................................................................................................................18

3SOFTWARE RELIABILITY TESTING
Introduction:
Reliability testing in software system testing is a technique for testing, which relates
to test the ability of a software to the function of a consistently given environmental
condition, which help the uncover issues in software functionality and design. The purpose if
this paper is to describe about the reliability in software testing. At present the reliability,
testing is very important as the systems are getting fast day-by-day and the system requires
failure free software. The paper will discuss about the principles of reliability, the challenges
and also it will discuss about the test plan with a suitable example. There are several steps in
the testing phase, the paper will also discuss about that. This paper will try to give an idea
about reliability of software and the models and metrics used for that. The paper will also
focus on how the reliability of a software can be improved.
1 Principles of reliability in software engineering:
Reliability in software engineering can be defined as, in a specific period within a
specified environment, probability of failure-less operation of a software. The mechanical
and electronic parts can become old with their usage and time but software will not wear out
or rust in its life cycle. The software will remain same unless the intentional change or up
gradation (Wohlin et al., 2013). Reliability is a part of the both non-functional and functional
testing such as load, stress, security and network testing. The operational profile is able to
help the guide managerial as well as the engineering decisions throughout life cycle of the
product, including requirements of the specification, design, testing and
implication.Example- Mobile software. The core principles of reliability in software
engineering are as follows:
 Failure cause
 Wear out

4SOFTWARE RELIABILITY TESTING
 Repairable system concept
 Time dependency and life cycle
 Environmental factors
 Reliability prediction
 Redundancy
 Interfaces
 Failure rate motivators
 Built with standard components.
The activity diagram for the reliability testing is given below:

5SOFTWARE RELIABILITY TESTING
Fig: Activity diagram
2 Issues and challenges:
At present, for the reliability testing of a particular software, huge challenges can
come in software system process. In general, it is believed that at the reliability testing phase
of a software some unidentified bugs can come which is one of the biggest challenges faced
by software testing currently (A and K, 2013). The reason behind this is, after passing the
present reliability test of a software there is no guaranty that no error or bug will come in the
future. The current challenges and issues are as follows:
 Reliability growth prediction and assessment
 Reliability estimation based on failure free working
 Reliability claims based on process quality
 Fault tolerance based on design diversity
 Formal verification.
3 Measuring reliability:
The major issues involved at the time of measuring reliability of a software are as
follows:
 Estimate the maintenance process of the stability of the software.
 Estimate the test suit efficiency for detecting the residual faults.
 Forecast and estimate the exposure risk for residual fault in the system
software during the operation.
 Focus more accurately scare identification of fault resources in software.
Mathematical Modelling in Software Reliability:
m (n) is the expected number of faults removal by nth test case.

6SOFTWARE RELIABILITY TESTING
1. Exponential model: m(n+1)-m(n)=b(a-m(n))
2. Imperfect Debugging Model: m(n+1)-m(n)=b(ap-m(n))
3. Generalized model: b(n+1)= bp/ 1+ (1-r)(1-bp)n+1/r
4 Reliability Measuring Process:
Reliability is basically a measure of consistency of one method or metric. Every
method or metric that the tester uses, including methods to uncover the usability problems in
the interface as well as expert judgement, have to be assessed for the reliability (Juristo,
Moreno and Vegas, 2013).
Even before establishing the validity, the tester can establish the reliability. One of the
most reliable software model for reliability is The Jelinski Moranda model. Many other
existing models are extensions or variants of this model, which is the basic. The assumptions
that are exist in this model are:
 The program contains n number of the initial faults, which is a fixed constant
but unknown.
 The intervals of time between the failure occurrences are independent to each
other.
 A fault that is corresponding is removed confidently whenever any failure
occurs.
The rate of program failure at the interval of ith is:
λ (ti) = Ø [ N – (i-1) ], i=1,2,…..N
Where,
Ø = proportional constant
N = initial number of faults

7SOFTWARE RELIABILITY TESTING
ti = distance of time between (i-1)th and (i)th failures.
The most common four ways to measure reliability for an empirical metric or method
are:
 Inter rater Reliability: The extend in which the observers or raters make
response in same way to a phenomenon that is previously given, is a measure
of reliability. Examples:
o The interface problems is being identified by the evaluators.
o The severity of any problem, Experts rating.
 Test retest Reliability: It is the most important part in software testing that
the tester must have to enable the ability of retesting of the software system
(ArXiv, 2015).
 Internal consistence Reliability: This measures how consistent the
participants are in responding to a set of the items.
 Parallel Forms Reliability: It defines when same or similar type of results are
getting from the variation of slight on the evaluation method or question too
establishes reliability (Yamada and Osaki, 2014).

8SOFTWARE RELIABILITY TESTING
Figure: Reliability measurement process
5 Steps to Develop a Test plan:
A test plan can be defined as a detailed document which outlines testing objectives,
resources (software, manpower, hardware) required for the testing process, testing strategy,
testing estimation, test schedule and test deliverables (Amin, Grunske and Colman, 2013).
The most important part in software testing is test plan. One of the most referable paper about
operational profile development is from John Musa. It break downs and describes the system
into five profiles that are different from each other. The profiles are given in the picture
below.

9SOFTWARE RELIABILITY TESTING
Figure: SRE process
The benefits of test plan are as follows:
 Guides the thought of developers and relate that with the actual rules that need
to be followed.
 Help the people who are outside of the development and testing (such as
customers and business managers) to understand about testing.
 Helps the developers to assume an idea about how much effort they will have
to give.
 Some important aspects such as test scope, test estimation, test strategy are
involved in test plan. Therefore, the management can review the plan and can
reuse it for other projects (Porru et al., 2017).

10SOFTWARE RELIABILITY TESTING
To create a test plan there are eight steps. The steps are as follows:
1. Product analysing: To analyse the product
2. Design the strategy of testing: Expert have to design a strategy for testing
3. Define test objectives: The expert have to define the objectives for testing
4. Define the test criteria: The criteria that is needed for testing also have to be
define.
5. Resource planning: Planning for the needed resources
6. Test environment planning: Planning for the environment of the test
7. Schedule and estimation: Assume within how much time the test will be done.
8. Determine the test deliverables: What variables will be needed for the testing.
Fig: Steps Test Plan
Analyse the
product
Design test
strategy
Define test
objectives
Define test
criteria
Resource
planning
Plan test
environment
Schedule and
estimation
Determine
test
deliverables

11SOFTWARE RELIABILITY TESTING
Compared to the other testing types reliability is costly. So proper management and
planning is required at the time of doing reliability testing and this includes all the testing
processes such as test schedule, test environment etc. (Okamura, Dohi and Osaki, 2013).
To begin the reliability testing the tester has to follow these things:
 Establish the reliability goals
 Develop the operational profile
 Planning and executing tests
 Use the test results for driving decisions.
The reliability testing can be performed in three categories. They are - Measurement,
modelling and Improvement (Birolini, 2013).
The major parameters that are involved in the software reliability testing are:
 Probability of failure free operation
 Time length of failure free operation
 The environment in that it is executed.
Figure: Reliability

12SOFTWARE RELIABILITY TESTING
6 Test case:
Test case for reliability testing can be defined as a set of variables or conditions that
the tester will be able to determine whether, under the testing phase a system satisfies all the
requirements or not as the test case will also determine that the system is working correctly or
not (Fenton and Bieman, 2014). The entire process of reliability test cases development also
can help to find the problems related to design and requirements of a software system or an
application (Jin and Jin, 2014).
The aspects of test cases for a software are as follows:
 A strong title need to be used: A strong title is very important for starting a
good test case. It is better to give the name of the test case along the similar
line as module that has been chosen fir testing (Pai, 2013). As an example, if
the login page is chosen for testing then “Login Page” should be included as
the title in test case.
 Include a strong description: What the tester is going to test in an application
should be described properly in the description. Sometimes the description
section might include some other information like test data, test environment
and assumptions (Rana et al., 2013). The description should be easy for
reading.
 Keep the test steps concise and clear: It is very important to keep in mind
that the test case should be very simple and the person who will write the test
case and the person who will execute the test must be different (Smart, 2014).
 Include the Expected result: The expected result gives an idea to the tester
what they would experience as a test result from each steps. This is the way
the tester will determine that the teat case is a fail or pass.

13SOFTWARE RELIABILITY TESTING
 Make it Reusable: A good test case can be reusable and will also provide a
long-term value to the team who will test the software.
Examples:
1)
Test Case ID 001 Test Case
Description
Test the login function in university
Created By Lio Reviewed
By
James Version 2.1
QA Tester’s Log Review comments from Lio
incorporate in version 2.1
Tester's Name Lio Date Tested 1-Jan-2019 Test Case
(Pass/Fail/No
t Executed)
Pass
S # Prerequisites: S # Test Data
1 Access to Firfox Browser 1 Userid = LM541
2 2 Pass = 000000
3 3
4 4
Test
Scenari
o
Verify on entering valid userid and password,
the customer can login
Step # Step Details Expected
Results
Actual Results Pass / Fail / Not
executed /
Suspended
1 Navigate to
http://abcdefg.co
m
Site should
open
As Expected Pass
2 Enter the Userid
and Password
Credential
can be
entered
As Expected Pass
3 Click on Submit The
Customer is
logged in
As Expected Pass

14SOFTWARE RELIABILITY TESTING
2)
Test Case ID 002 Test Case
Description
Test the Login Functionality in school
Created By Salena Reviewed
By
Justin Version 2.23
QA Tester’s Log Review comments from
Salena incorporate in version
2.23
Tester's Name Salena Date Tested 1-Feb-2019 Test Case
(Pass/Fail/No
t Executed)
Pass
S # Prerequisites: S # Test Data
1 Access to safari Browser 1 Userid = LM542
2 2 Pass = 000000
3 3
4 4
Test
Scenari
o
Verify on entering valid userid and password,
the customer can login
Step # Step Details Expected
Results
Actual Results Pass / Fail / Not
executed /
Suspended
1 Navigate to
http://abcdefg.co
m
Site should
open
As Expected Pass
2 Enter the Userid
and Password
Credential
can be
entered
As Expected Pass
3 Click on Submit The student
is logged in
As Expected Pass

15SOFTWARE RELIABILITY TESTING
7 Example test plan:
For the test plan, BDD has been chosen. A BDD scenario can be defined as a written
description of the behaviour of a product from one or more perspective from the user
viewpoint. Software testing by using BDD scenario means the tests and requirements can be
a combination into one specification. In most of the cases, the scenario or description that is
written can be converted easily into the automate tests.
Scenario:
A scenario is basically represents particularly one functionality that is under testing.
The scenario will help the users to understand, what the testing is actually all about. Each of
the scenario must follow the GIVEN, WHEN AND THEN FORMAT. The language is
known as ‘gherkin’.
Feature: The feature of login functionality.
Scenario: Functionality of Login.
Given: The user will navigate to the SOFTWAREHELP.COM
When: The user will login using the username as ‘USER’
And: The password is ‘password’
Then: The login should be successful
And: The homepage must be displayed
Example of Background:
Background:
Given: The user logged in as the database administrator
And: All the junk files are cleared.

16SOFTWARE RELIABILITY TESTING
Reliability matrices:
Test case no. Test case Test steps Test data Expected result
1 John
Verify login
1) Go to login
page.
2) Enter userId
3) Enter
password
4) Click login
User id- john
Password- 000
Login
Successful
2 Potter User id- potter
Password- 1111
Login
Successful
3 Sam User id- Sam
Password- 222
Login
Successful
8 Determine reliability of system:
To determine the reliability of an entire system, the tester will gather all the
performance data from the entire system or from the subsystems (optional condition). The by
using the statistics for calculating the empirical reliability for the specified condition. Also by
building up a model that is mathematical based on reliability of the functions individually,
components, failure modes and the subsystems (Dees, Wynne, Hellesoy, 2013).
To ensure that the login functionality is working or not the cucumber test was running
for verifying if it is working or not.
Scenario Outline: Functionality of Login
Given: The user will navigate to the SOFTWAREHELP.COM

17SOFTWARE RELIABILITY TESTING
When: The user logs in using the (username) and (password)
Then: The login should be successful.
The examples have been used for passing different types of arguments in a tabular format.
Vertical pipes have been used for separating the two different columns.
9 Normal testing for verification vs. reliability testing:
In reliability, it refers to a degree to which the scale produces results that is consistant,
when the repeated measurements are done. However, in normal testing, the extent in which
research instruments are measured and what that is intended for measuring. There are three
types of test plan in normal testing. The Master test plan, it is a single test plan of high level
for a product which unifies all of the other test plans. Testing level that is Specific test plan
and another type of test plan is phase test plan which addresses typically a test phase. The
normal testing, the total test time may be reduced for the items that are highly reliable.
However, in reliability testing the total test time may be reduced whether the tester can run all
the tests sequentially. In reliability testing there has a better chance to pass the testing phase
than the normal testing.
Conclusion:
Presently a very important role is being played by the computers I n our daily life and
high quality software is always needed there. The most measureable aspect quality of
software is software reliability. There can be three categories in software reliability and they
are Modelling, Improvement and Measurement. Although, to test a software completely is
not possible; but proper testing and sufficient testing will improve the reliability of a software
to great extent.

18SOFTWARE RELIABILITY TESTING
References:
Katherine, A.V. and Alagarsamy, K., 2013. A fuzzy mathematical model for peformance
testing in cloud computing using user defined parameters. International Journal of Software
Engineering & Applications, 4(4), p.27.
Ismail, F.F., Razali, R. and Mansor, Z., 2019. Considerations for Cost Estimation of Software
Testing Outsourcing Projects. International Journal on Advanced Science, Engineering and
Information Technology, 9(1).
Wohlin, C., Runeson, P., Neto, P.A.D.M.S., Engström, E., do Carmo Machado, I. and De
Almeida, E.S., 2013. On the reliability of mapping studies in software engineering. Journal
of Systems and Software, 86(10), pp.2594-2610.
Modarres, M., Kaminskiy, M.P. and Krivtsov, V., 2016. Reliability engineering and risk
analysis: a practical guide. CRC press.
Özekici, S. ed., 2013. Reliability and maintenance of complex systems (Vol. 154). Springer
Science & Business Media.
Juristo, N. and Moreno, A.M., 2013. Basics of software engineering experimentation.
Springer Science & Business Media.
Spichkova, M., Liu, H., Laali, M. and Schmidt, H.W., 2015. Human factors in software
reliability engineering. arXiv preprint arXiv:1503.03584.
Yamada, S., 2014. Software reliability modeling: fundamentals and applications (Vol. 5).
Tokyo: Springer.
Amin, A., Grunske, L. and Colman, A., 2013. An approach to software reliability prediction
based on time series modeling. Journal of Systems and Software, 86(7), pp.1923-1932.

19SOFTWARE RELIABILITY TESTING
Porru, S., Pinna, A., Marchesi, M. and Tonelli, R., 2017, May. Blockchain-oriented software
engineering: challenges and new directions. In 2017 IEEE/ACM 39th International
Conference on Software Engineering Companion (ICSE-C)(pp. 169-171). IEEE.
Okamura, H., Dohi, T. and Osaki, S., 2013. Software reliability growth models with normal
failure time distributions. Reliability Engineering & System Safety, 116, pp.135-141.
Birolini, A., 2013. Reliability engineering: theory and practice. Springer Science & Business
Media.
Fenton, N. and Bieman, J., 2014. Software metrics: a rigorous and practical approach. CRC
press.
Jin, C. and Jin, S.W., 2014. Software reliability prediction model based on support vector
regression with improved estimation of distribution algorithms. Applied Soft Computing, 15,
pp.113-120.
Pai, G.J., 2013. A survey of software reliability models. arXiv preprint arXiv:1304.4539.
Fitzgerald, B. and Stol, K.J., 2014, June. Continuous software engineering and beyond: trends
and challenges. In Proceedings of the 1st International Workshop on Rapid Continuous
Software Engineering (pp. 1-9). ACM.
Rana, R., Staron, M., Mellegård, N., Berger, C., Hansson, J., Nilsson, M. and Törner, F.,
2013, June. Evaluation of standard reliability growth models in the context of automotive
software systems. In International Conference on Product Focused Software Process
Improvement (pp. 324-329). Springer, Berlin, Heidelberg.
Smart, J.F., 2014. BDD in Action. Manning Publications.
Dees, I., Wynne, M. and Hellesoy, A., 2013. Cucumber Recipes: Automate Anything with
BDD Tools and Techniques. Pragmatic Bookshelf.