De Montfort University - IMAT5209: Usability Evaluation Report

This report presents a comprehensive usability evaluation of an interactive e-commerce system. It begins by defining interactive systems and their users, focusing on an e-commerce website as a primary example. The report outlines various use cases, such as registration, login, product browsing, and checkout, and then delves into the key usability requirements, including privacy policies, efficiency, user engagement, flow, and error tolerance. The evaluation methodology employed is cognitive walkthrough, detailing the selection of users, definition of system goals, identification of user paths, and the use of questionnaires. The evaluation section describes the experiences of managers, operators, and customers within the system, highlighting their tasks, successes, and challenges, along with violations of design guidelines. The findings provide insights into the system's strengths and weaknesses, offering a practical assessment of the e-commerce website's usability.

Interactive system 1
USABILITY EVALUATION OF AN INTERACTIVE SYSTEM
By
(Name)
(Course)
(Professor’s Name)
(Institution)
(State)
(Date)

Interactive system 2
Part ONE: The interactive system and its users: E-commerce website
An interactive system is defined as a computer system that interacts with people. Some of the
examples include; government and e-commerce websites. This paper has focused on e-commerce
websites. E-commerce is the practice of carrying out business transactions online over the
internet (Hussain and Mkpojiogu, 2016, p.1). The increase in the number of internet users is one
of the factors that has contributed vastly to e-commerce. There are many users of e-commerce
systems not limited to customers, managers and operators (Zhang, Wang and Zhang, 2012, p.2).
The managers view transaction reports from an e-commerce system, hence enabling them to
make some important decisions. On the other hand, customers view products that are displayed
on the website (Lee and Koubek, 2010, p.331). They can also select a particular product and add
it on the cart. The operators are responsible for updating, removing and editing product’s details.
They also ensure that operations are running smoothly. Currently, the number of e-commerce
users in the United Kingdom is approximately 45 million.
Part TWO: The use cases
Although there are several use cases in an e-commerce system, but this paper has focused on
buying of products only.
Register: The new customer registers into the system by entering his/her details such as
username, first name, second name, password, repeat password and country.
Login: After successfully registering into the system, the customer is directed to the login
page where he is required to provide his username and password.
Browse products: The customer views different products that have been displayed in the
website (Ullah, Alauddin and Zaman, 2016, p.3).

Interactive system 3
Search product: Here, the customer searchers for a specific product that he/she is interested
to buy.
Add product to shopping cart: The customer adds a particular product that he/she is
interested to buy to the shopping cart.
Checkout: This step is concerned with payment option where the credit card is validated
(Goh et al., 2013, p.145).
Part THREE: The usability requirements
The following are usability requirements of an e-commerce system:
 Privacy policies: An e-commerce website should enhance privacy of the highest level.
Therefore, important customer’s credentials such as e-mail should not be accessed by
third parties. Apart from e-mail, the system should integrate end-to-end encryption
mechanisms, hence customer’s message cannot be accessed by other customers. The
system should not allow simple password to be used by customers. It should allow a
minimum of 10 characters so that it can prevent brute-force attacks (Sivaji et al., 2011,
p.222).
 Efficiency: The system should be able to run efficiently without errors. The transactions
should take less than 10 minutes to be completed. In contrast, the system is regarded to be
ineffective when the transaction has taken more than 20 minutes. Still on efficiency, the
customers should be able to reach to the desired state or achieve their targets in an easy
way (Chelule, Herselman and Greunen, 2010).
 User engagement: The system should be able to interact with its users. Hence, the
questions asked by customers should be answered on a real-time basis. This can be
achieved by integrating LiveChat software. The system should be able to collect and

Interactive system 4
analyze customer’s views automatically, hence necessary steps are taken by maintenance
team to keep on improving it to increase customer satisfaction (Mitrevski and Hristoski,
2014, p.321).
 Flow: The business transactions should be completed successfully without unnecessary
delays. Different processes should run simultaneously without interfering with each
other. The system should also be consistent, hence be able to update transactions and
processes when the need arises. It should be able to maintain the same data in case of
decentralized databases (Travis, 2017).
 Error tolerance: An e-commerce system should be error-free. Hence, all transactions
should be processed correctly. Errors can lead to unnecessary disruptions and also lose of
money on the side of customers as well the company (Travis, 2017).
Part FOUR: The evaluation methodology
An e-commerce system is evaluated by systematic evaluation method whereby cognitive
walkthrough is used. This method focuses on how new system users carry out their tasks to
achieve specified goals. It also identifies some of the problems that users encounter while
carrying out different tasks. It consists of the following steps:
 Users’ selection. Here, the people who will evaluate an e-commerce system are
selected. There is a certain criteria that is used for selection. For instance, sampling
methods such as simple random sampling can be used. The following are some of the
questions that can be used to select users:
i. Are you familiar with some of the functionalities of an e-commerce system?
ii. Do you understand how transactions are carried out in an e-commerce
system?

Interactive system 5
iii. Have you ever interacted with an e-commerce system? (Hussain, Mkpojiogu
and Kamal, 2016, p.30).
 Define goals of the system users: In this step, the goals that users’ should achieve
are stated in advance. Some of the examples of the goals include; logging in and
registering to the system. However, there are several things that are considered while
defining goals. For instance, the product and services offered by the company are
compared to other related companies. Besides, some of the benefits that are
associated with given products is equally important (Paz and Pow-Sang, 2016, p.166).
 Identify the paths: This entail some tasks that a user is required to perform before
he/she achieves a particular goal (Huang, 2011, p.399). For example, the following
actions should be performed by a new customer to register and login to the e-
commerce system.
 Navigate to home page.
 Click register button.
 Enter your first name and click next.
 Enter your second name and click next.
 Enter your username and click next.
 Enter your password and click next.
 Re-enter your password and click next.
 Click ‘register’ button.
 Navigate to the login page.
 Enter you username and click next.
 Enter your password and click next.

Interactive system 6
 Click login button (Travis, 2017).
 Asking questions: The system users such as customer and managers are asked questions
related to how tasks are performed by an e-commerce system. The questions are intended
to identify some of the usability issues that are related to an e-commerce systems that is
being evaluated. The following are some of the questions that users should be asked:
o Have you been able to follow the correct path as specified?
o Have you managed to add products to the shopping cart? (Hasan, Morris and
Probets, 2012, p.708)
Part Five: The evaluation
First and foremost, the managers tried their level best to view business reports. They filled
required details such as first name, second name, email, user type, user ID, password and retype
password. In user type, the managers were required to select one of the following options;
management or operator. Therefore, they selected management option (Li and Li, 2011, p.5299).
They also specified management category such as finance and strategic management. After
entering required details, they clicked next. Having filled all the necessary details, they clicked
register button. The managers who filled all the necessary details navigated to the login page
where they entered their user ID and password. Subsequently, they clicked login button. Having
logged in successfully, they were directed to the management page where they clicked view
reports button (Hasan, Morris and Probets, 2013, p.231). Different managers were directed to
different pages when they clicked login button. For instance, sales managers were able to view
the total number of sales by clicking the ‘view sales’ button. On the other hand, operational
managers viewed reports related to manufacturing.

Interactive system 7
However, some of the managers failed to register successfully as they provided invalid email
addresses. Most of the managers who did such mistake forgot to include @gmail.com, hence
violating design guidelines. Regarding performance, more than 95% of the managers completed
a particular transaction in less than 10 minutes. This shows that the e-commerce system was
efficient (Huang and Benyoucef, 2013, p.247).
Apart from managers, the operators of the system also underwent registration and login process.
However, they were required to fill different details as compared to managers. They navigated to
management home page where they clicked register button. They entered the following details;
first name, second name, email, user type, user ID, password and retype password. In contrast to
managers, they selected operator under user type. They also specified different operator types.
Then, they navigated to the login page after a successful registration process. The system
operators performed various tasks such as inserting new product’s details, editing and deleting
existing products (Knijnenburg et al., 2012, p.445).
Regarding inserting of a new product, they clicked add product button. They filled the following
product’s details; name of the product, product ID, price and features. Apart from adding new
products, they also edited existing products. This was achieved by using product ID. They
clicked update product button where they were requested to enter the product ID. Then, they
clicked submit button and the system displayed existing information about the product from the
database. They edited product’s information such as price and clicked submit button. They
received a message that they have updated a given product successfully (Pu, Chen and Hu, 2011,
p.162). However, some of the system operators were unable to update product’s details. This is
because they were unable to enter the correct product’s ID.

Interactive system 8
In addition to product’s update, system operators also deleted existing products from the
database. They achieved this by logging onto their account and clicked delete product button.
They were directed to product’s delete page where they inserted the product ID and clicked
submit button. However, some of them were asked to provide a valid product ID in order to
continue with the process. The operators also cancelled orders as requested by customers through
live chat or telephone. They achieved this by entering the order ID and clicking submit button.
However, not all operators managed to carry out their tasks correctly. The number of those who
managed to add, edit and delete a product in less than 10minutes was estimated to 93%. Those
who took more than 20 minutes failed to identify a place where they can access recent features
and functionalities, hence they took more time to go back to the home page before proceeding to
the other pages (Aboutorabi et al., 2015, p.2). On the other hand, operators who failed to carry
out these tasks completely were unable to retrieve order and product IDs from the database.
Regarding violation of design guidelines, some of the system operators failed to enter product’s
price in two decimal places.
Last but not list, the customers also managed to register to the e-commerce system. However,
they provided different information as compared to managers and the operators (Maity and Dass,
2014, p.35). They also registered in a different page. They navigated to the customer registration
page and clicked register button. They entered the following details; first name, last name,
customer ID, country, state, address, email, password and re-type password. Having filled all the
necessary details, they clicked submit button and were directed to the login page where they
entered their user ID and password. After successful login, they clicked view products button
where they were directed on a different page. They were able to view products that were
displayed (Lin et al., 2011, p.1984).

Interactive system 9
Other than viewing displayed products, they were also able to search for a given product.
However, product’s name is used in searching rather than the product ID. Subsequently, they
clicked submit button after inserting product’s name. The system returned findings of the
searches showing specific products that the customer requested. The customers ordered for the
products that they were interested to purchase. They added such products to the shopping cart.
Afterwards, they viewed products that they added to the shopping cart. There were some cases
that the customers wanted to cancel specific orders (Zhang, Wang and Zhang, 2011, p.501).
They achieved this by messaging the system operators via the live chat. Other than purchasing
products, customers also asked questions related to products and services that are offered by the
company. They clicked the live chat that was found in every page and started the chat by texting
the word hello. Then, the available operator provides a response to some of the questions asked
on a real-time basis (Peng and Lai, 2014, p.38).
Regarding performance, 87% of the customers managed to add products to the shopping cart in
less than 5 minutes. In contrast, those who took more time lacked idea of searching specific
products, hence they used a long way to find the product that they were interested to purchase.
Also, some of them were not very familiar with the system’s interface, hence took more time to
understand the correct path that they can follow to reach the desired goals.
Part SIX: The findings of the evaluation
In conclusion, the following are some of the ways in which the findings relate to the results of
the evaluation procedure:
 The goals that were defined in the evaluation procedure were attained. For instance, most
of the system users such as managers logged onto the system successfully. Successful

Interactive system 10
login involved the registration process. The managers entered the following information
to register to the e-commerce system; first name, second name, email, user type, user ID,
password and retype password. After successful registration, they clicked login page
where they inserted their user ID and password (Travis, 2017).
Other than the goals, the process of the identifying the path which was described in the
evaluation procedure was also evident. For instance, the customers managed to follow all
the necessary steps to add a product to the cart, for example, register, login, view
products, search products and add product to the shopping cart. Registration was
achieved by providing the following details; first name, last name, customer ID, country,
state, address, email, password and re-type password. After registration, they logged onto
the system by entering user ID and password. After login, they clicked view products
button and other processes such as searching for the product basing on identified path till
they achieved their goals (Travis, 2017).
However, the following usability problems could affect smooth running of day-to-day business
activities:
o Failure of the system operators to retrieve order and product IDs from the database.
Basing on the evaluation results, some of the system operators were not familiar with the
system functionalities, hence unable to retrieve some data from the database such as
product IDs. This prevented system operators from carrying out some major tasks such as
updating and deleting products. Also, some were inserting the wrong product IDs. Thus,
this will increase the number of errors in the system such as miscalculations (Travis,
2017).

Interactive system 11
o Some of the system operators were unable to locate recent pages and features that they
were able to access. This made them waste a lot of time going back to the home page
before proceeding to other pages. This increases time consuming, hence lowering
productivity.
o Some of the customers were unfamiliar with the system’s interface. Hence, this can lower
their morale to use the system since it is somehow complex rather than user-friendly
(Chelule, Herselman and Greunen, 2010).
The following are some of the strengths of evaluation process:
 The evaluation process involved all major users of an e-commerce system such as
managers, customers and system operators.
 Inexperienced users participated in the system evaluation process.
However, the evaluation process has some weaknesses. The following are some of them:
 The evaluation process did not involve the disadvantaged members of the society such as
the old age.
 Gender biasness was evident as the number of male participants surpassed that of female
participants by more than half (Köpcke, Thor and Rahm, 2010, p. 485).
The following are the successes of the e-commerce system evaluation process:
The evaluation process followed steps that were outlined in the evaluation methodology.
However, the following are some of its failures:
The evaluation process is not well designed (Chelule, Herselman and Greunen, 2010).

Interactive system 12
References
Aboutorabi, S.H., Rezapour, M., Moradi, M. and Ghadiri, N. (2015, August). Performance
evaluation of SQL and MongoDB databases for big e-commerce data. In 2015 International
Symposium on Computer Science and Software Engineering (CSSE) (pp. 1-7). IEEE.
Chelule, E., Herselman, M. and Greunen, D.V. (2010). E-commerce usability: do we need
guidelines for emerging economies? Available from:
https://researchspace.csir.co.za/dspace/handle/10204/4763 [Accessed 2 May 2019].
Goh, K.N., Chen, Y.Y., Lai, F.W., Daud, S.C., Sivaji, A. and Soo, S.T. (2013, April). A
comparison of usability testing methods for an e-commerce website: A case study on a Malaysia
online gift shop. In 2013 10th International Conference on Information Technology: New
Generations (pp. 143-150). IEEE.
Hasan, L., Morris, A. and Probets, S. (2012). A comparison of usability evaluation methods for
evaluating e-commerce websites. Behaviour & Information Technology, 31(7), pp.707-737.
Hasan, L., Morris, A. and Probets, S. (2013). E-commerce websites for developing countries–a
usability evaluation framework. Online Information Review, 37(2), pp.231-251.
Huang, Z. and Benyoucef, M. (2013). From e-commerce to social commerce: A close look at
design features. Electronic Commerce Research and Applications, 12(4), pp.246-259.

Interactive system 13
Huang, S.L. (2011). Designing utility-based recommender systems for e-commerce: Evaluation
of preference-elicitation methods. Electronic Commerce Research and Applications, 10(4),
pp.398-407
Hussain, A., Mkpojiogu, E.O. and Kamal, F.M. (2016). A systematic review on usability
evaluation methods for m-commerce apps. Journal of Telecommunication, Electronic and
Computer Engineering (JTEC), 8(10), pp.29-34.
Hussain, A. and Mkpojiogu, E.O. (2016, August). Usability evaluation techniques in mobile
commerce applications: a systematic review. In AIP Conference Proceedings (Vol. 1761, No. 1,
p. 020049). AIP Publishing.
Knijnenburg, B.P., Willemsen, M.C., Gantner, Z., Soncu, H. and Newell, C. (2012). Explaining
the user experience of recommender systems. User Modeling and User-Adapted
Interaction, 22(4-5), pp.441-504.
Köpcke, H., Thor, A. and Rahm, E. (2010). Evaluation of entity resolution approaches on real-
world match problems. Proceedings of the VLDB Endowment, 3(1-2), pp.484-493.
Lee, S. and Koubek, R.J. (2010). The effects of usability and web design attributes on user
preference for e-commerce web sites. Computers in Industry, 61(4), pp.329-341.
Li, F. and Li, Y. (2011). Usability evaluation of e-commerce on B2C websites in
China. Procedia Engineering, 15, pp.5299-5304.
Lin, C., Lin, H.C.K., Huang, Y.A. and Jalleh, G. (2011). The fit between organizational B2B e-
commerce policy, IT maturity and evaluation practices on B2B e-commerce performance in

Interactive system 14
Australian healthcare organizations. African Journal of Business Management, 5(5), pp.1983-
2005.
Maity, M. and Dass, M. (2014). Consumer decision-making across modern and traditional
channels: E-commerce, m-commerce, in-store. Decision Support Systems, 61, pp.34-46.
Mitrevski, P.J. and Hristoski, I.S. (2014). Behavioral-based performability modeling and
evaluation of e-commerce systems. Electronic Commerce Research and Applications, 13(5),
pp.320-340.
Paz, F. and Pow-Sang, J.A. (2016). A systematic mapping review of usability evaluation
methods for software development process. International Journal of Software Engineering and
Its Applications, 10(1), pp.165-178. Available from:
https://pdfs.semanticscholar.org/1892/1b0cf7f3d4b0340baf16a9b107e477ef505f.pdf [Accessed 2
May 2019].
Pu, P., Chen, L. and Hu, R. (2011, October). A user-centric evaluation framework for
recommender systems. In Proceedings of the fifth ACM conference on Recommender
systems (pp. 157-164). Association for Computing Machinery.
Peng, L. and Lai, L. (2014). A service innovation evaluation framework for tourism e-commerce
in China based on BP neural network. Electronic Markets, 24(1), pp.37-46.
Sivaji, A., Downe, A.G., Mazlan, M.F., Soo, S.T. and Abdullah, A. (2011, June). Importance of
incorporating fundamental usability with social & trust elements for e-commerce website.
In 2011 International Conference on Business, Engineering and Industrial Applications (pp.
221-226). IEEE.

Interactive system 15
Travis, D. (2017). E-commerce usability: tools and techniques to perfect the on-line experience.
CRC Press. Available from:
https://www.crcpress.com/E-Commerce-Usability-Tools-and-Techniques-to-Perfect-the-On-
Line-Experience/Travis/p/book/9780415258340 [Accessed 2 May 2019].
Ullah, S.E., Alauddin, T. and Zaman, H.U. (2016, January). Developing an E-commerce website.
In 2016 International Conference on Microelectronics, Computing and Communications
(MicroCom) (pp. 1-4). IEEE.
Zhang, H., Wang, Y. and Zhang, X. (2011, July). Transaction similarity-based contextual trust
evaluation in e-commerce and e-service environments. In 2011 IEEE International Conference
on Web Services (pp. 500-507). IEEE.
Zhang, H., Wang, Y. and Zhang, X. (2012, December). A trust vector approach to transaction
context-aware trust evaluation in e-commerce and e-service environments. In 2012 Fifth IEEE
International Conference on Service-Oriented Computing and Applications (SOCA) (pp. 1-8).
IEEE.