Journal of Systems and Software PDF

The Journal of Systems and Software 137 (2018) 50–66
Contentslists available at ScienceDirect
The Journal of Systemsand Software
journal homepage:www.elsevier.com/locate/jss
Softwareengineeringproblems and their relationshipto perceived
learning and customersatisfactionon a software capstoneproject
Jari Vanhanena,∗
, Timo O.A. Lehtinenb, CasperLasseniusa
a Aalto University, Department of Computer Science, P.O. BOX 15400, FI-00076 AALTO, Finland
b Academy of Finland, P.O. BOX 131, FI-00531 Helsinki, Finland
a r t i c l e i n f o
Article history:
Received 20 June 2017
Accepted 5 November 2017
Available online 16 November 2017
Keywords:
Capstone project
Education
Learning
Customer satisfaction
Problems
Software engineering
a b s t r a c t
In educationalprojects,havingstudentsencounterproblemsis desirable,if it increaseslearning.However,
in capstoneprojects with industrial customers,negativeeffectsproblems can have on customersatisfac-
tion must be considered.We conducteda surveyin a capstoneproject course in order to study problems,
learningand customersatisfactionrelatedto elevensoftwareengineeringtopics.On the average,students
working in the managerialroles learned quite a lot about each topic, and the developerslearned moder-
ately,but the degreeof learning varied a lot among the teams,and among the team members.The most
extensivelyencounteredproblems were related to testing,task management,effort estimationand tech-
nology skills. The developerscontributedquite a lot to solving problems with technologyskills, but only
moderatelyor less with other topics, whereasthe managerscontributedquite a lot with most of the top-
ics. Contributingto solving problems increasedlearning moderatelyfor most of the topics. The increases
were highest with maintainingmotivationand technologyskills. Encounteringproblems with task man-
agement,customerexpectationsand customercommunicationaffectedcustomersatisfactionvery nega-
tively. When consideringboth learning and customersatisfaction,the best topics to encounterproblems
in were effort estimation,testing,and technologyskills.
© 2017The Authors. Published by ElsevierInc.
This is an open accessarticle under the CC BY license.(http://creativecommons.org/licenses/by/4.0/)
1. Introduction
In a capstoneproject, engineeringstudentssolve real-life prob-
lems in the context of a large, realistic project. As defined by
Fincher et al. (2001) the capstone project aims “to integrateand
consolidateacquiredconceptsand skills throughuse on projectwork”.
Capstoneprojects are commonly recommendedand used in engi-
neering education (ACM/IEEE CS, 2015; Pyster, 2009; Todd et al.,
1995).
We have organizeda capstoneproject course in software devel-
opment at Aalto University for over two decades.On the course,
teams of 7–9 students carry out real projects for real customers
mostly from industry during a six-month time period. The learn-
ing process includes 1) conducting the necessarysoftware engi-
neering tasks from understandingthe customer’sproblem to the
delivery of functional software, 2) getting feedback on the inter-
mediate and final results from the customer,and 3) reflecting on
the used work practiceswith the team’smentor.The used software
∗ Corresponding author.
E-mail addresses: [e-mail] (J. Vanhanen), [e-mail]
(C. Lassenius).
developmentprocess is iterative meaning that the working, feed-
back and reflection cycle is repeatedseveraltimes. Real customers
increase the realism of the projects and the student motivation.
Unexpectedchallengesprovide valuablelearning opportunitiesthat
can be discussedamong the teams.
In recent years,we have started to think about the relationship
betweenlearning and strugglingwith software engineeringrelated
problems in the projects. For example, does learning about re-
quirementsengineeringchallengesand practicessuffer if the cus-
tomer can express her needs very clearly, or if a team’s men-
tor helps the team avoid all typical pitfalls in requirementsengi-
neering. Industrial projects aim at maximizing customer satisfac-
tion, e.g.,by avoiding problems as much as possible, whereas ed-
ucational projects should focus on maximizing learning. Capstone
projects with industrial customersneed to focus on both learning
and customersatisfaction.In order to find dozens of real topic pro-
posals from industrial partners each year, the course must have a
good reputation among them. However, learning may be in con-
flict with customer satisfaction if problems increase learning but
decreasecustomer satisfaction.Thus, teachersof capstoneproject
courses could benefit from a better understandingof what kind
of problems studentstypically encounterin capstoneprojects,and
https://doi.org/10.1016/j.jss.2017.11.021
0164-1212/© 2017 The Authors.Published by Elsevier Inc.This is an open access article under the CC BY license.( http://creativecommons.org/licenses/by/4.0/ )

J. Vanhanen et al. / The Journal of Systems and Software 137 (2018) 50–66 51
how much the different problems affect students’ learning and
customersatisfaction.
In order to increase understanding of students’ learning and
customer satisfaction,and their relationship with software engi-
neering problems in the context of a capstone project, we con-
ducted a survey in our course. We focused on a select list of top-
ics based on a previous qualitative study, in which we gathered
hundreds of problems that our capstone project teams encoun-
tered during a previous run of the course (Vanhanen and Lehti-
nen, 2014). In this paper,we analyzehow commonly the individual
students working in different roles and teams encounter certain
software engineering problems and how much they learn about
the studied topics. Furthermore,we analyzethe effect of the prob-
lems on learning and on customer satisfaction.We conclude the
paper by analyzing the cost-benefit ratio of decreasedcustomer
satisfactionand increasedlearning for the various topics in order
to identify for which topics struggling with the problems could be
recommended.
Section 2 summarizes previous research related to problems,
learning and customersatisfactionin capstoneproject courses,and
to measuringlearning. Section 3 introduces the software capstone
project course at Aalto University.Section 4 presentsthe research
method. Section 5 presentsand discussesthe results of our study,
and the conclusionsare drawn in Section 6.
2. Relatedwork
Below we discuss previous researchabout software engineering
problems, learning and customer satisfactionin capstoneprojects.
Furthermore,as measuringlearning is in central role in our study,
we discuss the challengesrelated to it based on previous studies
on the topic.
2.1.Softwareengineeringproblems,learningand customer
satisfactionin capstonecourses
The idea of learning through problems during a software de-
velopment project course is not new. Dawson (2000) describes
a project course, where twenty dirty tricks were used to disrupt
the student’sprogress.The tricks were realistic challengesrelated
to, e.g., requirementsengineering,scheduling,work practices and
tools. Basedon informal feedbackthe course has been very educa-
tional, but the paper does not comparethe learning outcomesto a
setting where the desired problems are not encountered.
Previous empirical studies on encounteredproblems, students’
learning outcomes,or their mutual relationship in the context of
computersciencecapstoneprojectsare rare, even though hundreds
of paperson capstoneprojects have been written. Dugan (2011)re-
viewed a sample of about 200 papers on capstoneprojects in com-
puting, but neither the learning outcomes nor the problems en-
countered by the project teams were among the common topics
that emergedfrom the reviewed papers. Some of the papers dis-
cussed the learning theory behind the course, but even they did
not evaluate the efficacy of the learning theory. Some customer
satisfaction related aspects(meeting the requirements,taking the
project into use, monetary savings from using the results) were
discussedrelated to the used course evaluationtechniques.
In our own literature searches, we found a few studies on
capstoneprojects in computing that provide quantitativeinforma-
tion on the problems encountered,or on students’ learning (see
Table 1), but we were not able to identify any papers with data
about their relationship.Pournaghshband(1990) collected the five
most serious problems encounteredby each of their project teams,
and lists the frequenciesof the most common ones. All but one
of the most common serious problems were related to teamwork
and social issues, and the remaining one was related to testing.
Ahtee and Poranen (2009) list the frequencies of realized risks,
and Koolmanojwong and Boehm (2013) the frequenciesof iden-
tified risks in their capstoneprojects. Risks can be considered as
problems for a project at least if they realize. In these papers,the
frequentrisks included the lack of technologyskills or lack of par-
ticipation by the students,and covered also quality assurance,re-
quirementsengineeringand scheduling.
Learningoutcomesare seldom evaluatedin detail in studies de-
scribing capstonecourses in computing.Either the papers present
no data, or the data is on a general level, typically mention-
ing that the students considered the course very useful or edu-
cational. Mahnic (2012) asked the students to evaluate their im-
provement in eight different skills such as programming,project
planning and management,effort estimation,and team work on a
scale of 1–5, where five meant maximal improvement.The median
value of improvementwas high (4) for practically all of the skills.
Brueggeet al. (2015) report on students’improvementduring the
course in ten skills such as programming,design and version con-
trol, and on the changesin improvementover a four-year time pe-
riod. During the latest run of the course, for each skill 70–84%of
students “agreed” or“strongly agreed” thattheir skills improved.
Broman et al. (2012) used an open question to survey the most
important things for professionallife learned during their capstone
course. The most common answers mentioned by 6–10 of the 19
students were technical knowledge,time management,usefulness
of agile methods,team communication,and collaboration.
Previous studies about customer satisfaction seem to be even
rarer.Short remarksabout the topic can be found in some capstone
project case studies. For example,Goold (2003) reports 80% suc-
cess rate over two years and 21 projects regardingschedule,scope
and quality. Lack of project monitoring and control is reported as
the main reason for the less successfulprojects. Clark (2005) re-
ports about a course where the clients evaluate the developed
software and the professionalismof the teams from severalpoints
of view. The results indicate high customer satisfaction,but Clark
notes that the clients like to reward the students by giving full
marks when in most casesthey are not warranted.
2.2. Measuringlearning
Measuring students’ learning in a capstone course is compli-
cated. The scope of topics a student can potentially learn is huge
and can vary largely among the project teams, especially if each
team has a different project topic and the freedom to choose
their work practices, implementation technologies,and develop-
ment tools. Furthermore,for many topics the learning goal is to
achieve such a deep level of knowledge that a student is able to
apply the knowledgein her future projects.Measuring the achieve-
ment of such knowledgecan be more difficult than, e.g.,measuring
the knowledge of terminology.
Any standardizedtest of learning would have to contain very
high-level questionsdue to the differencesamong the projects,and
it would be difficult to come up with questions that would mea-
sure the deeper levels of knowledge. Furthermore,if a test is ar-
ranged only after the course or if the results of the project are
used as a measure of learning, it is not possible to differentiate
between what a student knew before the course and what she
learned during the course. Exams involving writing essayswould
allow the assessmentof deeper knowledge,but it would be labo-
rious to cover even the central areas of software engineering,and
difficult to evaluatethe essaysobjectively.
A simple measure of learning that is commonly used in stud-
ies is to ask the students to evaluate their learning themselves.
Unfortunately,the reliability of that method is limited. In a meta-
synthesis of 22 meta-analysesthe mean correlation between the
self-evaluationof ability and performanceoutcome was found to

52 J. Vanhanen et al. / The Journal of Systems and Software 137 (2018) 50–66
Table 1
Previous studies on encountered problems and learning.
Study Sample Focus
Encountered problems by capstone teams
Pournaghshband (1990) 54 students, two courses The most common serious problems listed by the students were: poor communication
among the members (72% of students), poor leadership (61%), failure to compromise
(56%), procrastination problems (54%), integration testing problems (44%), lack of
cooperation (30%), lack of confidence (28%).
Ahtee and Poranen (2009) 76 projects, two instances of
the course
The most commonly realized risks were related to: tools and skills (61% of projects),
scheduling problems (61%), technology problems (53%), working and studying during
the project (45%), motivation level low (36%), illnesses and social problems (34%),
communication problems (32%), requirements (32%).
Koolmanojwong and
Boehm (2013)
86 teams, five instances of the
course
The most commonly identified risk categories were: 1. Architecture complexity, quality
tradeoffs, 2. Personnel shortfalls, 3. Budget and schedule constraints, 4. COTS and
other independently evolving systems, 5. Customer-developer-user team cohesion, 6.
Requirements volatility, 7. User interface mismatch, 8. Process quality assurance, 9.
Requirements mismatch, 10. Acquisition and contracting process mismatches.
Learning outcomes
Mahnic (2012) 52 students in 13 teams The median value of improvement in students’ skills was high (4) on scale 1–5 for
practically all of the skills that students evaluated: familiarity with agile approach,
programming, project planning and management, effort estimation, “big picture”
about software development process, team work, customer interaction, and
communication.
Bruegge et al. (2015) 178 students in 40 projects,
four instances of the course
After the latest instance of the course, 70–84% of students “agreed” or “strongly
agreed” for each of the studied skills (requirements engineering, system design,
modelling, programming, version control, release management, communication, team
work, presentation, demo management) thattheir skills improved.
Broman et al. (2012) 19 students The students listed technical knowledge, time management, usefulness of agile
methods, team communication, and collaboration as the most commonly learned
important things for professional life.
be 0.29 (Zell and Krizan, 2014). However, there is evidence that
students’perceptionsof learning might be more reliable when they
assessdeeper,more practical learning, which is similar to the sce-
nario in capstonecourses.In a study by Stehle et al. (2012), medi-
cal students’perceptionsof learning on a course did not correlate
(r = −0.10)with their results of a multiple-choice examination,but
correlatedstrongly (r = 0.50) with the results of a test where they
showed their practical skills with simulated patients or simulators.
3. The capstoneprojectcourse
In the capstone project course at Aalto University, student
teams develop real software for real customers.When this study
was conducted,fifteen projects were carried out by teams of seven
to nine students.Below, we describe the various stakeholdersand
roles involved, and the common software development process
used in all the projects.
3.1.Projectstakeholdersand roles
Each project team has four to six Bachelor-levelstudentswork-
ing as developers.The course is scheduledfor the third (last) year
of their bachelorstudies,and they have already studied many pro-
grammingand computersciencecourses,and an introductory soft-
ware engineeringcourse.
Each project team has also three software engineeringexperts:
a project manager,a quality manager,and an architect. They are
typically Master-levelsoftware engineeringstudents,who are par-
ticipating the course for the second time. They have already taken
the same capstone project course as developers,and thereafter
studied some advanced software engineering courses. However,
due to the limited number of the Master-levelstudentssome vol-
unteersamong the Bachelor-levelstudentsalso work in these roles.
The software engineeringexperts are responsiblefor project man-
agement,requirementsengineering,quality assurance,and archi-
tectural design.
Each team may choose the three software engineeringexperts
and three developersthemselves.The course staff assigns the re-
maining developersevenly to all the teams based on their skills
and interests. The course is a mandatory part of the studies for
practically all the participants.Their years of presenceat the uni-
versity distributes quite evenly among three, four, five and more
than five years.
The teacherlooks for the customercandidatesbefore the course
begins.When the study was conducted,there were 26 topics avail-
able to be chosen by the fifteen teams.Most of the customersare
from the industry, but there are also some real topics available
from the university. During the projects, each customer actively
participatesin the requirementsdefinition work as well as moni-
tors the project progress.In the beginningof the project, the avail-
able effort is fixed to 125–200 h per student based on the cred-
its each student aims at. Therefore,the customer must prune the
scope of the project during the project according to the progress
of the team.
The course staff consistsof the course teacherand severalmen-
tors, who are previous students of the course. Each mentor typi-
cally guides two teams in issues related to the software develop-
ment process. Before the projects begin, there are a few lectures
related to the course arrangements,topic presentations,and used
software developmentprocess.During the project the teacher ar-
rangessix experienceexchangesessions,where individual students
from all of the teamscan join to discusson problemswith a partic-
ular theme.The project evaluationis basedon both the work prac-
tices used and the results achieved.All projects have three phases,
after which both the mentor and the customer gives their team
points and concretefeedback.
3.2. Softwaredevelopmentprocess
All the teams must apply the same predefinedsoftware devel-
opment processframework.Some of the work practiceshave been
defined strictly due to their educationalor critical nature, but for
many practiceseach team has a lot of freedom to customizethem
into their particular project. Many teams work a lot in a collocated
manner having one to two weekly work sessions,but there are of-
ten some studentswho cannot attend all the sessions.A few teams

J. Vanhanen et al. / The Journal of Systems and Software 137 (2018) 50–66 53
work mainly in a distributed manner,e.g.,due to the incompatible
schedulesof the team members.
All projects have three phases.Each phase ends in a project re-
view with all the stakeholdersincluding the customer,the men-
tor and the course teacher.In the project reviews, the project re-
sults are demonstrated,used work practices are discussed, and
project status such as product quality and resource spending are
presented.Before the project reviews, the teams must conduct a
retrospective,where they analyze the used work practices.
The first phase lasts four weeks and focuses on setting up the
project. It typically includes getting to know the team, deciding
work practices, identifying the project goals, understandingwhy
the system is built and for whom, identifying the most important
requirements,identifying risks, drafting the architecture,choos-
ing the implementationtechnologies,and setting up the develop-
ment environment.During the first phase, the teams are required
to write a project plan and a requirementsdocument. However,
documentingthe individual requirementsin detail is not required
before they are chosen for the implementation in the upcoming
phases.
The second and third phaseslast about six weeks and focus on
developingand deliveringfeatures.These phasesmust be split into
two or three sprints. Each sprint starts with sprint planning,where
the sprint goals, deliverablesand tasks are planned together with
the customer.During each phase the teams aim at implementing,
testing and deliveringsoftware that fulfils the chosenrequirements
to the customer.
Quality assurance is emphasized in the process framework.
Each team must identify the most important quality goals, and
choose and schedule the practices needed for achieving them.
There is much freedom in choosing the practices,but each team
must at least 1) preparetest casesfor functional testing that cover
at least half of the implementedsystem requirements,2) perform
a reasonableamount of unit level testing, 3) use a coding stan-
dard, 4) organize a code review for at least one critical component
of the system,and 5) arrangepeer testing with one other team at
the end of the project using exploratorytesting.
The process framework helps the teams define their devel-
opment process more quickly than from scratch. It also tries to
ensure that all teams get practical experience in using certain
software engineering practices that are aligned with the educa-
tional goals of the course. Some of the required practices or doc-
uments can be unnecessaryfor fulfilling the customer’sgoals for
the project, but from the teacher’spoint of view it is better to have
some overheadin the process,e.g.a risk managementprocess,if it
provides valuable educational opportunities or decreasesthe rate
of totally failed projects.
4. Researchmethod
4.1.Background
This study builds upon our previous study about the prob-
lems encounteredon a previous run of the same capstoneproject
course (Vanhanenand Lehtinen,2014). In that study,we conducted
2-hour-long retrospectivesin eleven student teams twice during
their projects.As a result, we identified hundreds of concretesoft-
ware engineeringproblems encounteredby the teams.
In this new study,we analyzeproblems,learning,and customer
satisfaction in the same course. We focus on eleven software en-
gineering topics (Table 2) with which problems were common ac-
cording to our previous study, as well as our experienceover the
years.The selectedtopics cover most of the common software en-
gineering topics, leaving out mainly the pure programming work
where the problems are typically very project-specificdue to the
different technologiesused in the projects.
Table 2
The studied software engineering topics.
Topic Abbreviation
1. Gaining an understanding of the features desired
by the customer
Features
2. Gaining an understanding of the quality
requirements desired by the customer
Quality requirements
3. Managing the customer’s expectations of the
project scope
Customer’s expectations
4. Planning and performing the testing Testing
5. Getting all the developers reasonably skilled
with the implementation technologies
Technology skills
6. Managing the versions of the source code Version control
7. Estimating the effort of the project’s tasks Effort estimation
8. Managing which tasks to do next and how Task management
9. Communicating within the team Team communication
10. Communicating with the customer Customercommunication
11. Maintaining the motivation of the team
members
Maintaining motivation
We used a questionnaire-basedsurvey to collect the data. The
topics were made comprehensibleto the respondents by giving
them simple, informative labels and by accompanying each of
them with 2–5 representative,concreteproblems(Table3) selected
from the data of the previous study.
4.2. Researchgoal and researchquestions
Our research goal is to better understand how much the stu-
dents learn from a capstoneproject, and whether struggling with
problems during the project affects their learning or customersat-
isfaction. As we are particularly interestedin the role of problems,
we focused on studying topics (Table 2) related to which students
are likely to encounterproblems.
We pose four researchquestions:
• RQ 1. How much do the students struggle with problems in
their projects?
◦ RQ 1a. To what extent do the students encounterproblems
in their projects?
◦ RQ 1b. How much do the encounteredproblems affect the
students’work?
◦ RQ 1c. How much do the studentscontribute to solving the
encounteredproblems?
• RQ 2. How much do the problems affect learning?
• RQ 3. How much do the studentslearn?
• RQ 4. How much do the problems affect customersatisfaction?
In RQ 1 and RQ 2 we study the individual students,but in RQ
3 we study the differencesin learning both among the individual
students and among the teams. In RQ 4, the analysis is done on
the team-levelonly, as customersatisfactionis a team-levelmetric.
For all research questions, we analyze how the student’s role as
a manager (project manager or quality manager) or a developer
(architector developer)affects the results.
In RQ 1a, we separatelystudy each of the 39 concreteproblems
grouped under the eleven topics. In RQ 1b, RQ 1c, RQ 2 and RQ 3
the unit of analysis is a topic as a whole, and in RQ 4 we study
both the concreteproblems and the topics.
In RQ 1, we study strugglingwith problemsfrom three points of
view: encounteringthem, own work being affected by them, and
contributing to solving them. In RQ 2 and RQ 3, we study learn-
ing about the purpose, challengesand work practicesrelated to a
topic. In RQ 2 and RQ 4, the effectsof the problems are studied by
calculating the correlation between struggling with the problems
and learning or customersatisfaction.