Critique: Large-Scale Study of Programming Languages and Code Quality

This report provides a critical analysis of a research paper that investigates the impact of programming languages on code quality using a large dataset from GitHub. The critique evaluates the paper's motivation, framing of related work, data collection methods, data analysis techniques, and the suitability of the methods employed. The paper's strengths include its use of a mixed-methods approach, combining regression modeling with visualization and text analytics, to study the effect of language features. However, the critique also identifies weaknesses, such as the lack of specific method support in the conclusion, the absence of a survey method for data collection from participants, and the failure to quantify the specific effects of language type on use. The analysis highlights the importance of considering various factors, including language design, team size, and project history, when assessing software quality. The report concludes by acknowledging the paper's contributions while suggesting areas for future research and improvement.

Critique paper
Is the research well-motivated? Why, why
not?
Multiple numbers of papers are considered
and analyzed regarding large scale study of
the programming language and Github
codes. In case of Github, lightweight code
review techniques are built into pull
requests. The team is also allowed to create
review processes that can improve the code
quality that may also fit into the workflow.
It has been determined that, pull requests are
primary and it defines how team can review
and improve the codes on Github which
seems to be motivating for the readers.
Is the paper well-framed in terms of
related work? Why? How?
It has been found that, prior works on
programming language cpmparison are fall
under three different categories in terms of
controlled experiment, survey and repository
mining. All three of the processes are
elaborated accurately throughout the paper.
The significant differences between code
qualities are defined in the paper. On the
other hand, it is also found that statistically
typed languages are less defect prone that
the general dynamic type. However, the
related work section is not illustrated like
other similar source papers. After study of
four different projects developed in C and
c++ the related work was presented.
However, possible all bugs were not
elaborated in the related wok section.
Does the paper tell you what data was
collected? What's missing?
The paper tells what different kinds of data
are being collected from various resources.
For each of the language different projects
are identified and illustrated. Apart from

that, project evolution history is also
elaborated in this paper. Apart from that, a
table is also elaborated containing language,
project details, total commits and also the
bug fix commits.
The sources of data were not mentioned
accurately. On the other hand, in the
methodology part how popular projets are
retrieved are also elaborated exceptionally
beautifully. Apart from that, in table four
characteristic of domains are also elaborated
nicely.
Does the paper tell you how this data was
analysed/interpreted and how the
research question was answered? Is
anything missing?
Similarly like the data collection section the
data analysis part is also elaborated
amazingly. Bugs categorization and
statistical methods are applied to analyze the
data collected from different sources. Not
only this but also, key word searching,
supervised classification are done for
analyzing the data collected. The whole
dataset is analyzed considering bug types,
bug description and search keywords and
phrases as well. The reasons for each bug
and also impact of the bugs on program
outputs are elaborate exceptionally well in
the paper.
Does the method appear suitable? If not,
why not?
While this paper states appreciably research
supporting the view that some kinds of
programming language and also utility of
codes on Github can have a positive impact
on student’s learning at the same time in the
conclusion no particular method that can
support the study is mentioned accurately.
The use of mix method is stated in the paper
elaborating text analysis, clustering and
visualization. How Github plays effective
role for evolving projects and for proposing

new features as well are elaborated. Infact
source codes can be changed and
implemented as well is also elaborated in
details in this nominated paper but not
followed by any specific method. In this
study large set of data are considered as
well. Due to this collective analysis and data
representation from the critique perspective
it is a good professionally represented paper.
Infact different intangible process factors are
elaborated without any method
specification.
Does the paper make sense? Is it well-
presented and easy to read? Why, why
not?
Some arguments are synthesized in the
paper combining multiple numbers of
regression models but there is connection
between the finding analysis and the result.
The research paper is unable to quantify the
specific effects og language type on use. In
addition to that at the conclusion it is
realized that survey method could have been
used to collect data from particioants but not
done earlier. These challenges were only
addressed but how to overcome these are not
mentioned here. Therefore, it can be said
that this paper is not a complete one and
requires future work.
References
Berger, E.D., Maj, P., Vitek, O. and Vitek, J., 2019.
FSE/CACM Rebuttal $^ 2$: Correcting A Large-
Scale Study of Programming Languages and Code
Quality in GitHub. arXiv preprint arXiv:1911.11894.
Kochhar, P.S., Wijedasa, D. and Lo, D., 2016,
March. A large scale study of multiple programming
languages and code quality. In 2016 IEEE 23rd
International Conference on Software Analysis,
Evolution, and Reengineering (SANER) (Vol. 1, pp.
563-573). IEEE.
Horschig, S., Mattis, T. and Hirschfeld, R., 2018,
April. Do Java programmers write better Python?
Studying off-language code quality on GitHub.
In Conference Companion of the 2nd International
Conference on Art, Science, and Engineering of
Programming (pp. 127-134).

Roehm, T., Veihelmann, D., Wagner, S. and
Juergens, E., 2019, January. Evaluating
Maintainability Prejudices with a Large-Scale Study
of Open-Source Projects. In International Conference
on Software Quality (pp. 151-171). Springer, Cham.