Post on 24-Jun-2020
1
Departamento deInformática
Empirical Methods inSoftware Engineering
Alessandro Garcia
August 2014
Departamento de Informática
Website do curso
• Com todos slides e informações:
http://www.inf.puc-rio.br/~inf2921/2014_2/
– Já está disponível a partir desta semana:
• “placeholder for the slides and key references”
• agenda com aulas e datas das apresentações
2
Departamento de Informática
Datas e Apresentações: Alterações
• Me enviem as datas de preferência, título e breve resumo até (no máximo) 17/09
• Apresentações do Tema do Trabalho– 24/09 e 8/10
– 24/09: Diego, Leonardo, Benjamin? Ok? Mais alguém? Trocas?
– Apresentação: 25 mins + 10 mins perguntas/sugestões
• Não haverá aula em 01/10– reposição na segunda parte de outubro
• Resultados dos Estudos Empíricos– Datas: 26 de novembro e 3 de dezembro
• Se o estudo empírico for em grupo, espera-se que todos os membros contribuam ao longo das apresentações
Departamento de Informática
Apresentação do Plano do Estudo
• Me enviem um e-mail até 17 de setembro, informando:– individual ou em dupla
– descrição do tema do estudo• título inicial
• qual o objetivo?
• você pode se juntar à um colega que já tem uma boa idéia:
– da aplicação à ser utilizada, e/ou
– dos objetivos do estudo
– fiquem à vontade para discutir idéias comigo
– Apresentação inicial: aprox. 25 mins + 5 mins perguntas
3
Departamento de Informática
Previous: Experimental Software Engineering
• Why?
– empirical studies in software engineering (SE) supports
• the advancement of the field through experimental processes
– common wisdom, intuition, speculation and proofs of concepts
are not reliable sources of credible knowledge
• transfer of research results to mainstream software
development requires empirical evidence
• What?
– experimental software engineering describes the methods and
techniques for performing empirical studies
• the process supports us in doing our studies correctly
Departamento de Informática
This Lecture…
• Brief overview
– Types of Empirical Methods
• using concrete examples
– studies developed as coursework in the past in this
course
Victor R. Basili, Richard W. Selby, David H. Hutchens: Experimentation in
Software Engineering. IEEE Trans. Software Eng. 12(7): 733-743 (1986)
Barbara A. Kitchenham, Shari Lawrence Pfleeger: Principles of survey
research part 6: data analysis. ACM SIGSOFT Software Engineering Notes
28(2): 24-27 (2003)
4
Departamento de Informática
1st Type: Case Study
• Observational, often in-situ for an ongoing project
– E.g. assess the effectiveness of RUP to promote overall design quality in company X and/or Y
– E.g. does the use of the product line approach accelerates time to market of Web information systems in company X? in Y?
• e.g. effort to instantiate a produt is lower?
• Study a phenomenon in a specific space and time period
– Applicable to dynamic or larger studies:
• Long term evaluations
• Industrial evaluations
– Can be also used to compare approaches
(e.g. on a project and sister project)
• not an experiment because:– subjects are not randomly selected
– little or no control over the experience of developers, tools employed, etc...
– Easier to plan than controlled experiments
Departamento de Informática
Case Study
• Harder to control, hence less useful for asserting causality
5
Departamento de Informática
Case Study
• Harder to control, hence less useful for asserting causality
Quais outras variáveis não controladas podem influenciar as
diferenças de resultados mais decisivamente?
Departamento de Informática
Case Study - Example
• Case: effect of using new modelling concepts when specifying aspects of test management
6
Departamento de Informática
Types of Empirical Strategies
• From Case Studies to Experiments:
Departamento de Informática
Experiments
• Most controlled form of study, laboratory, manipulation of
variables:
– E.g. time spent to develop and test a few auxiliary functions using:
• pair programming vs. solo programming
• testing last vs. testing first
– Controlled environment, where it is possible/doable to control:
• selection of subjects (e.g. similar backgrounds)
• learning curve
• application to both novices (students) and industry
programmers
• randomization over subjects
7
Departamento de Informática
Continuous Modularity
• support: warn and provide feedback to developers when editing the code
13
Before After
Ongoing
Continuously
Inbetween
Programmer
editing code
specifying
designing
compiling
committing
maintaining
evolving
Departamento de Informática
Continuous Modularity Support
14
Did I violate any
design rule?
How is the
modularity ‘health’
of my code?
Will I introduce
merge conflicts?
Did I introduce security
vulnerabilities?
How should I
refactor my code?
How to implement
this ‘catch’ block?
Is my list of assertions
ok?
8
Departamento de Informática
Claim: contextualized support...
15
m1(){
c0();
c1();
…
ci(); // throws T1
…
cn();
}
... for instance, to help coding exception handling
IDEs (e.g. Eclipse) only provide bad recommendations
How to implement
my catch block
here?
Departamento de Informática
IDE view with extra-information and code-examples provided by the heuristics
JimboEH: supporting programmers of exception handling code
Continuous Robust Programming
9
Departamento de Informática
Continuous Code Quality
17
[Murphy-hill, 2013] Emerson R. Murphy-Hill, Titus
Barik, Andrew P. Black: Interactive ambient visualizations for soft advice. Information Visualization 12(2):
107-132 (2013)
• What is it?
interactive support for
modularity awareness
• Stench Blossom
(flor de cheiro)
Departamento de Informática
Continuous Code Anomaly Detection
• What is it? Active and Explanation Views...
18
Which anomaly?
Which elementscomprise a anomaly?
10
Departamento de Informática
Continuous Modularity Support
19
Did I violate any
design rule?
How is the
modularity ‘health’
of my code?
Will I introduce
merge conflicts?
Did I introduce security
vulnerabilities?
How should I
refactor my code?
How to implement
this ‘catch’ block?
Is my list of assertions
ok?
How to perform a controlled experiment in this context?
Possible?
Departamento de Informática
Continuous Modularity Support
20
Did I violate any
design rule?
How is the
modularity ‘health’
of my code?
Will I introduce
merge conflicts?
Did I introduce security
vulnerabilities?
How should I
refactor my code?
How to implement
this ‘catch’ block?
Is my list of assertions
ok?Continous Modularity vs.
Convention Modularity
Impossible to perform an controlled all-encompassing EXPERIMENT in this context
11
Departamento de Informática
Continuous Modularity Support
21
Did I violate any
design rule?
How is the
modularity ‘health’
of my code?
Will I introduce
merge conflicts?
Did I introduce security
vulnerabilities?
How should I
refactor my code?
How to implement
this ‘catch’ block?
Is my list of assertions
ok?
Impossible to perform an controlled all-encompassing EXPERIMENT in this context:
too many variables to understand and control
e.g. levels of expertise of programer addressing each of
these questions; too many complementary tools to each
task; etc..
Departamento de Informática
Solution: choose a specific controllable context (and question)
22
Did I violate any
design rule?
How is the
modularity ‘health’
of my code?
Will I introduce
merge conflicts?
Did I introduce security
vulnerabilities?
How should I
refactor my code?
How to implement
this ‘catch’ block?
Is my list of assertions
ok?
12
Departamento de Informática
Much higher identification of anomalies
ContinuousDetection
ConventionalDetection
Controle: nível de
experiência across
types of subjects
and within subjects
Departamento de Informática
Initial evidence of higher refactoring quality
• As developers get feedback when they are working in that
context
24
Continuous Detection Conventional Detection
Bad
refactoring
Appropriate
refactoring
13
Departamento de Informática
ContinuousDetection
ConventionalDetection
Controle: nível de
experiência across
types of subjects
and within subjects
Uncontrolled candidate variable that
might have influenced the main
results?
Departamento de Informática
ContinuousDetection
ConventionalDetection
Controle: nível de
experiência across
types of subjects
and within subjects
Learning curve
• always same application or one always the same technique after
another
14
Departamento de Informática
Experiments
• Example of how to control subjects’ experience and learning curve
• Comparing three variability management techniques: from artifact comprehension point of view
– GenArch+, PureVariants, CIDE (similar to FeatureMapper)
Departamento de Informática
Experiments
• Most controlled form of study, laboratory, manipulation of
variables:
– Reproducible
• Because they demand less time in the execution
• The artefacts are produced just for the experiment
– … and can be reused (“as they are”) in experiment repetitions
– Validate more specific claims
• E.g. evaluate time spent to change a piece of module using
specific variability mechanisms
– conditional compilation vs. AOP
• E.g. efficacy of the metrics to detect a bad smell in a certain
design
15
Departamento de Informática
Experiments
• Another example:
– comparing the effort of using different model
composition techniques
Departamento de Informática
Experiments
• Example of applying the heuristic-based and specification-based techniques
Heuristics-based
Specification-
based
16
Departamento de Informática
Experiments
• The experiment was conducted with
– 16 subjects were professionals
– 8 subjects were students with professional experience
Departamento de Informática
Experiments
• The process…
17
Departamento de Informática
Intuition
RSA
Epsilon
Effort Correctness
L
H
L
H
Departamento de Informática
Results: Effort
RQ1: Which approach requires less effort to apply?
Med
ian
of
Eff
ort
(m
in) Total Effort
18
Departamento de Informática
Experimental Results: Effort
RQ1: Which approach requires less effort to apply?
• Effort
Epsilon required more effort than IBM RSA
Epsilon required more effort than merge/override
Total Effort
Departamento de Informática
Experiments
• Resultados:
19
Departamento de Informática
Difficulties on conducting this experiment
• Training was too expensive and needs to be carefully designed
– it took one day in each institution + preparation
– examples chosen cannot influence the participants in
favour of one technique
• Most professional developers were already used to apply RSA mechanisms
– how to take them to a similar level of knowledge with
the use of Epsilon and manual heuristics?
– we needed to apply questionnaire/exercises to check
their level of knowledge before the experiment itself
Departamento de Informática
Difficulties on conducting this experiment
• While the heuristics available are essentially the same…
– the DSLs for specification-based composition are very
different
• E.g. Epsilon against MATA
– therefore, we could not claim the results could be
generalized
– however, we can only use stable tools in experiments
• we are measuring time/correctness
20
Departamento deInformática
Empirical Methods inSoftware Engineering
Alessandro Garcia
August 2014