Applying Inter-Rater Reliability and Agreement in collaborative Grounded Theory studies in software engineering



Downloads per month over past year

Díaz, Jessica and Pérez, Jorge and Gallardo, Carolina and González Prieto, Ángel (2022) Applying Inter-Rater Reliability and Agreement in collaborative Grounded Theory studies in software engineering. Journal of Systems and Software, 195 . ISSN 0164-1212

[thumbnail of 1-s2.0-S0164121222001960-main.pdf]
Creative Commons Attribution Non-commercial No Derivatives.


Official URL:


Context: The qualitative research on empirical software engineering that uses Grounded Theory is increasing (GT). The trustworthiness, rigor, and transparency of GT qualitative data analysis can benefit, among others, when multiple analysts juxtapose diverse perspectives and collaborate to develop a common code frame based on a consensual and consistent interpretation. Inter-Rater Reliability (IRR) and/or Inter-Rater Agreement (IRA) are commonly used techniques to measure consensus, and thus develop a shared interpretation. However, minimal guidance is available about how and when to measure IRR/IRA during the iterative process of GT, so researchers have been using ad hoc methods for years.
Objective: This paper presents a process for systematically measuring IRR/IRA in GT studies, when appropriate, which is grounded in a previous systematic mapping study on collaborative GT in the field of software engineering.
Methods: Meta-science guided us to analyze the issues and challenges of collaborative GT and formalize a process to measure IRR/IRA in GT. Results: This process guides researchers to incrementally generate a theory while ensuring consensus on the constructs that support it, improving trustworthiness, rigor, and transparency, and promoting the communicability, reflexivity, and replicability of the research.
Conclusion: The application of this process to a GT study seems to support its feasibility. In the absence of further confirmation, this would represent the first step in a de facto standard to be applied to those GT studies that may benefit from IRR/IRA techniques.

Item Type:Article
Additional Information:

CRUE-CSIC (Acuerdos Transformativos 2022)

Uncontrolled Keywords:Grounded Theory, Inter-Rater Reliability, Inter-Rater Agreement
Subjects:Sciences > Computer science
Sciences > Mathematics > Applied statistics
ID Code:75201
Deposited On:08 Nov 2022 08:37
Last Modified:19 Dec 2022 09:18

Origin of downloads

Repository Staff Only: item control page