Machine Learning Assisted Software Maintainability Assessments

Project Description

Contributions and Results

This research contributes several assets to the state of the art:

• Our structured evaluation model sheds light on how industrial maintainability assessments are performed by experts at an industrial partner. 
• As a basis for our ML-driven research, we contribute a manually labeled software maintainability dataset. Here, we selected a representative sample from nine software projects and surveyed 70 professionals affiliated with 17 institutions. The resulting dataset contains the consensus of the experts for a total of 519 code files.
• Using this dataset, we compare several ML classification algorithms and investigate different ways to represent the source code as input to these models. 
• Eventually, the best-performing classifiers are consolidated into a prototypical tool that combines the necessary data preparation, inference of the maintainability ratings, and a web-based frontend.

In summary, this study shows that ML, especially metric-based classification models, contributes to efficient maintainability assessments by identifying hard-to-maintain code with high precision. However, despite the good performance, we conclude the final interpretation of the findings should remain in the hands of a human analyst.

A Software Maintainability Dataset

Before controlling the maintainability of software systems, we need to assess them. Current automatic approaches have received criticism because their results often do not reflect the opinion of experts or are biased towards a small group of experts, or use problematic surrogate measures for software maintainability. In contrast, we refer to the consensus judgment of at least three study participants per code file. In summary, 70 professionals assessed code from 9 open and closed-source Java projects with a combined size of 1.4 million source lines of code. The assessment covers an overall judgment as well as an assessment of several subdimensions of maintainability. The resulting dataset contains the consensus of the experts for 519 Java classes. Interestingly, our analysis revealed that disagreement between evaluators occurs frequently. This emphasizes the need to refer to the consensus of several experts. Also, the average deviation of an expert from the consensus allows for an illustrative human-level performance baseline to evaluate the trained ML models.

For more details about the creation of this dataset, please refer to: M. Schnappinger, A. Fietzke, and A. Pretschner, "Defining a Software Maintainability Dataset: Collecting, Aggregating and Analysing Expert Evaluations of Software Maintainability", International Conference on Software Maintenance and Evolution (ICSME), 2020

The dataset, i.e. code plus labels and instructions on how to use the data, is available from here