AI Assistants Survey
Results of a survey of 481 developers aimed at how AI assistants are used, focusing on specific software development activities and stages.
Data Paper Paper pre-print
Datasets, benchmarks, & tools 🛠️
This page aims to list in a convenient manner all the artifacts of my software engineering research that can be directly reused. This includes various datasets (for training models or for studying further), benchmarks for evaluation, and tools for different aspects of software engineering research.
Datasets & benchmarks
With the advent of AI, benchmarks left the halls of academia and became mainstream in all conversations about technology. From the very start, JetBrains Research had a very open policy about publishing data — we basically published everything we were legally allowed to. Because of that, quite a few datasets and benchmarks accumulated that I had something to do with.
CommitChronicle
A large-scale, diverse multilingual dataset for commit message generation and completion.
Data Paper Paper pre-printDapStep
A dataset of stack traces and developer labels for assignee predictions.
Data Paper Paper pre-printJupyter Notebooks
A dataset of 847,881 properly-licensed Jupyter notebooks.
Data Video Paper Paper pre-print🏆 ACM SIGSOFT Distinguished Paper Award 🏆
Long Code Arena
A suite of benchmarks for code-related tasks with large contexts, up to a whole code repository. It currently spans six different tasks and contains six datasets.
Data Paper pre-printPandasPlotBench
A human-curated benchmark designed to evaluate language models' effectiveness as assistants in visual data exploration.
Data Paper Paper pre-printRefactoring Usage Survey
Results of the largest survey on software refactoring: 1,183 users of IntelliJ-based IDEs.
Data Video Paper Paper pre-printrequirement.txt Files
A dataset of requirements.txt files for 7,132 Python projects, with their versions from 2011 to 2020.
Data Paper pre-printSlowOps
An industrial of stack traces and their categories, aimed at evaluating different models for stack trace deduplication.
Data Paper Paper pre-printTools
In my research, I have personally developed and maintained only one tool: Buckwheat, a multi-language tokenizer for extracting identifiers from source code. However, I participated in the development of several other tools or applied them for the analysis of large corpora of code.
AntiCopyPaster
A plugin for IntelliJ IDEA that tracks the pasting of code inside the IDE and suggests appropriate Extract Method refactorings to combat the propagation of duplicates.
Tool Paper Paper pre-printJupyter Logs Tooling
A toolset for collecting code changes in Jupyter notebooks during development time and viewing them later.
Tool Paper Paper pre-printKOALA
A plugin for running controlled experiments and collecting data directly within JetBrains IDEs.
Tool Paper Paper pre-printKotless
Kotlin serverless framework that aims to reduce the routine of serverless deployment by generating it straight from the application code.
Tool Video Paper Paper pre-printLitmusKt
A litmus testing tool for Kotlin.
Tool Video Paper Paper pre-printLupa
A framework for the large scale analysis of programming language usage.
Tool Video Paper Paper pre-printMatroskin
A library for the large scale analysis of Jupyter notebooks.
Tool Video Paper Paper pre-print🏆 ACM SIGSOFT Distinguished Paper Award 🏆
PyNose
A test smell detector for Python.
Tool Video Paper Paper pre-printPythonChangeMiner
A tool for creating fine-grained program dependence graphs (fgPDG) for Python code, creating change graphs from them, and mining these graphs for change patterns.
Tool Paper pre-printRevizor
A plugin for PyCharm that takes the recurrent change patterns and highlights potential places for their application in the developer’s code editor.
Tool Video Paper Paper pre-printRhubarb
A tool for clustering similar MOOC solutions and selecting diverse examples.
Tool Paper Paper pre-printSorrel
A plugin for IntelliJ IDEA for managing licenses and detecting license incompatibilities.
Tool Video Paper Paper pre-print🏆 Best Tool Demo Award 🏆
Sosed
A tool for discovering similar software projects.
Tool Video Paper Paper pre-print