prathik@chicago:~$

Automated program repair with an LLM

Gives an LLM the crashing input and stack trace along with the code. It fixed 11 of 13 programs.

when
Sep – Dec 2024
where
University of Illinois Chicago (research project)
role
Researcher, paper author
stack
Python, AFL / AFL++, GDB, GPT-4o mini, C / C++
links
paper
problem
An LLM reads code but cannot see what the program does when it runs, which matters for crashes like buffer overflows.
built
A pipeline that fuzzes the program with AFL, collects GDB stack traces, and gives the LLM the code plus the crashing inputs and traces.
result
11 of 13 programs fixed (85%), up from 5 of 13 (38%) with code alone, in half the attempts.
Contents
The question
The pipeline
Results
Next

The question

LLMs read code well, but they can't see what a program does when it runs. For bugs like buffer overflows, it matters a lot which input crashes the program and where it crashes. I wanted to know how much that runtime evidence helps a model fix the crash.

The idea: use fuzzing to find crashes and GDB to locate them, then hand the model that evidence along with the code.

The pipeline

Six steps, then a loop

Program repair pipeline: strip comments, LLM writes a seed script, AFL fuzzing finds and shrinks crashing inputs, GDB collects stack traces, the LLM writes a patch, then the patch is compiled, tested and scored, looping back if it is not fixed
Fig. 1 The repair pipeline. Purple steps use the LLM, green steps run tools on the program.
  1. Strip the comments from the source so the model can't lean on human hints. In later rounds the model's own comments stay in, as its memory of what it tried.
  2. Have the LLM write a script that generates valid starting inputs for the fuzzer, and note whether the program reads stdin or a file.
  3. Run AFL or AFL++ to find crashing inputs, then shrink each one to a minimal input with afl-tmin.
  4. Collect a stack trace for each crash with GDB. Deduplicate them with an FNV-1a hash and keep the five most representative.
  5. Give the model the code, the minimized crashing inputs and the traces, and ask for a patch.
  6. Compile and test each patch and score it: does it compile, does the crash go away, do the tests pass, and how much did it change. Loop, with the model remembering earlier attempts.

Results

Three ways of prompting, same 13 programs

I tested on 13 crashing C programs: 10 that I wrote and 3 from AFL's demos. I compared giving the model the code only, the code plus stack traces, and the code plus stack traces and crashing inputs.

fixed, code only
38%
fixed, with stack traces
69%
fixed, with traces and crashing inputs
85%
Bar charts: share of 13 programs fixed rose from 38 to 69 to 85 percent, and median attempts fell from 4 to 3 to 2
Fig. 2 Fix rate and median attempts for each prompt.
  • Code only: 5 of 13 fixed (38%), median 4 attempts.
  • Code plus stack traces: 9 of 13 (69%), median 3 attempts.
  • Code plus stack traces and crashing inputs: 11 of 13 (85%), median 2 attempts.

Runtime evidence roughly doubled the fix rate and halved the attempts needed.

Next

  • Programs that span several files.
  • Symbolic execution for better fault localization.
  • The SARD and Juliet benchmarks.
  • Parallel fuzzing, to find crashes faster.

< back to projects/