GPT-4 Turbo with Vision is a step backwards for coding

OpenAI's GPT-4 Turbo with Vision model scores worse on aider's code editing benchmarks than all the previous GPT-4 models. In particular, it seems much more prone to "lazy coding" than the existing GPT-4 Turbo "preview" models.

APR 9, 2024

Claude 3 beats GPT-4 on Aider's code editing benchmark

Claude 3 Opus outperforms all of OpenAI's models on Aider's code editing benchmark, making it the best available model for pair programming with AI.

MAR 8, 2024

The January GPT-4 Turbo is lazier than the last version

The new `gpt-4-0125-preview` model is quantiatively lazier at coding than previous GPT-4 versions, according to a new "laziness" benchmark.

JAN 25, 2024

Unified diffs make GPT-4 Turbo 3X less lazy

GPT-4 Turbo has a problem with lazy coding, which can be signiciantly improved by asking for code changes formatted as unified diffs.

DEC 21, 2023

Speed benchmarks of GPT-4 Turbo and gpt-3.5-turbo-1106

This report provides a detailed comparison of the speed of GPT-4 Turbo and gpt-3.5-turbo-1106 models based on the aider benchmarking suite.

NOV 6, 2023

Code editing benchmarks for OpenAI's "1106" models

A quantitative comparison of the code editing capabilities of the new GPT-3.5 and GPT-4 versions that were released in Nov 2023.

NOV 6, 2023

Building a better repository map with tree sitter

Tree-sitter allows aider to build a repo map that better summarizes large code bases.

OCT 22, 2023

GPT code editing benchmarks

Benchmarking GPT-3.5 and GPT-4 code editing skill using a new code editing benchmark suite based on the Exercism python exercises.

JUL 2, 2023

Improving GPT-4's codebase understanding with ctags

Using ctags to build a "repository map" to increase GPT-4's ability to understand a large code base.

MAY 25, 2023