Free-Tier Benchmark: Assessment framework for free LLMs

Read the analysis notebook

Free-Tier Benchmark began as three separate assignments for a graduate course. For Perplexity, ChatGPT, and Gemini, each was measured by answering narrow questions. Pulled into one table they answer a broader one. Where do free language models break when they read code, and how do they differ? The comments were stripped for every cell to prevent the model from reading the answer from the text. The same prompt asked it to identify the main calculation, and to explain the output. A stopwatch recorded how long the model took to generate an answer. Additionally, it recorded how long the code took to run in Colab. Only free versions were used. Every number here is computed live from those raw measurements.

Method

Each model was scored on three things and timed twice. Accuracy is the share of the code’s real key points the model correctly identified. Hallucination rate is the share of its statements that were wrong or fabricated. Efficiency is characters of response per second. For every cell, a stopwatch caught how long the model took to answer set against how long the code itself took to run in Colab. Perplexity read a notebook containing 5 cells, ChatGPT read another with 12 cells, and Gemini read another with 9 cells.

Results

The three models sit at different points of a single tradeoff. ChatGPT is the slowest at about 25 seconds a response and the most accurate at 90 percent with only 3.5 percent hallucination. Perplexity produces text fastest and hallucinates the most at 26 percent. Gemini falls between them at 78 percent accuracy. One caution belongs next to these bars. Each model read a different notebook, so the ranking mixes the model with the material. The ChatGPT notebook was the computationally lightest of the three. The pattern is real within this data and a controlled ranking would need every model on the same cells.

Discussion

None of the three run the code. Each model answered every cell in roughly the same time no matter how long the code took to execute. Colab needed 82 seconds to run the heaviest cell for Perplexity, and 61 seconds for Gemini’s heaviest cell. Both models answered those in under 11 seconds. A model that was truly computing the result would slow down on the hard cells. These do not. They read the code and describe it. Regarding the heavy cells, all three produced wrong plots and matched wrong numbers together, which is the same result seen from the other side.

Related Work

Public benchmarks approach model quality from other axes, and AidanBench is a good reference point. It puts the same set of open ended questions in front of every model, and asks each one to keep producing fresh answers to the same question. A judge model scores each answer for coherence, and an embedding distance scores it for novelty against everything the model already said. Generation stops when an answer turns incoherent, or repeats an earlier idea. The summed novelty becomes the score in thoat instance. Because every model faces identical prompts, its leaderboard is a controlled comparison. Free-Tier Benchmark measures much differently with faithfulness on code rather than novelty on open questions, and with a human reading of each cell as ground truth rather than a judge model. There is no focus on API endpoints. The contrast also marks the clearest next step for this project: placing the same cells in front of every model.

Limitations

This is a probe, and not a benchmark leaderboard. One person annotated every study to ensure no agreement across annotators. Each model was tested on its free tier only, and paid versions with real code execution would very likely behave differently. Each model also read a different notebook. The cross model ranking mixes the model with the material, and reads as indicative rather than controlled. The samples are small at 5, 12, and 9 cells. These bounds mark where the numbers stop rather than a flaw in the method.

Availability

This is an analysis notebook, and not a deployed app. There is no live demo to launch. It runs in a few seconds on any machine with Python. The rendered version linked above shows every figure and number next to the code that produced it.