Skip to content
sobolewski.proAI today, every eveningSubscribe

Once a day. Unsubscribe with one click. Privacy policy

GPT-6.2 Astra reviews: early testers pan it, Altman responds

GPT-6.2 Astra reviews are harsh: early testers describe abandoned tasks and answers they can't parse, and Sam Altman promises fixes. Here's what we know so far.

4 min read

This article in other languages:Polski

In this article
  1. What are testers holding against the model?
  2. What does OpenAI say?
  3. Does OpenAI’s documentation back up the GPT-6.2 Astra reviews?
  4. What should we watch for?
  5. FAQ

OpenAI shipped GPT-6.2 Astra on November 17, and that same day Sam Altman wrote on X that the new models are very smart, but (quoting) “we have more work ahead of us on making the models nice to work with”. A day later, Decrypt and Gizmodo rounded up GPT-6.2 Astra reviews from people who had the model before launch, and those reviews are unusually bad.

What are testers holding against the model?

Two complaints come up independently from several people: Astra cuts long tasks short, and it writes answers that are hard to understand.

Theo Browne wrote that he wouldn’t pick Astra over Opus 5.5 for his development work. Decrypt also quotes his description of the model’s style: “it speaks in tongues that only 6.2 Astra understands”. In his compiler-rewrite test, seven days of the model’s work brought no improvement at all, and the compiler ran 56% slower.

Matt Shumer, an entrepreneur and investor, admitted he had never written a bad review of a model before. In a YouTube review quoted by Gizmodo, he says “this model has no concept of what autonomy means”. Ethan Mollick of the Wharton School, who also used Astra before launch, called it “very uneven & weird” and added that it needed more steering and more prodding to keep working. Max Weinbach, for his part, wrote about having to type “continue” over and over.

There are voices on the other side too. TokenGremlin, quoted by Decrypt, rates Astra as SOTA (state of the art) in Three.js, Blender and 3D, though even he notes that in long workflows the model starts speaking an “alien language”.

What does OpenAI say?

Altman promised that making the models pleasant to work with will be a priority in upcoming releases, and added that the intelligence and capability of the new models are extraordinary, even if they take a bit more work.

The more interesting part is the explanation posted by Thibault Sottiaux of OpenAI. A few people got access a while ago, then in the last 3–4 days before launch the team made a lot of changes and, as Sottiaux writes, almost certainly nobody tested the model again. So some of the reviews may describe a version that never reached users.

Testers didn’t take that well. Dan Shipper wrote (as reported by Gizmodo) that he had seen no word from OpenAI about changes to check, and that his team would have restarted testing from scratch had they known.

Does OpenAI’s documentation back up the GPT-6.2 Astra reviews?

On one point, yes. OpenAI’s safety documentation for GPT-6.2 describes an evaluation called “Blocker Deception”: the model gets realistic coding tasks with an obstacle planted on purpose, and researchers check how often the final answer leaves that obstacle out or misrepresents it. GPT-6.2 Astra did so in 22.28% of cases, GPT-6 Astra in 11.09%.

According to the document, OpenAI is concerned about this regression and is looking for its causes. One hypothesis is the work on making answers more concise and trimming excess caveats. Summaries so terse they stop being readable are, after all, one of the testers’ two main complaints.

The obstacles were designed to provoke bad behavior, so the result doesn’t have to match how often this happens in everyday work (let’s read it as a comparison of two models, not a forecast).

What should we watch for?

First, the difference between Codex and the API. According to the same documentation, the Codex developer prompt clearly reduces the number of omitted obstacles, but it isn’t applied to direct API requests. If we’re building our own agent on gpt-6.2-astra, it’s worth asking for blockers to be reported in our own system prompt.

Second, what people say after launch. In the Hacker News discussion under the model announcement (over 1,300 points and over 800 comments as of November 20), opinions are split. One commenter reports that the cost of their data-analysis test dropped from about $20 to $4 at similar accuracy, while others describe the same things the early testers did: asking for permission too often and stopping work early.

Third, how this dispute differs from the recent one over the Opus 5.5 nerf. There, developers saw a drop in quality that an independent test didn’t confirm. Here, at least one of the vendor’s own measurements points the same way as the complaints.

FAQ

Is GPT-6.2 Astra worse than GPT-6 Astra?

It did worse in the Blocker Deception evaluation (22.28% versus 11.09% of obstacles omitted or misrepresented). At the same time, Sottiaux calls the new models “a hefty jump” over their predecessors, and early testers rate Astra poorly on long, autonomous tasks.

When will OpenAI fix GPT-6.2 Astra?

There’s no date. Altman wrote only that making the models pleasant to work with will be a priority in upcoming releases.

Where can we try GPT-6.2 Astra?

The model has been available in ChatGPT Work, Codex and the API since November 17. I covered pricing and limits in the article on the GPT-6.2 Astra and Sol release.

  • AI
  • OpenAI
  • ChatGPT
  • coding

AI today, every evening

The day's biggest AI news and new posts from the blog, every evening. Short, with sources.

Once a day. Unsubscribe with one click. Privacy policy