|
ResearchAudio · Paper of the week
300+ papers on AI agents that make 3D graphics. An AI agent helped read them.
7-minute read · 9 diagrams · Paper: arXiv 2611.01860
|
|
Here is a fact that surprises many people: almost all computer graphics is code. A 3D chair in a game, the light in a film scene, a CAD part for a factory. Somewhere, a program builds it.
And AI is now very good at writing code. So researchers started to give AI agents the tools of a 3D artist. The agent writes the code, renders the result, looks at it, and fixes what is wrong.
This field grew so fast that nobody could keep up. A team from Stanford, UT Austin, Berkeley, Cambridge, and Simon Fraser tried to map it. There were too many papers to read by hand. So they built an AI agent to help them read the papers about AI agents.
This issue shows you the map, the robot librarian, and the four trends it found. One trend shows how these agents are learning to work more like human artists.
|
|
Note: This is a survey paper. It does not build a new graphics agent. It organizes the work of others and finds patterns in it.
|
|
|
In this issue
1. What a “graphics agent” is (and is not)
2. The map: four parts of every system
3. The robot librarian
4. What 300+ papers have in common
5. CAD vs. scenes: two different worlds
6. The eight open problems
|
|
|
Part 1 of 6
What a “graphics agent” is
|
|
A human 3D artist works in a loop. Build something. Look at it. Change it. Look again.
A graphics agent does the same. Instead of changing every setting of a material by hand, you can type one sentence, such as “make this look like old brass,” and add a reference photo. The agent does the rest.
|
| Diagram 1 · The loop of a graphics agent |
💬 You ask “A wooden chair with curved legs” |
→ |
⌨ Agent writes code for example, a Blender script |
→ |
⚙ Software runs it render, simulate, measure |
→ |
👁 Agent checks “The back leg is too short” |
| ↺ Fix and repeat, until the result matches the request |
|
|
|
The authors set a strict definition. Many systems that use AI for graphics are not agents. A system counts only if it passes all three tests:
|
| Diagram 2 · Is it an agent? Three tests |
|
✓ It IS an agent if:
1. An LLM makes at least some of the decisions.
2. It runs real operations in graphics software.
3. It uses feedback from the results to choose its next step.
|
✗ It is NOT an agent if:
It writes the output in one pass, with no checking.
It is a special network trained only for one graphics task.
It uses feedback only during training or final scoring.
|
|
|
Part 2 of 6
The map: four parts of every system
|
|
The authors found a second problem. CAD, animation, and simulation researchers build very similar parts. But they give those parts different names. So it is hard to see that two papers had the same idea.
Their answer is one shared vocabulary. Every graphics agent has four parts:
|
| Diagram 3 · The four-part map (taxonomy) |
1. Specification · the job
What task? (2D, static 3D, moving 3D.) What input? (text, image, video, a 3D file, a document.) What output? (code, a structured file, an editable scene, or a video.)
|
| ↓ |
2. Base model · the brain
A text model, a model that also sees images, or a mix. Used as-is, or trained more for graphics.
|
| ↓ |
3. Harness · the manager
Plan & act: who decides the steps, and in what order.
Feedback: who checks the work, and how.
Memory: what is kept, for how long, and who can see it.
|
| ↓ |
4. Graphics interface & tools · the hands
How the agent acts (code, structured files, tool calls, or clicking a GUI), in which software, and with what resources: 3D asset libraries, past examples, and documentation.
|
|
|
|
The most interesting part is the harness, and especially feedback. When the agent makes a chair, who decides that the chair is wrong? The survey lists five kinds of checker:
|
| Diagram 4 · Who checks the work? |
| 🪝 Self-check | The agent compares its own render with the target image. |
| 🧐 Critic | A second agent judges work it did not make, such as a room layout. |
| 📏 Code rules | A program tests exact limits: “Does this door hinge turn the right amount?” |
| 🌎 Physics test | Run a simulation: “Does the cup fall off the table under gravity?” |
| 👤 Human | A person gives notes during the work, like an art director. |
|
|
After the check, the agent must decide how to fix the problem. The survey names five strategies. Remember these. One of them is the biggest trend in Part 4.
|
| Diagram 5 · Five ways to fix a mistake |
🗑 Regenerate throw it away, start again |
|
🧹 Local patch fix only the broken part |
|
↩ Roll back return to an earlier good version |
|
🗺 Replan change the plan itself |
|
🏆 Pick the best make several, keep the winner |
Example of roll back: one system (SceneSmith) saves checkpoints of a 3D room. If the critic’s score goes down after a change, it undoes the change.
|
|
|
Part 3 of 6
The robot librarian
|
|
Now the clever part. A normal survey is written by people who read every paper. Here, the field grows too fast for that. So the team built an agentic literature survey system.
It ran on GPT-6 Astra. It found 350 papers published from 2024 to September 2026. Then the humans checked by hand that each paper was really on topic.
|
| Diagram 6 · How the robot librarian works |
1. Find Searches the web and paper lists. Improves its search words. Removes copies. Makes a reading queue. |
→ |
2. Read with evidence Reads text, figures, and tables. Collects proof before it labels the paper with the four-part map. |
→ |
3. Save Stores labels, insights, limits, and page references. Next run continues from there. |
| Good habits built in: it keeps “what the paper says” apart from “what the agent thinks.” And when the evidence is weak, it raises a flag instead of guessing. |
|
|
|
The authors plan to release the system publicly. They also packaged it as a “skill” that other agent tools can use. So in principle, you could point it at your own research field.
|
|
Keep reading: Next, the numbers. One design choice went from 50% of papers to 83% over the period studied. It is the same habit that good human artists have.
|
|
|
Part 4 of 6
What 300+ papers have in common
|
|
With every paper labelled in the same way, patterns appear. Here are the main numbers.
|
| Diagram 7 · The field, in numbers |
|
👁 Agents that check by looking at images
of the 304 works that use feedback from observations. Models that can see images made this easy.
🔗 Agents that combine more than one type of check
A picture can look right and still be wrong. Measurements and physics catch what the eye misses.
|
📈 The big trend: local patching (fix only the broken part)
| Start of period | |
| Latest period | |
|
|
📖 Agents that use documentation and rules (latest period)
|
|
|
|
Look at local patching again. A beginner who makes a mistake often starts over. An expert fixes only the broken part, and keeps everything that works. The agents are moving from the first habit to the second.
The rise of documentation tells a similar story. Graphics tools are complex. Agents now read the manual, such as the Blender API reference, before they act.
The authors sum it up: the field is moving toward agents that use richer feedback, make targeted fixes, and depend more on written knowledge.
|
|
Part 5 of 6
CAD vs. scenes: two different worlds
|
|
The shared map also shows where tasks are different. The clearest contrast is between CAD (precise parts for engineering) and 3D scenes (rooms, worlds, layouts).
|
| Diagram 8 · Same map, different habits |
|
🔧 CAD parts |
🏠 3D scenes |
| Main check |
Runtime errors: does the code run? |
Images and object positions: does it look right? |
| How it fixes |
More often regenerates everything |
More often patches one object |
| Main resource |
Documentation and rules |
Asset libraries (ready-made objects) |
| Why? In a room, you can move one chair alone. In CAD, step 7 depends on steps 1 to 6. Change one step and the rest can break. So a full restart is often safer. |
|
|
Two more findings from the conclusion. First, most systems use ready-made AI models, with little extra training for graphics. The new ideas are mostly in the harness, not the model. Second, code and structured files are still the main way agents act. Clicking in a GUI is used only when the software has no better option.
|
|
Part 6 of 6
The eight open problems
|
|
The survey ends with a list of what is still unsolved. If you want a research topic, start here.
|
| Diagram 9 · What is still unsolved |
| 1. Teamwork. Many agents, many tools, one coherent result. |
|
2. Choosing the action. When to write code, when to click, when to call a tool. |
|
| 3. Agent-first tools. Most graphics software was built for humans. Agents may need their own languages. |
|
4. Long projects. Remember decisions, learn from failures, stop early mistakes from spreading. |
|
| 5. Show your work. Let users see the plan and edits without drowning in detail. |
|
6. Vague requests. “Make it cleaner.” When to guess, and when to ask? |
|
| 7. Fair testing. AI judges miss small errors. Results change from run to run. |
|
8. Credit. Who is the author when an agent did much of the work? |
|
|
|
Problem 7 matters most for anyone reading results in this field. The authors say image-reading AI judges often miss small errors, do not match human taste, and ignore physical and geometric rules. Agent results also change from run to run.
There is one more trap. Many systems use closed, commercial models. So when a system scores well, it is hard to tell if the credit goes to the new design or just to a better model.
|
What to keep in mind
|
| The librarian checks format, not truth. The authors say this themselves. The system makes sure every label exists and is valid. It does not prove each label is correct. |
|
| Trends describe papers, not quality. “83% use local patching” tells you what researchers do. It does not prove that local patching is better. |
|
| A fast-moving target. The corpus stops at September 2026. In a field growing this fast, the map will need updates. The authors call it an “adaptive framework” for that reason. |
|
|
The takeaway
Graphics is code. AI writes code. So AI agents are becoming 3D artists, engineers, and animators.
The best ones work like skilled humans: they look, they measure, they read the manual, and they fix only what is broken.
And the field is now so big that the people who study it need an AI agent of their own just to keep up.
|
|
| Quick facts |
| Paper | Advances in Agentic Computer Graphics |
| Authors | Hengyu Liu, Jihyeon Je, Ruiqi Wang, and 12 others, including Leonidas Guibas, Maneesh Agrawala, and Jiajun Wu |
| Institutions | UT Austin, Stanford, Simon Fraser, UC Berkeley, Cambridge |
| Posted | 1 November 2026 |
| Type | Survey (state-of-the-art report) |
| Corpus | 350 papers (2024 to Sept 2026), found by an agent on GPT-6 Astra, checked by hand |
|
|
|
|
One question for you: would you use an AI agent to read and sort the papers in your field? What would you trust it with?
Hit reply and tell me. I read every answer.
|
ResearchAudio · Hard research papers, in simple English.
Forward this to one friend who works in 3D, games, or design.
|