Can a language model learn something as subjective as artistic taste? An open experiment with TRL and OpenEnv attempts to answer that question by training a model to write JavaScript that generates watercolor paintings. The result is not an image created directly by a visual model, but code that can be read, modified, and executed.
The original idea belongs to Surya Narreddi, who demonstrated a model capable of painting watercolor compositions using the p5.brush library on top of p5.js. The video quickly surpassed 1.5 million views. This implementation reproduces the approach with all its components published openly on Hugging Face.
When Code Becomes a Paintbrush
The model generates programs of approximately 150 lines in JavaScript. These programs use p5.brush, a library that attempts to simulate an entire artistic medium, not just draw geometric shapes.
The ink can bleed, the paper has texture, and the brushstrokes acquire volume. When the model uses fillBleed(0.25), for example, it is controlling how far the pigment extends beyond the edge of a shape.
Of the 47 methods available in the library, training allows the model to use only 10:
scaleBrushesnoStrokefillnoFillfillBleedfillTexturebeginShapevertexendShapecircle
Why limit the tools so much? Because a watercolor painting does not need every possibility offered by a graphics engine. By restricting the action space, the model must solve the image using filled shapes, transparency, and variations in pigment. The limitation ends up becoming part of the style.
The prompt also includes a simple instruction: paint each petal two or three times, starting with a large layer and following it with a smaller, more opaque one. Although it may seem like a minor detail, this change produced results with more color.
The Real Challenge: Training with Human Taste
In mathematics or programming, there is usually a verifiable answer. A problem has a solution, and a program can pass or fail a test. In this case, there is no correct watercolor painting.
The model must learn an aesthetic preference. How do you turn something so subjective into a training signal? With a collection of references and several automatic evaluators.
The reference set contains 178 paintings generated by models. They all start from openly licensed hibiscus photographs sourced from iNaturalist. Four open-source models generated the sketches, and a vision model provided feedback during three rounds of refinement.
Afterward, each image was evaluated manually and classified into two levels: love, for favorites, and okay, for acceptable images. That personal selection became the practical definition of what the system should consider a good painting.
The bottleneck is not only generating images. It is also deciding which ones are worth keeping.
How the Reward Works
The training combines four signals. Each one measures a different aspect of the result:
| Signal | Weight | What it evaluates |
|---|---|---|
| Validation gate | 0.05 | Whether the code compiles, paints, and does not cheat |
| Length | 0.05 | Gentle pressure toward longer programs |
| Comparative judge | 0.60 | Style compared with reference paintings |
| HPSv3 | 0.30 | General aesthetic preference for the image |
HPSv3 is an open 7-billion-parameter model that receives an image and a text description to estimate how much a person might like it. It was trained using human comparisons between images, so it represents an approximate average of collective taste.
The second evaluator is Qwen3-VL-30B-A3B-Instruct, used through Hugging Face inference providers. This judge compares the candidate painting with four references and considers elements such as soft edges, translucent washes, and pigment bleeding.
Each comparison is performed in both possible orders. The final score represents the proportion of times the candidate painting beats its rivals.
This is where an important difference appears: HPSv3 asks whether the image looks like an attractive flower, while the comparative judge asks whether it is well painted according to the selected style. One measures a more general preference, while the other attempts to capture the specific taste of the person who built the reference set.
Three Training Runs, Three Ways to Understand Beauty
Three versions of the model were trained by changing only the weights of the two main judges:
| Run | Comparative judge | HPSv3 | Purpose |
|---|---|---|---|
judge-led | 0.60 | 0.30 | Original blend |
hps-led | 0.30 | 0.60 | Middle ground |
hps-only | 0.00 | 0.90 | Validation with a single judge |
The results showed improvements in all three runs:
| Run | Steps | First third | Last third | Change |
|---|---|---|---|---|
hps-only | 60 | 0.58 | 0.71 | +0.13 |
judge-led | 110 | 0.45 | 0.72 | +0.27 |
hps-led | 110 | 0.57 | 0.82 | +0.24 |
At first, the model mainly learned to avoid failed paintings: nearly empty canvases, shapeless blotches, or barely recognizable flowers. Later, the comparative judge made the difference.
With HPSv3 as the only signal, the paintings became more reliable, but not necessarily more interesting. The model learned to produce something that looked like a flower and quickly reached a plateau.
When the reference-based judge was added, the system had a reason to keep improving. Pigment coverage doubled in the runs that used the judge, and the best paintings moved closer to the manually selected style.
GRPO, LoRA, and the Adjustments That Made Training Work
The model used was Qwen/Qwen3.5-35B-A3B, trained with GRPO and LoRA adapters. GRPO allows the model to generate several results for the same instruction, compare them through rewards, and adjust the model toward the better-rated alternatives.
The implementation required correcting four important settings in TRL:
| Setting | Before | After | Reason |
|---|---|---|---|
| Learning rate | 2e-5 | 5e-5 | Increase LoRA's ability to adapt |
| Scheduler | linear | constant_with_warmup | Prevent the rate from dropping too early |
scale_rewards | group | none | Prevent a gate rejection from affecting the entire group |
| Target modules | Manual list | all-linear | Adapt more layers of the expert model |
The switch to all-linear was especially important. The model uses a mixture-of-experts architecture, so the usual list of layers for a dense model left too many parts untrained. With the new setting, LoRA reached the available linear layers and learned enough to improve the paintings.
The complete training runs on Hugging Face through Jobs. The reinforcement environment operates as one Space, the HPSv3 evaluator as another Space, and the comparative judge is queried through Inference Providers.
Once the two Spaces have been duplicated and the reward variables configured, the process can be launched with a single hf jobs command. The environment renders each sketch in Chromium without a GPU, while the trainer adjusts the model based on the scores it receives.
The Infrastructure Is Also Part of the Experiment
A 60-step run required nearly 18 hours on an H200. The 110-step run took approximately 34 hours. Each step produces eight results and usually takes between 15 and 18 minutes, with 70% to 80% of that time spent rendering.
Each image takes between 69 and 96 seconds to render. The environment does not use a GPU, and p5.brush performs expensive pixel operations to simulate textures and bleeding.
There were also infrastructure problems that directly affected the reward. Some rendering failures or missing evaluator responses were recorded as a score of zero, just like a genuinely poor painting. This introduced noise into the learning process.
The solution was to return None in those cases and exclude the faulty run from the group. Overall, failures represented around 1.5% of the results, although they reached 5.2% in the worst run.
In addition, a bug was found in OpenEnv: a WebSocket connection closed by the server remained cached and caused all subsequent calls to fail. The fix was submitted to the project and helped keep the runs stable.
What the Model Learned—and What Is Still Missing
The system did not follow every instruction in the prompt. It was asked to generate between 15 and 30 filled shapes, but the actual average ended up between 7 and 9. The number of shapes had almost no relationship with the reward.
This reveals something useful: during reinforcement learning, instructions that are not backed by a reward tend to lose importance. The model follows what improves its score, not necessarily what the prompt describes most clearly.
Strong visual convergence also appeared. Because the reference set consisted primarily of hibiscuses, the paintings ended up looking similar to one another. The model learned what a good painting was, but within a very narrow space.
To obtain more varied results, you would need to build a more diverse collection. Changing the reference set would automatically change the taste guiding the reward, without modifying the environment's code.
Possible next experiments include:
- Multi-step training, allowing the model to see what it has just painted.
- Using smaller models, such as a 4-billion-parameter version.
- Preliminary supervised fine-tuning with the reference sketches.
- Explicit rewards for pigment usage.
- Gradually increasing the difficulty of the references.
- Expanding the list of methods allowed in
p5.brush. - Measuring the consistency of the comparative judge by repeating the evaluation of the same image.
The possibility of using smaller models is especially interesting. In preliminary experiments, a 4-billion-parameter model was already able to produce valid sketches. If it can also learn the style through GRPO, training costs could be reduced considerably.
A Different Look at AI-Generated Art
This project brings to mind the early years of generative AI art, when tools such as DeepDream stopped being simple technical experiments and began to be used to explore new visual forms.
The difference is that here the model does not deliver only a final image. It delivers a program. You can read it, edit it, run it again, and observe the decisions behind each brushstroke.
The result is also not trying to achieve the statistical perfection that tends to dominate today's image generators. The paintings are irregular, soft, and occasionally imperfect. That imperfection seems to be part of their appeal.
But the most important question is not whether a model can learn to paint. It is who decides what it means to paint well. In this experiment, the answer lies in 178 model-generated images selected by one person.
That is the lesson that goes beyond watercolors: when you train AI with aesthetic rewards, curation stops being a detail and becomes the system's invisible architecture. The model can generate, compare, and optimize, but someone has to decide what deserves to be considered valuable.