SenseNova-U1.5 Upgrades to 4K and Fixes Patch Seams in SenseTime's Encoder-Free AI

SenseTime published a technical report this week for SenseNova-U1.5, an 8-billion-parameter native multimodal model that handles image understanding, generation, and editing inside a single architecture — with no visual encoder and no variational autoencoder. The report, posted to arXiv on September 10 by a team of 65 researchers and appearing in HuggingFace's September 11 daily paper digest, documents a five-stage training pipeline that builds a general-purpose visual intelligence system from raw pixels and text, without any of the pretrained visual modules that currently underpin most image-generating models. The paper's most consequential claim — that a model learning directly from raw pixels can transfer structural knowledge gained during understanding to the task of visual creation — moves the field's debate from theory to demonstrated evidence.

The Architectural Divide the Paper Is Designed to Close

Most visual AI systems today are assembled from two independently trained subsystems. On the perception side, pretrained visual encoders such as CLIP or SigLIP translate raw images into high-level semantic feature vectors. On the generation side, a variational autoencoder compresses those images into a compact latent space, where a diffusion or flow-matching model can operate efficiently. The Janus-Pro model from DeepSeek, for example, maintains entirely separate parameter pathways for understanding and generation, routing through a SigLIP encoder for perception and a VQ tokenizer for synthesis. BAGEL, FLUX.1-Dev, and most commercial image generators including those underlying GPT-4o's image capability all rely on some version of this split.

The split is practical: encoders and VAEs trained on large datasets provide strong starting points that reduce compute requirements for fine-tuning. But SenseTime argues it is structurally limiting. When understanding and generation operate in separate representation spaces — one optimized for semantic abstraction, the other for pixel-level fidelity — the two systems cannot easily share what they have learned. Perception cannot directly inform generation; generation cannot directly inform perception. Any transfer between them requires an explicit bridge layer or architectural handoff, introducing additional engineering complexity and potential information loss.

SenseNova-U1.5 takes a different route. It processes both raw image pixels and text tokens inside a single transformer backbone, learning a shared representation from the ground up. The model never passes control from an encoder to a decoder through a separately optimized latent space. Understanding and generation are co-trained on the same parameters from the start, a design the team calls NEO-unify.

Read more: DeepSeek adds vision to V4-Flash, covering the modular vs. unified multimodal architecture divide

How the Architecture Actually Works

The model's visual processing begins with a pair of convolutional projections that downsample each input image by a total factor of 32, yielding one visual token per 32×32-pixel region. Two-dimensional sinusoidal positional embeddings preserve spatial coordinates, and the resulting token sequence is fed into the same transformer backbone that handles text tokens. No pretrained vision encoder contributes to this step. The visual tokens and text tokens are projected into a shared hidden space and processed jointly.

For generation, SenseNova-U1.5 uses flow matching directly in pixel space rather than in a VAE latent space. This is significant: rather than generating within a compressed bottleneck and then decoding outward, the model learns to denoise noisy pixel sequences directly toward coherent images. The training objective combines three losses: autoregressive cross-entropy for text prediction, pixel-space flow matching supervised against a velocity field, and LPIPS perceptual loss to enforce structural and textual consistency in generated images.

The model uses a Mixture-of-Transformers (MoT) design rather than a single undifferentiated backbone. Clean image-text tokens and noise-conditioned generation tokens are interleaved in one sequence, with separate attention projection, normalization, and feedforward parameters for each token type. Crucially, these two streams share self-attention, so understanding and generation states can exchange information at every transformer layer through cross-stream attention — without requiring a separate fusion module. Attention from clean context to generation tokens is permitted; attention from generation states back to clean context is explicitly masked to prevent stochastic noise from corrupting perceptual representations.

Solving the Patch Seam Problem

SenseNova-U1.5's predecessor, SenseNova-U1, established the feasibility of encoder-free pixel-space generation but ran into a structural problem at high resolutions. Because U1 used an independent MLP head to decode each 32×32 visual token separately, neighboring tokens could not share information during the final stage of image formation. At standard resolutions this produced acceptable results. At 4K, the artifacts became pronounced: seams at patch boundaries, texture discontinuities across edges, and geometric inconsistencies that grew worse as resolution increased.

U1.5 replaces the per-patch MLP with a lightweight spatial decoder. The process works in stages: the model first reshapes the backbone's hidden state sequence back into a two-dimensional spatial grid, then applies Pixel Shuffle upsampling in three stages with upsampling factors of 2, 2, and 8. Between each upsampling stage, 3×3 convolutions allow information from neighboring token regions to propagate before pixels are finalized. The result is that color, texture, and geometry near patch boundaries are determined jointly rather than independently, substantially reducing seam artifacts while adding only modest computational overhead. The model now supports native resolution synthesis from 512×512 through 4096×4096.

Resolution itself also becomes an explicit conditioning signal. As image size increases, the effective noise magnitude changes, and a naive model trained at one resolution generalizes poorly to another. U1.5 addresses this with a resolution-aware noise-scale embedding, extending the normalization reference resolution from U1's 2048×2048 to 4096×4096. This embedding is added directly to the diffusion timestep representation at each forward pass, giving the denoising network explicit awareness of resolution-dependent noise statistics.

A Specialize-Then-Unify Post-Training Strategy

The most structurally novel contribution of the paper is not the architecture change but the post-training approach, which the team describes as specialize-then-unify. Visual creation spans tasks with fundamentally different reward signals: aesthetic preference, text legibility, complex layout design, and fine-grained image editing each require distinct optimization incentives. Jointly optimizing all four inside one policy, the authors argue, entangles competing gradients and dilutes task-specific gains.

The solution is to train four separate expert models using reinforcement learning, then merge their capabilities into the base model through multi-expert on-policy distillation.

The four RL experts are each trained with task-specific data, reward functions, sampling strategies, and regularization. The Aesthetic Expert uses the HPSv3++ human preference reward model combined with PaddleOCR text-fidelity scoring, with Coefficients-Preserving Sampling (CPS, η=0.7) to introduce stochastic exploration during rollout. The OCR Expert specializes in bilingual text rendering for Chinese and English, using a multiset intersection-over-union reward that penalizes missing, duplicated, or malformed characters. The Editing Expert trains on a five-dimensional reward that separately measures instruction fulfillment, visual quality of edits, preservation of unmodified regions, and text rendering where applicable — aggregated by taking the minimum across dimensions, so no one strong score can mask a critical failure. The Infographic Expert undergoes a three-stage process: an OCR warmup phase, a Direct Preference Optimization (DPO) phase for aesthetic quality on curated preference pairs, and a final alternating-reward phase combining OCR and HPSv3++.

Once the four experts are trained and frozen, on-policy distillation transfers their capabilities into the base model. The student generates its own denoising trajectories; at each sampled timestep, the student's predicted velocity field is compared to the frozen expert's prediction from the same starting state. The loss minimizes the L2 distance between these velocity fields under a stop-gradient on the student trajectory, preventing full backpropagation through the denoising chain. A progressive Beta-distribution timestep scheduling protocol gradually shifts optimization emphasis from early high-noise steps — which establish global composition — to later low-noise steps that refine text fidelity and fine detail. This prevents the student from overfitting to coarse structure while neglecting typographic quality.

Where SenseNova-U1.5 Sits in the Competitive Field

The field of unified multimodal models has been moving in two directions simultaneously. Modular-hybrid models such as Janus-Pro, BAGEL, Show-o, and OmniGen2 keep separate encoder and VAE components but attempt to unify understanding and generation through shared attention or routing mechanisms. These have achieved strong benchmark numbers; OmniGen2 at 4B parameters reported a DPG-Bench overall score of 88.81 in that model's own evaluation. Native models — processing raw pixels without an intermediate tokenizer — form a smaller but growing group. SenseNova-U1.5 is the most thoroughly documented 8B native model with public weights to date.

On benchmarks reported by SenseTime for U1 (the immediate predecessor), the model achieved a GenEval compositional generation score of 0.91, outperforming Qwen-Image at 0.87, BAGEL at 0.82, and FLUX.1-Dev at 0.82 in the same evaluation, according to the team's own tables. For text rendering, U1 achieved 0.940 word accuracy on CVTG-2K, a Chinese-English multi-region text benchmark. These figures come from company-run evaluations and have not been independently reproduced by third parties as of the date of this article. U1.5 benchmark comparisons against closed-source models (GPT-Image-2, Qwen-Image2-Pro, Seedream 5.0) appear in the paper's tables, with the U1.5 model with prompt enhancement scoring 60.13 on the Qwen-Image-Bench overall metric, above the closed-source GPT-Image-1.5 at 59.65 — but again, these figures are company-reported and should be treated as claims pending independent reproduction.

The most meaningful competitive gap is not on headline benchmarks but on capability coverage at 8B scale. Most 8B-class open-source models perform either understanding or generation well; fewer handle both. SenseNova-U1.5 claims to cover image understanding, generation at up to 4K, instruction-following image editing with multi-reference inputs, interleaved text-image generation, and bilingual typography — all within one model checkpoint.

What Remains Unverified and Where the Technology Still Falls Short

Several claims in the paper require independent evaluation before they can be taken as established results. Benchmark scores for U1.5 on editing benchmarks (ImgEdit, GEdit-Bench, WeEdit, OmniRef-Bench, RISEBench) and interleaved generation benchmarks (OpenING, VBVR-Pro-Bench) appear in the paper but are company-run. The paper does not include a third-party or external evaluation group, which is standard for technical reports at this stage but limits confidence in comparative claims.

The training data composition is described in general terms — "carefully curated generation and editing data," "high-quality real and synthetic data" — but exact dataset provenance, size, and licensing are not fully enumerated in the publicly available version of the paper. Researchers seeking to reproduce the training pipeline will need this information before the open-source code release is complete.

The paper also notes architectural limitations. The Attribute Binding dimension of GenEval — which tests whether generated images correctly assign attributes to the right objects in compositional scenes — remained below the performance of OneCAT and Mogao on that specific benchmark in U1. It is unclear whether U1.5 closes this gap fully. The model's understanding benchmark scores for U1.5 specifically (as opposed to U1) are not surfaced in the portions of the paper available for review. Finally, the model's safety evaluation methodology is not described; there is no red-team report or adversarial robustness disclosure in the arXiv submission.

Training Code Release and What Comes Next

SenseTime has stated in the paper that training code — covering supervised fine-tuning, reinforcement learning post-training, and on-policy distillation — will be open-sourced. A timeline is not specified. The GitHub repository for the SenseNova-U1 series already hosts configuration files, inference code, and ComfyUI workflow integrations; the training code release would substantially expand reproducibility for the research community, since the multi-expert RL and distillation pipeline described in the paper is significantly more complex than standard SFT workflows.

The paper's conclusion identifies a direction that extends beyond image generation: SenseNova-U1 already showed preliminary results on vision-language-action (VLA) tasks and world modeling, suggesting that a natively unified representation could eventually serve as a foundation for embodied AI systems — robots that need to perceive, reason, and act within a shared computational architecture. U1.5 narrows the gap between what current modular systems achieve in image synthesis quality and what encoder-free architectures can produce, which matters most if the long-term goal is a single representation that covers perception, imagination, planning, and physical action. Whether the specialize-then-unify post-training strategy scales to those downstream settings, or whether it produces experts that are difficult to consolidate without task interference, is a question the next generation of experiments will need to answer.