Construct the text latent
Learn a high-capacity, prefix-ordered latent sequence whose successive positions encode progressively longer token prefixes.
Abstract
Language remains an outlier in modern generative modeling: while images, video, and audio are increasingly modeled in continuous latent spaces, text generation still relies predominantly on discrete tokens. Existing continuous language models either inherit embedding spaces not designed for generation and decoding jointly, or compress autoencoded latents to make diffusion easier at the cost of token-level fidelity. We challenge this prevailing design compromise. Instead of simplifying the representation to accommodate the generative model, we preserve a high-capacity, decodable text latent and design the diffusion model to learn its distribution directly.
We introduce AURORA-LM, a continuous-latent diffusion language model that separates the construction of a decodable text representation from the modeling of its generative distribution. To obtain such a representation, we use a Query-based Encoder-Decoder that organizes text into a high-capacity, prefix-aligned latent sequence. We then introduce a Block-causal Diffusion Transformer that learns the distribution of these full-width latents through flow matching, generating blocks from left to right while denoising the positions within each block in parallel. However, retaining a high-capacity latent representation for accurate token decoding also makes its distribution more challenging for the diffusion model to learn. AURORA-LM addresses this difficulty by restricting only the noisy-input pathway while retaining the full clean-latent prediction target, allowing the generative model to accommodate the full-width latent without reducing decoder-facing capacity. We further calibrate the noise-level distribution to the latent width, accounting for how the effective signal strength changes with representation dimensionality. Finally, we introduce self-trajectory consistency to bridge the gap between training on independently sampled noisy states and inference through iterative denoising.
Across comprehensive comparisons, AURORA-LM achieves the strongest performance among the evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum conditional summarization. Scaling to 1B parameters with approximately 1,500 EFLOPs of total compute yields further gains and surpasses a larger publicly released latent-diffusion language model under a matched evaluation protocol. Our results demonstrate that continuous language generation can effectively bridge diffusion-based generative modeling and discrete token decoding through a high-capacity, causally structured, and decodable text representation. All experiments are conducted on Ascend NPUs.
The AURORA-LM framework overview
A high-capacity, decodable, prefix-ordered continuous text latent.
And distribution modeling designed around it.
AURORA-LM treats continuous language generation as two connected problems. It first constructs a decodable, causally ordered text latent, then freezes that interface and trains a block-causal denoiser to model the induced latent distribution.
Model the induced latent distribution
Standardize the encoder outputs and train a block-causal denoiser with flow matching.
Generate and decode
Denoise latent blocks from left to right, invert standardization, and recover token logits with the fixed decoder.
Continuous text latent construction
A prefix-ordered latent connects continuous generation to discrete token recovery.
The representation must preserve enough information for accurate decoding while organizing latent positions in an order that supports blockwise left-to-right generation. AURORA-LM learns this interface rather than inheriting it from token embeddings or a pretrained encoder.
Write expanding token prefixes into latent positions.
Successive latent queries attend to progressively longer token prefixes and preceding latent states. The resulting latent sequence accumulates text from left to right.
Recover each token from the matching latent prefix.
The decoder receives learned output queries rather than the original tokens. Its matched visibility constraint forces token content to be recoverable from the corresponding latent prefix.
Block-causal latent modeling
Model the latent prior with sequential conditioning across blocks and parallel denoising within each block.
The prefix order learned by the autoencoder induces a block-causal factorization of the latent prior. Each target block is conditioned on all preceding clean blocks, while positions inside the current block are denoised jointly.
Condition on completed prefixes
The model represents the joint latent distribution as a product of block conditionals.
Noise every target block
Each clean block is independently placed on a linear flow-matching path from data to Gaussian noise.
Evaluate all blocks together
A two-stream attention mask exposes each noisy block only to itself and its clean prefix, enabling parallel teacher-forced training.
Further study · latent capacity
Why preserve a high-capacity latent?
Further studies examine how much information the decoder-facing latent should retain while remaining suitable for generative modeling, through channel capacity and sequence retention.
All widths reconstruct clean inputs, but wider latents preserve exact token information over a much broader corruption range.
After calibrating the noise schedule for each width, higher-capacity latents yield consistently stronger generation quality across the tested widths.
Sequence compression reduces the number of positions the prior generates, offering a controllable efficiency–quality trade-off. Moderate compression incurs only a small reduction, whereas aggressive compression lowers MAUVE.
Learning the full-width latent distribution
Model the full-width latent distribution without sacrificing decoding fidelity.
The frozen decoder requires a full-width clean target for accurate token recovery. AURORA-LM therefore changes how the denoiser reads and learns from corrupted latents, rather than compressing the clean representation itself.
A low-rank projection is applied only to the corrupted input before Transformer processing. The output head continues to predict the full clean latent.
All block conditionals are trained in parallel to directly recover clean latents. The noise-level distribution is calibrated to the selected latent width.
A detached draft clean-latent estimate is fed back to the main prediction pass, exposing the denoiser to the conditioning signal used during iterative inference.
Design evidence · full-width modeling
What enables full-width latent modeling without compressing the decoder-facing latent?
With the clean target held fixed, controlled ablations isolate the prior-side choices: noisy-input width, noise allocation, prediction target, and loss space.
A very narrow pathway loses useful information, but reading the corrupted state at full width brings no consistent gain. A moderate bottleneck preserves the full clean target while simplifying the noisy input.
Across tan-d and logit-normal schedules, MAUVE rises as more training mass is assigned to high-noise states. The final system adopts a high-noise calibrated schedule.
Predicting x0 with an x0-space loss performs best under both tested noisy-input widths.
| Target | Loss | Bottlenecked input | Full-width input |
|---|---|---|---|
| x0 | x0 | 0.815 | 0.807 |
| x0 | v | 0.059 | 0.017 |
| v | x0 | 0.729 | 0.325 |
| v | v | 0.653 | 0.301 |
Self-trajectory consistency
Align neighboring predictions along the trajectory the model actually follows.
One model-induced update
t′ = PrevS(t) < t
α̃t′ = (t′/t) αt + (1 − t′/t) sg(ẑθ(αt, t))
Lct = || ẑθ(αt, t) − sg(ẑθEMA(α̃t′, t′)) ||22
Flow matching anchors each prediction to the clean training target. The consistency term makes adjacent solver states agree on that target, with gradients flowing only through the online prediction.
Flow matching trains on independently sampled noisy states, but inference repeatedly follows a single model-induced denoising trajectory. Errors can therefore accumulate when clean-latent predictions disagree between successive solver states.
Self-trajectory consistency aligns the online model's clean estimate before and after one detached Euler update. A slowly moving EMA model provides the stable target at the neighboring lower-noise state.
Latent generation and text decoding
Generate blocks left to right while denoising positions within each block in parallel.
During training, each target block conditions on encoder-produced clean prefix blocks. At inference, completed generated blocks replace that prefix; for conditional generation, the encoded prompt initializes the prefix. The denoiser transforms a newly sampled Gaussian block from noise to a clean latent while conditioning on all completed blocks.
Blockwise latent sampling
Unconditional generation starts without a prefix; conditional generation initializes the prefix by encoding the prompt.
Control the trajectory
SC-CFG guides unconditional generation through self-conditioning, while standard classifier-free guidance strengthens prompt conditioning.
Return to token space
After all blocks are generated, inverse standardization restores the decoder-compatible latent and the frozen query decoder produces token logits.
Design evidence · efficient blockwise generation
Few-step quality and block granularity shape blockwise generation.
These studies examine two factors behind efficient blockwise generation: self-trajectory consistency, a training-time objective that stabilizes short solver trajectories, and block granularity, which trades sequential conditioning against within-block parallelism at inference.
Consistency improves MAUVE at every tested step budget, with the largest gains in the few-step regime. Its advantage narrows as the solver trajectory becomes more finely discretized.
Smaller blocks provide a more complete generated prefix but reduce parallelism. The selected Q = 16 reaches 0.816 MAUVE while reducing sequential block-generation stages from 32 to 8 relative to Q = 4.
System-level evaluation
Results across generation settings and model scales.
We evaluate AURORA-LM-S on OpenWebText free generation and XSum conditional summarization, then examine AURORA-LM-L on a suite of nine public language benchmarks. Reported parameter counts refer only to the block-causal denoiser.
We compare AURORA-LM-S with autoregressive, discrete-diffusion, and continuous-flow baselines on 1,024-token OpenWebText free generation. It achieves the lowest Gen-PPL and highest MAUVE among the evaluated systems. We also train and evaluate AURORA-LM-S on XSum conditional summarization, where it attains the best ROUGE scores across all three metrics.
We next examine whether the formulation remains effective when the block-causal denoiser is scaled to approximately 1B parameters. AURORA-LM-L is trained with approximately 1,500 EFLOPs of total training compute. Under a shared nine-task generative prompt-and-matching protocol, it surpasses a larger publicly released latent-diffusion language model.
Detailed results
The tables reproduce the reported comparisons. OpenWebText baselines are evaluated in our pipeline with 1,000 generated samples per evaluation seed; XSum values marked † are reported by ELF. Model-size and sampling details are provided in the paper appendix.
OpenWebText unconditional generation1,024-token free generation. Lower Gen-PPL is better; higher entropy and MAUVE are better.
| Model | Gen-PPL ↓ | Entropy ↑ | MAUVE ↑ |
|---|---|---|---|
| AR | 39.40 | 5.605 | 0.851 |
| Duo | 86.57 | 5.566 | 0.704 |
| Duo-distilled | 78.22 | 5.574 | 0.715 |
| SEDD | 119.82 | 5.644 | 0.693 |
| MDLM | 121.36 | 5.664 | 0.668 |
| ELF-B | 24.11 | 5.155 | 0.229 |
| AURORA-LM-S | 23.56 | 5.241 | 0.890 |
XSum conditional generationROUGE-1, ROUGE-2, and ROUGE-L; higher is better. † Values collected by ELF.
| Model | R-1 ↑ | R-2 ↑ | R-L ↑ |
|---|---|---|---|
| ELF-B† | 36.0 | 12.2 | 27.8 |
| AR† | 30.5 | 10.2 | 24.4 |
| MDLM† | 33.4 | 11.6 | 25.8 |
| Duo† | 31.4 | 10.1 | 25.0 |
| E2D2† | 28.4 | 8.3 | 22.0 |
| SeqDiffuSeq† | 19.3 | 1.7 | 14.1 |
| AURORA-LM-S | 36.6 | 13.4 | 28.9 |
Nine-task benchmark evaluationAll entries are percentages. Avg is the macro average over the nine tasks; higher is better for every metric.
| Model | Avg ↑ | MMLU | ARC-C | OBQA | HellaSwag | WinoGrande | StoryCloze | SIQA | RACE | SQuAD EM |
|---|---|---|---|---|---|---|---|---|---|---|
| AURORA-LM-L (1B) | 32.6 | 22.2 | 21.2 | 27.8 | 18.4 | 50.3 | 54.8 | 30.2 | 30.6 | 38.2 |
| Cola-DLM (1.8B) | 25.1 | 19.6 | 20.6 | 24.2 | 5.7 | 45.3 | 33.8 | 26.8 | 24.2 | 25.7 |
Conclusion
AURORA-LM decouples representation learning from distribution modeling. A Query-based Encoder-Decoder first constructs a high-capacity, causally ordered latent sequence for accurate token recovery. The autoencoder is then frozen, and a block-causal denoiser learns the resulting full-width latent distribution through flow matching.
The controlled studies explain why the final system takes this form: wider latents preserve token information and support stronger generation; a moderate bottleneck simplifies only the noisy-input pathway; high-noise calibration and direct clean-latent regression improve full-width modeling; and self-trajectory consistency stabilizes few-step generation. Blockwise sampling then balances left-to-right conditioning with parallel denoising.
Together, these results establish a continuous interface between diffusion-based generation and discrete token decoding, and provide a foundation for scaling continuous language models to longer contexts, broader language capabilities, and eventually unified generation across language and other continuous modalities.