H company introduces neomme: a new multimodal encoder family without vision tower and causal decoder

H Company has unveiled NeoMME, a family of multimodal encoders with 260M and 800M parameters, designed to eliminate the vision tower and causal decoder typically found in similar models. The encoders process both text and 32×32 RGB image patches through a single Transformer, trained from random initialization. NeoMME-Retriever, a retrieval fine-tuned version, achieves 0.523 nDCG@10 on ViDoRe v3 at 260M parameters. The 260M model indexes 51.3 pages per second on an NVIDIA L40S and encodes queries in 78.3 ms on a CPU-only host. The model supports a 16,384-token context and uses symmetric sliding-window attention, grouped-query attention, and squared-ReLU MLPs. The tokenizer, a BPE with a 131,072-entry vocabulary, emits 44.4% fewer tokens than ModernBERT across 14 languages. Pretraining involves discrete masked diffusion over text, optionally conditioned on visible image patches. The 260M model scores 0.523 nDCG@10 on ViDoRe v3, slightly below ColQwen2.5-v0.2 but significantly above other sub-300M models. Indexing is fast, with NeoMME-260M encoding 51.3 pages per second on an L40S, outperforming ColModernVBERT by 1.97×.