We explore coverless steganography using a Large Language Model (LLM) to guide an arithmetic coding decoder in generating stego-texts, aiming to embed secret bits efficiently while maintaining natural language fluency. Our approach optimizes a replacement probability distribution under a KL divergence constraint, offering a closed-form solution and addressing practical challenges such as tokenization mismatch, vocabulary truncation, and integration with previous stega methods like Discop.