Archive — US — techcrunch
Kog is going deeper to squeeze more inference out of GPUs
Kog, a French startup, seeks to enhance AI inference speeds on conventional GPUs through software optimization. Gaining traction after a successful demo, Kog anticipates targeting enterprises burdened by slow AI workflows. Despite skepticism, CEO Gaël Delalleau believes their software can achieve significant acceleration on larger models, although full potential still awaits demonstration to secure more funding.
Key points
- Kog aims to leverage conventional GPUs (like AMD MI300X and NVIDIA H200) for faster AI inference.
- The startup's demo showed impressive TPS (3,000) using a small 2 billion parameter model.
- CEO Gaël Delalleau emphasizes that GPUs can outperform expectations with proper optimization.
- Kog's early feedback includes 200 business leads focusing on accelerating AI workflows.
- Delalleau's background in solid-state physics and cybersecurity aids in their GPU engineering approach.
- Kog plans to support larger models and aims for a 10x speed demonstration by September to secure more funding.
Why it matters
- The growing demand for faster AI inference reflects broader trends in AI model complexities.
- Kog's strategy aligns with the 2025 trend of optimizing existing hardware for efficiency over solely developing new chips.
- Other companies like ZML are also pursuing hardware-agnostic solutions, indicating a shift in the industry.
Kog's innovative approach aims to reshape GPU utility in AI, showcasing potential for significant advancements in inference speed as the landscape evolves, proving imperative for future funding and market sustainability.