UK — The Register
PyTorch 2.15 tackles the 'one bad GPU' problem
The PyTorch 2.15 release includes a rewritten backend for Nvidia's NCCL library, aimed at improving fault diagnosis and recovery when GPUs work together. The new backend allows only one communicator per process group, can exclude slow or crashed GPUs with a third-party orchestrator, and adds a "flight recorder" that captures operations for inspection after failures. The legacy backend remains as a rollback option in 2.15 and is slated for removal in the 2.16 release, planned for early next year.