The letter

One letter a week, starting soon.

The week's signal, what shipped, what is worth running tonight, and the editor's note. No tracking, no ads, nothing else.

We keep your address, your language and the date you joined, nothing else. Every letter has a one-click unsubscribe link that deletes the record.

← Accept All   Archive
PyTorch BlogTools

Training a 1T-Parameter MoE on EU public infrastructure with PyTorch and TorchTitan

11.12

TL;DR Scaling pretraining for a ~1-trillion-parameter Mixture-of-Experts (MoE) model on shared, public infrastructure was previously considered unfeasible due to unique architectural and interconnect…

Read at PyTorch Blog ↗

More from PyTorch Blog on Accept All

More from PyTorch Blog on Accept All.