Public
- Public
- Network
- Groups
- Featured
- Popular
- People

Conversation

Notices

Embed this notice
counternotions (counternotions@mastodon.social)'s status on Tuesday, 26-Nov-2024 05:18:23 JST counternotions

The incredible lightness of ‘weightless bits and bytes’.
↓
In conversation about 5 months ago from mastodon.social permalink
Attachments
1. “Today’s largest AI clusters use around 100,000 Nvidia H100 chips and consume roughly 100 megawatts of power. Next year's clusters will scale to 300,000 to 500,000 chips requiring nearly a gigawatt of power about the consumption of a small city,” the firm said. That’s imposing real physical constraints, as “the tech giants who built empires on weightless bits and bytes are now grappling with atoms steel, copper, water rights, and critically, natural gas,” according to the firm. […] “Apple has quietly published research showing how to run large language models directly on devices with limited memory. By leveraging novel storage techniques and the inherent sparsity in AI models, the company can reduce memory requirements by 50% while maintaining performance. Advances like these could enable future iPhones and Macs to run sophisticated AI models locally and privately a development that could radically reshape the entire Al infrastructure landscape,” the firm said.
  https://files.mastodon.social/media_attachments/files/113/545/379/257/010/733/original/b006945cf610fdaf.jpg

Feeds