Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

[marcobambini]

Article URL: GitHub - sqliteai/waste: Run the full 2.78-trillion-parameter Kimi K3 model beyond available RAM by streaming activated weights directly from NVMe. A dependency-free, embeddable C inference engine.

Comments URL: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s | Hacker News

Points: 155

# Comments: 61

Continue reading...