- cross-posted to:
- stupidpol@sh.itjust.works
- fuck_ai@lemmy.world
- cross-posted to:
- stupidpol@sh.itjust.works
- fuck_ai@lemmy.world
I have not read the article yet but i think this is a good topic to discuss here.
I have not read the article yet but i think this is a good topic to discuss here.
mixture-of-experts but the experts are the five heads.
this is technically possible already, just that it takes so much resources that not many people could make this model and the cost would be prohibitive for what is a hobby project. This is why China is developing forces of production first.
It might be cheap enough to make a LoRA actually. It would still probably take a week or so, but doable. :)
we have to do this actually
https://github.com/percy-raskova/marxists.org-rag-db
It is in progress, if you’d like to help develop this let me know!