Viewing a single comment thread. View all comments

timedacorn369 t1_jcqg4v6 wrote

Reply to comment by simpleuserhere in [Research] Alpaca 7B language model running on my Pixel 7 by simpleuserhere

What is the performance hit with various levels of quantization??

10

starstruckmon t1_jcrbf0m wrote

You can see some benchmarks here

https://github.com/qwopqwop200/GPTQ-for-LLaMa

11

Taenk t1_jcs53iw wrote

The results for LLaMA-33B quantised to 3bit are rather interesting. That would be an extremely potent LLM capable of running on consumer hardware. Pity that there are no test results for the 2bit version.

3

starstruckmon t1_jcswg1g wrote

I've heard from some experienced testers that the 33B model is shockingly bad compared to even the 13B one. Despite what the benchmarks say. That we should either use the 65B one ( very good apparently ) or stick to 13B/7B. Not because of any technical reason but random luck/chance involved with training these models and the resultant quality.

I wonder if there's any truth to it. If you've tested it yourself, I'd love to hear what you thought.

5

Taenk t1_jctdmvi wrote

I haven’t tried the larger models unfortunately. However I wonder how the model could be „shockingly bad“ despite having almost three times the parameter count.

2

starstruckmon t1_jcte34d wrote

🤷

Sometimes models just come out crap. Like BLOOM which has almost the same number of parameters as GPT3, but is absolute garbage in any practical use case. Like a kid from two smart parents that turns out dumb. Just blind chance.

Or they could be wrong. 🤷

3