adt t1_jbbzba8 wrote on March 7, 2023 at 11:06 PM

Reply to comment by __Maximum__ in [D] Can someone explain the discrepancy between the findings of LLaMA and Chinchilla? by __Maximum__

There are a few that 'feel' that way. Try Megatron-11B (~200:1) based on RoBERTa (6,198:1). Wayyyyy ahead of its time, and I've matched it with much larger models in some testing.

https://app.inferkit.com/demo

Here's the full table of Chinchilla-align comparisons:

https://lifearchitect.ai/models-table/

whata_wonderful_day t1_jbcxdwf wrote on March 8, 2023 at 3:23 AM

Nice! How did you get access to Megatron-11B? I can't find it online anywhere

Jepacor t1_jbdrovb wrote on March 8, 2023 at 8:57 AM

The link to the model is in the Google sheets they linked : https://github.com/facebookresearch/fairseq/blob/main/examples/megatron_11b/README.md

whata_wonderful_day t1_jbhp4gb wrote on March 9, 2023 at 2:53 AM

Thanks, alas I thought it was an encoder model. I've been on the lookout for a big one, largest I've seen is deberta V2 with 1.5B params

Maximum OP t1_jbdqy5c wrote on March 8, 2023 at 8:47 AM

Thanks for the links. Looks like RoBERTa did not gain a lot from the additional trainings, only minor improvements, but yeah, it was a tiny model. How was this not a good lesson? Why did people need Chinchilla? Maybe it's just having a lot of data comes easy so people gather as much as possible, even though they know they will go maximum 1 epoch over it.