Viewing a single comment thread. View all comments

maizeq t1_j66b3l5 wrote

Reply to comment by data-drone in [R] SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot by Secure-Technology-78

Chinchilla (70B) is trained with 1.4 trillion, so 140B would presumably need at least 2.8 trillion (it scales linearly afaik).

I’m not sure a 2.8 trillion token dataset actually exists

3

rainy_moon_bear t1_j676oo9 wrote

This is something people don't seem to understand. Pretty much all models 100B+ are undertrained.

3

Taenk t1_j688cev wrote

> I’m not sure a 2.8 trillion token dataset actually exists

DeepMind's Massive Text is assumed to be 10TB large, the largest publically available dataset is The Pile and weighs in at about 820GB.

A 2.8 trillion token dataset would need to be more than 20TB large, which could be possible by including more of Common Crawl - weighing in at 380TiB - or non-English resources. I have a suspicion that training LLMs on more languages, especially outside of the Indo-European family, will improve performance within the Indo-European family.

2

maizeq t1_j69vuec wrote

Nice. How are you converting between dataset size and number of tokens?

Doesn’t common crawl get deduplicated and that’s why the number of usable tokens decreases - or is it also curation? How much of that 380TiB is actually utilisable.

Given the ostensibly impressive performance of the bilingual GLM-130B (Chinese+English) model that came out of Tsinghua university that might very well be the case.

1