Comments

You must log in or register to comment.

Cheap_Meeting t1_itjeomy wrote

As far as I know, there is no language model that runs in 4 seconds on a CPU and has competitive results.

9

ethereumturk t1_itj6hx2 wrote

SBert biencoder

3

fastglow t1_itlyz8a wrote

SBert is for comparing sentences/phrases. No reason to use that over a regular transformer encoder-decoder for language modelling, and getting that to process 2000 tokens in less than 4 seconds would be challenging without efficiency-augmenting methods like quantization, pruning, distillation, etc.

2

uniklas t1_itkdu21 wrote

Depends on what you want to do, as for metrics it will depend completely on your cpu.

For example if your task in any way related to comparing two strings you may use sbert models, which have their performance metrics listed here https://www.sbert.net/docs/pretrained_models.html

2

leepenkman t1_itnz3pn wrote

Bloom/gpt neo/codegen is a good one and all come in various sizes, CPUs are slow though vs GPUs even old GPUs.

Also try https://text-generator.io which is a much more cost effective way to prototype new tools/what's possible right now, way cheaper than OpenAi, does text and code and a 100 requests a month free teir, then you can drop down to doing it yourself once your at a large scale.

Text generator runs under a second normally on fast GPUs

1

hawkxor t1_itw58sy wrote

How does the quality of text-generator.io compare?

1

Fabiano_Dicheti t1_itj8c91 wrote

If you are talking about text recognition Just do a Recurrent Neural Network in C++

−9