[D] Is anyone working on ML models that infer and train at the same time?

Submitted by Cogwheel t3_113448t in MachineLearning

In brains, the neural networks are transformed by the act of "inference". Neurons that have recently fired are more likely to fire again given the same input. Individual neural pathways can be created or destroyed based on the behavior of neurons around them. This leads me (through various leaps of logic and "faith") to suspect that some amount of mutability over time is required for an AI to exhibit sentience.

So far, all of the ML models I've seen distinctly separate training from inference. Every model that we put into production is a fixed snapshot of the most recent round of training. ChatGPT, for instance, is just the same exact model being incrementally fed both your prompts and its own previous output. This does create a sort of feedback, but in my mind it is not actually "experiencing" the conversation with you.

So I'm wondering if there are any serious attempts in the works to create an AI that is able to transform itself dynamically. E.g. having some kind of reinforcement learning module built into inference so that each new inference fundamentally (rather than superficially) incorporates its past experiences into its future predictions.

12

Comments

You must log in or register to comment.

terath t1_j8oemyz wrote

Another key phrase to use with google scholar is "online learning", this is where you have a stream of new examples and you update a model one example at a time. Usually you can use the model for inference at any point in this process, and some algorithms in this area are designed to be a bit more aggressive or at least to control the update rates to more quickly more more slowly adapt to new data.

21

cantfindaname2take t1_j8omdij wrote

This! Online learning is a very common term that is used in time series modeling, for example in anomaly or change point detection.

6

CabSauce t1_j8nuthg wrote

You can update a model with new data at any time. Production models are often updated at intervals with monitoring. There are a few challenges to updating continuously.

  1. Distributed models would have to be updated. How do we update weights from two sources? (There might be options for this, I haven't looked.)
  2. Potential for undesirable and unstable predictions/generations.
  3. I think you'd have to allow the weights to update pretty dramatically at each inference to get any real variation. I think this would lead to #2
  4. Attention components probably do what you're looking for more accurately and efficiently.
15

currentscurrents t1_j8p83n7 wrote

>Distributed models would have to be updated. How do we update weights from two sources? (There might be options for this, I haven't looked.)

This sounds like federated learning.

6

Cogwheel OP t1_j8o29pz wrote

>1. Distributed models would have to be updated. How do we update weights from two sources? (There might be options for this, I haven't looked.)

This strikes me as more of a software/hardware engineering challenge rather than one of network and training architecture. Definitely a challenge though.

>2. Potential for undesirable and unstable predictions/generations.

I think the same is true for humans. Given enough "perverse" inputs we can all go crazy. So it's definitely something to think about and mitigate. There would definitely need to be components built to work against these "forces"

>3. I think you'd have to allow the weights to update pretty dramatically at each inference to get any real variation. I think this would lead to #2

Interesting point... The time between acts of inference in an ML model are on the order of clocks (milliseconds for realtime perception systems, seconds to minutes for things like ChatGPT). Whereas animals experience essentially continuous input. Our eyes alone present us with many Mbps of data, is it were.

So without these vast swathes of data constantly being fed in, the alternative is to make bigger changes based on the limited data.

>4. Attention components probably do what you're looking for more accurately and efficiently.

Attention had crossed my mind when I posted this. I agree its intention is to accomplish a kind of weight redistribution based on previous input. But I still think this is more superficial/ephemeral than what I'm asking about. Humans certainly have attention mechanisms in our brains, but those attention mechanisms are subject to the same kinds of changes over time as the rest.

2

crt09 t1_j8o2yc4 wrote

I think what you're looking for is the ML field called "continual learning", where an agent is put into practice and learns in the real world how to improve as it acts in it (from my understanding). afaik its one of the more experimental ones

6

Cogwheel OP t1_j8o4qci wrote

Thanks! This seems to be the term I was looking for.

1

Binliner42 t1_j8p5iqj wrote

Reinforcement learning?

5

HyugenAI t1_j8o211j wrote

>I'm wondering if there are any serious attempts in the works to create an AI that is able to transform itself dynamically

I'm not sure what you call a "serious attempt". I programmed neural networks that try to do that for a specific task.

I can try to summarize it. Basically I have 3 models:

  • image > (model 1) > image embedding (self supervised)
  • sound > (model 2a) > sound embedding > (model 2b) > sound (autoencoder)
  • image embedding > (model 3) > sound embedding (association between embeddings based on temporal proximity)

When I launch the program, all models train and infer "simultaneously". The input is a video (or multiple images / sounds, or a livestream). Model1 reads multiple images and train on these images, it doesn't need labels. Model2 does the same thing for sounds. While Model1 and Model2 train, they produce embeddings. Model3 trains on these embeddings.

So the global model is able to associate images and sounds, without supervision, and models train and infer simultaneously. They improve themselves continously, and you can point the camera towards an object, say the name of the object, and after perhaps 1minute the model will have learned the association and will "repeat" the name of the object just from the image.

For inference the path is:

  • image > (model1) > image embedding > (model3) > sound embedding > (model2b) > sound

While models train, they simultaneously do this path. So they train and infer "simultaneously", and the model doesn't explicitly need labels / supervision. Though it needs the temporal association between the name of the object and the image of an object (which is probably what we also first used to learn words). Also I say "simultaneously" because in the code, it's just a while loop with two instructions: (1) a model does a backprop (2) it infers. It's not exactly at the same time, there are still two separate processes, but it's the same while loop. Though Model1 can train while Model2 infers. You could consider that Model1+Model2+Model3 are just one big model.

  • Does it reproduce our brain regarding how neurons work? No. So it doesn't reproduce what you described regarding how neurons work.
  • Is it sentient? It depends on what you think "sentience" is. It's probably not, based on what we usually call "sentience".

What most models are doing now is much more efficient, practical and reliable than what I described. Though it doesn't exactly reproduce how we learn things. But that's probably not what most people would want in their models. They prefer more efficient, pratical and reliable models. If a model trains continously, it's much harder to check that it continues to have good results.

3

Cogwheel OP t1_j8o2st4 wrote

> What most models are doing now is much more efficient, practical and reliable than what I described. Though it doesn't exactly reproduce how we learn things. But that's probably not what most people would want in their models. They prefer more efficient, pratical and reliable models.

Yeah, I guess the distinction here is whether one is using an ML model as a means to an end or as an end in itself. I imagine a researcher interested in AGI would be much more likely to take this kind of approach than someone trying to sell their ML models to industry.

Edit: anyone care to discuss why you downvoted?

−2

HyugenAI t1_j8o524o wrote

I agree, and I did that because I'm interested in AGI. It would probably be quite hard to build a business plan around this model, and I wasn't funded to build it.

1

sid_276 t1_j8q4yfx wrote

It depends on where in the ML spectrum you are. In RL it’s common to set agents to set some fraction of their time exploring and some other exploiting the environment. In neural nets there is the whole “online learning” field that addresses just that. It is generally possible but not always practical. There are other ways to update information. You mention ChatGPT. One way is giving them access to browsing to provide updated results. Technically one could retrain it on the conversations. I believe they will do it. But practically it makes more sense making it in batches e. g. Once a week or when a lot of new data has been accumulated. But yeah if you Google (or Bing lol) Online learning you will find a lot of papers

2

ML4Bratwurst t1_j8qu836 wrote

Yes. It's called online learning

2

PredictorX1 t1_j8ntphr wrote

Some modeling algorithms (naive bayes and local models, like k-nearest neighbor or kernel regression) can be updated immediately. In some sense, they can be used for recall and training very close in time.

1

deitscherdeifl t1_j8om1z7 wrote

Im not deep in it, but maybe hierarchical temporal memory by numenta ist interesting for you

1

BrotherAmazing t1_j8ps3he wrote

Yes, a lot of people are doing this and have done this kind of thing using very different approaches. The field of lifelong machine learning is just one relevant area.

1

teb311 t1_j8qdjqv wrote

It’s not popular to do “online learning” for a variety of reasons. u/CabSauce gave a nice list. One reason I wanted to add was that many models are exposed to relatively uncontrolled input and that can backfire badly. Google “Microsoft Tay Twitter” for a cautionary tale. Garbage in garbage out; letting your model learn in an uncontrolled environment risks inputting (lots of) garbage, and sometimes even malicious/adversarial data. Making matters worse, since the garbage affects the model in real time the actively-getting-worse predictions just get made/published/used in a production setting.

In most cases the upside to continuous learning is small compared to batched releases, but it makes a lot of stuff harder and more risky.

1

Jeff-Galliot t1_j8s982g wrote

Do you mean other than real-time inferencing, which can be implemented using Edge Devices? This way, we should have a mother model to be trained once and use edge for real-time inferencing. Meanwhile, we can use continuous learning methods for retraining the main model.

1

No-Intern2507 t1_j8vtfqh wrote

dood iits not too smart to train on its wonky trashy results you know ? give it some thought actually instead of crapping sEnTiEnCe fantasy

1

Cogwheel OP t1_j8wn063 wrote

what crawled up your ass and died? if I was really taking that bit seriously do you think I would've written "through various leaps of logic and 'faith'"? Do you really think there's no value in the overall conversation or were you just triggered?

So many other people have actually answered the question that your response seems completely asinine

1

MysteryInc152 t1_j8wx6tx wrote

Not very necessary. An LLMs Brain might be static itself but the connections it makes between neurons are very much dynamic. That's why in context learning is possible. LLMs already mimic meta learning and fine-tuning when you few shot.

https://arxiv.org/abs/2212.10559#:~:text=Language%20Models%20Secretly%20Perform%20Gradient%20Descent%20as%20Meta%20Optimizers,-Damai%20Dai%2C%20Yutao&text=Abstract%3A%20Large%20pretrained%20language%20models,Context%20Learning%20(ICL)%20ability.

1

marcus_hk t1_j8ypfa6 wrote

Take a look at Hebbian and adaptive resonance models. No backprop, no distinct training/inference phases.

1