Viewing a single comment thread. View all comments

midasp t1_izybqp8 wrote

Reply to comment by gwern in [D] G. Hinton proposes FF – an alternative to Backprop by mrx-ai

You are right. Intuitively, it's just rewarding correct inputs and penalizing wrong inputs. Which is largely similar to how many RL policies learn. FF seem like it will be able to discriminate, but it won't be able to encode and embed features the way back prop does. It would not identify common features. If you try to train a typical back prop based u-net architecture network, my instincts say it likely would not work since the discriminating information is not distributed across the entire network.

7

Akrenion t1_j011yxv wrote

U-net is specifically designed for backprop. It the skip connections are helpful for bp. We might need to rethink architectures for other approaches as well.

3

ChuckSeven t1_j016l2h wrote

That's actually a fair point. The optimisation lottery if you will. Where architectures are biased because they are designed around the algorithms that can be scaled and have shown to "work".

2