I went ahead and asked ChatGPT your question for you:

No, Google is not a language transformer like ChatGPT. While Google has developed language models like BERT and GPT-3, it is primarily a search engine that uses various algorithms to deliver search results to users. Google's language models are used to improve search results and to power services like Google Assistant, but they are not the primary focus of the company.

As for ChatGPT, it is true that its main purpose is to generate human-like text based on prompts provided by users. However, it is not just a search engine that can talk back. It is a complex machine learning model that has been trained on vast amounts of text data and uses advanced natural language processing techniques to generate responses.

While it is true that the development of more algorithms and computing power is necessary for further advancements in AI, ChatGPT and other similar models have already made significant strides in the field of natural language processing. They have the potential to be used in a wide range of applications, including language translation, content creation, and customer service. However, it is unlikely that they will completely replace human jobs, as they are still limited by their inability to understand and reason about the world in the way that humans do.

First author is a high-school student. Impressive.

That's not my takeway. GLM-130B is even behind OPT according to the mean win rate, and the instruction tuned version of OPT in turn is worse than FLAN-T5 which is a 10x smaller model (https://arxiv.org/pdf/2212.12017.pdf Table 14)

English due to data availability.

Are any benchmark scores such as MMLU or BigBench available for Aleph Alpha's models?

In terms of Consumer Apps, the Poe app from Quora has access to two models from Open AI and one from Anthropic.

Perplexity.ai, YouChat and Neeva are search engines that integrated LLMs.

Google has an AI + Search Event on Wednesday where they are likely to announce something as well.

In terms of APIs and getting a feeling for these models, I would use OpenAI's APIs. Their models are the best publically available models. Open Source models are still far behind.

No, what u/mulokisch is saying is that the cookie disclaimers are unrelated to GDPR.

Also, you might want to tell them that you thought their paper was really well executed and if they would be willing to chat and if there are internship opportunities on their team (if you are interested in that).

It's a preprint server, meant to publish preprints.

We can't afford that in Machine Learning by the time your paper has passed the review process it's going to be outdated.

Data scientist is not a very well-defined job title. It can involve more or less social interaction depending on the company.

But you could get more social interaction by doing pair programming with your coworkers or asking to be assigned more work involving social interaction, e.g. talking to clients or mentoring new coworkers.

You could eventually transition into another career path such as people management, education, project management, program management, sales, etc.

I think the main limitations of LLMs are:

1. Hallucinations: They will make up facts.
2. Alignment/Safety: They will sometimes give undesirable outputs.
3. "Honesty": They cannot make reliable statements about their own knowledge and capabilities.
4. Reliability: They can perform a lot of tasks, but often not reliably.
5. Long-context (& lack of memory): They cannot (trivially) be used if the input size exceeds the context length.
6. Generalization: They often require task-specific finetuning or prompting.
7. Single modality: They cannot easily perform tasks on audio, image, video.
8. Input/Output paradigm: It is unclear on how to use them for tasks which don't have a specific inputs and outputs (e.g. tasks which require taking many steps).
9. Agency: LLMs don't act as agents which have their own goals.
10. Cost: Both training and inference incur significant cost.

Yes, they are trained on a much larger amount of language data than a human sees in their lifetime.

However, I would argue that it's a worthwhile trade-off. Computers can more easily ingest a large amount of data. Humans get feedback from the environment (like their parents), can cross-reference different modalities, and have inductive biases.

I tried using it but the user interface is so much worse than Twitter. Any tips?

Did you do an evaluation of the quality impact?

Colab

I actually think prompt engineering is becoming less and less important with things such as ChatGPT.

That now they can collect data to make it even better.

Did you hear about any parties on Thursday. Apparently Google, OpenAI and DeepMind are all tomorrow.

I'm down.

This reads like an announcement for the release of a traditional piece of software. It would be nice if you could instead publish some metrics such as FID or ideally side-by-side human evaluation against SD 1.5 / DALLE-2.

One of the best things about the machine learning community is that we have been taking a rational metrics-driving approach. I hope that as ML gets more and more real-world use cases, and both open-source and commercial applications that are not tied to academic research become more prevalent, we don't lose that.

I'm not sure if that question was directed at you specifically.

>but the transformer is stuck and loss doesn't decrease after abt 20k steps.

Presumably they meant training loss, which would indicate that this is an optimization problem.

I disagree with some of the other advice here. I would suggest starting with something that you know works. That means you could either use a training setup from another modality such as vision or text and apply it to your data, or you could try to reproduce a result from the literature first.

Rereading my own comment, it could have been phrased better. Let me try again:

I think you are taking OP's question too literally. At least as I understand it, the intent of OP's question was: "Why are self-supervised autoregressive models the predominant form of generative models for language? Intuitively it would seem that the training process should be closer to how humans learn language."