Toots from 2023-06-16
“To make sure that learning is sustained over a long time period, one needs to make sure that a…
“To make sure that learning is sustained over a long time period, one needs to make sure that access to the original data source is preserved and that additional data not generated by LLMs remain available over time. The need to distinguish data generated by LLMs from other data raises questions around the provenance of content that is crawled from the Internet: it is unclear how content generated by LLMs can be tracked at scale.”
https://arxiv.org/pdf/2305.17493v2.pdf
Mastodon Source 🐘
“But the point of this blog post isn’t to summarize this specific paper. Rather, it’s to ca…
“But the point of this blog post isn’t to summarize this specific paper. Rather, it’s to call attention to the fact that anomaly response as a problem that we will face over and over again. Too often, we dismiss the anomaly we just faced in an incident as a weird, one-off occurrence.”
https://surfingcomplexity.blog/2023/06/10/treating-uncertainty-as-a-first-class-concern/
Mastodon Source 🐘
Looking forward to this read… “We find that use of model-generated content in training cause…
Looking forward to this read…
“We find that use of model-generated content in training causes irreversible defects in the resulting models, where tails of the original content distribution disappear. We refer to this effect as Model Collapse and show that it can occur in Variational Autoencoders, Gaussian Mixture Models and LLMs.”
https://arxiv.org/abs/2305.17493