When Bad Data Leads to Good Models
arxiv.org
In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model's output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of “quality” from the perspective of pre- and post-training co-design. Specifically, we explore the possibility that pre-training on more toxic data can lead to better control in post-training, ultimately decreasing a model’s output toxicity. First, we use a toy experiment to study how data composition affects the geometry of features in the representation space. Next, through controlled experiments with Olmo-1B models trained on varying ratios of clean and toxic data, we find that the concept of toxicity enjoys a less entangled linear representation as the proportion of toxic data increases. Furthermore, we show that although toxic data increases the generational toxicity of the base model, it also makes the toxicity easier to remove. Evaluations on Toxigen and Real Toxicity Prompts demonstrate that models trained on toxic data achieve a better trade-off between reducing generational toxicity and preserving general capabilities when detoxifying techniques such as inference-time intervention (ITI) are applied. Our findings suggest that, with post-training taken into account, bad data may lead to good models.

  • masterspace@lemmy.caEnglish
    13·
    5 days ago

    Oh that’s neat and makes sense.

    iiuc, basically when neural networks have enough neurons / pathways to encode a small amount of information, you have each neuron being used to encode a single ‘feature’, but as the amount of things it needs to encode for grows, it has to use individual neurons to encode multiple different things which entangles those concepts together.

    So if you don’t have toxic training data, then the general pattern of toxicity isn’t very strong in the training data, so the concept of toxicity gets entangled with lots of other stuff, then when you tell the model to not be toxic it avoids a bunch of useful things.

    If you instead feed it enough toxic data during training (but not too much), then the pattern of toxicity is more strongly isolated in the neuron encoding and less entangled with everything else, so when you tell it to not be toxic it doesn’t impact everything else as much.

    • gtrcoiEnglish
      1·
      4 days ago

      I’m confused by all of this, but why is there a “too much”, like if I threw all of the toxic sites into the training data and told the ai “this stuff here is toxic” doesn’t ai just figure out the pattern for toxicity and target it precisely?

  • FizzyOrangeEnglish
    4·
    4 days ago

    I guess if you don’t train it on toxicity it has no idea what you’re talking about when you ask it to remove it. Makes sense.

  • CodexArcanum@lemmy.dbzer0.comEnglish
    2·
    5 days ago

    Gotta love how they do such a solid job defining “toxicity” and don’t just hand wave it away as either “any data from 4chan is toxic” or “whatever the benchmark for toxicity says.”

    What exactly are these “toxic” behaviors we’re training to censor? KYS statements? Bad advice? Telling people that glue goes on pizza? Coming out as gay? Criticising $GOVERNMENT?