Casual hobbyist, not an expert here.

It WAS working… About eight months ago, I trained a bunch of embeddings and hypernetworks and it all worked great.

Cut to the present, I want to do some more training. I’ve updated Automatic1111 several times, but nothing else about my setup has changed. However, whenever I try to train anything (embeddings, hypernetworks or loras), loss is NaN for 4 out of 5 steps right from the get go. As the training progresses, loss becomes NaN for 9 out of 10 steps, then 19 out of 20 steps around step 3,000, which is as far as I’ve gotten. Hypernetworks just don’t work at that point and embeddings produce garbage.

I have googled like crazy, and found

A few threads, where the best hint is that (at least 8-9 months ago) xformers broke training. Well, I’ve messed around with xformers, uninstalled and reinstalled xformers, eaten xformers for breakfast. Behavior is the same.

Lower training rate I have set my training rate to 0.0000000000000005. Behavior is identical.

My system is on the low end for VRAM (8G). I have TWO 8G cards, so I wish I could train on both like I can for Llama. But I also think that’s not it, because my OLD embeddings and hypernetworks came out great and still work.

Any thoughts here?

  • Hubi@feddit.deEnglish
    5·
    1 year ago

    I’ve only had this happen when I messed something up in my training settings. Keep in mind that a very slow learning rate can cause this as well. That said, maybe try training with kohya_ss, it’s much simpler and less resource-heavy than the training extension in Auto1111.

  • BrianTheeBiscuiteer@lemmy.worldEnglish
    4·
    1 year ago

    Depending on how the dependencies (i.e. xformers) are versioned you can make a new clone of A1111 and checkout a commit from ~8 months ago to see if it works again. Of course I would also recommend trying a fresh install.

    • The Bard in Green@lemmy.starlightkel.xyzOPEnglish
      1·
      1 year ago

      Of course I would also recommend trying a fresh install.

      Way ahead of you there. I’ve reinstalled the current version four or five times at this point.

      make a new clone of A1111 and checkout a commit from ~8 months ago

      This is a good idea. I’ve tried two different old versions from old commit hashes so far and both have crashed with other problems. It seems like (lol) both versions of A1111 put their venv in the same place, so the old versions are barfing on some dependencies with version numbers that are too high and they ALSO broke my current version by downgrading some other dependencies (easy fix, just wipe it out and reinstall it again). I’m trying to debug this, because I COULD see a world where I have an old version of A1111 training on one card while the NEW version generates on the other.

  • Scrubbles@poptalk.scrubbles.techEnglish
    2·
    1 year ago

    Check out the readme and the advanced parameters. Iirc there’s literally a checkbox for like half vae or something that says “check this if you’re getting NaN errors”