Your memories would completely break--to the point where you won't be able to get them back, no matter how much nova we collect. There wouldn't be anything left of who you are, apparently.

It sounds like... I guess people are getting "infected" with this, so their own nova starts to splinter. And the only way to stop that from happening is to remove them from the Realm before it splinters entirely and then infects the next person.