The big issue they’ve had, imo, is that there’s too many bots and too many IRC bridges from stubborn oldies who refused to migrate away from IRC.

There was a time when freenode went down that many channels were experiencing an exodus, which would have been the perfect time for Matrix adoption, but people were far too stubborn with IRC despite its many flaws (privacy, lack of history, poor feature set, etc.)