this.
Say I invest 100 stat points into 2 styles of sword MA, you break my sword....because it isn't made out of diamond, then all of a sudden, even though i'm lvl 90 and you are lvl 60, I can't even strangle you because I didn't bother to spend another 50 stat points to learn a hand to hand combat style so my blocks and punches are all D rank--and there is nothing I can do about it because you destroyed my sword and you don't have a sword so I can't even steal yours.
What other way is there in the game where you can do this to someone? The answer--there really isn't! The only things that come close don't even come close. Using seastone requires constant contact and doesn't effect static boosts given by simply consuming a DF. Yami Yami can erase those passives but also has to be in contact and this is literally the signature defining feature of the fruit--if other things could do this to DF users then the yami wouldn't be significant in any way.
So in order to balance the idea that an unnamed sword that a master swordsman is using can be destroyed, you'd have to make a mechanic that could effectively completely erase the combat stat investment of an opponent using any of the other kinds of combat styles--right? Does this sound like a head-ache? Is this making no sense? Cool, lets just keep it simple and keep doing what we have been doing

[automerge]1586375353[/automerge]
then break their arm so they can't swing their sword.