Even CQC users have flying slashes and shockwaves that they can use so...I think the downside of being an ONLY CQC specialist is facing status effect types like Leo Burns and Feltyrosa's explosive body coatings. Those two are really hard to face against the doable but being a CQC is just a hard to no go.
On another note, hand-to-hand CQC against long sword wielders always has the advantage as long as they keep the range so the swordsmen would find it hard to even launch their swings. You only need to block the pommel of the weapon and DPR is no longer a question.
Or you can even have a MA like the explosive boxing from the colosseum guy, that way you get statuses and other stuff too.

