I don't feel like invisible units in their current state are much of a challenge because all you really need to counter them is a sentry post and if you didn't even want a sentry post all you would need to do is buy thorns and invisible units would kill themselves pretty quickly. You actually don't need a sentry post until invisible bosses come. With that being said I feel like they could be given some sort of buff to make enabling invisible mode more of a challenge.
Just something I took a minute to think of but maybe an easy way to make invisible units harder would be to make an upgrade you have to get called ' Eye of the enemy ' maybe or anything but until you get it invisible units would completely ignore armor or a % of armor and could possibly have multiple tiers ( If this would take a lot of time to code or implement no worries ) If anyone else has any suggestions on changes that could be made to invisible units please feel free