You going to have to sedate him and pretty well. I moderately sedate my Grace before vet trips but this is because she's so shy and she just panics. She can't get a good exam otherwise and truthfully on her 2nd year check up (and one yr she was fine LOL) I really thought she'd give herself a heart attack or stroke
There was just no need for her fear. A little happy pill and she's fine.
There is a WIDE range in the acepromozine and Grace just gets enough to take the edge off - I'm betting you need more than that since you can't even handle your cat. Any type of pill you could slip into food to zonk him out and get him in a carrier would be great. They can handle him at the vet - the will ketamine him if they have to to examine him.
There is the less drastic approach of only feeding him in a carrier that stays in a room with him all the time. Get him used to the carrier being a pleasant place to sleep or eat. Then one day before the trip to the vet - put food in there then just close the door behind him. He won't panic because you will have done this before and of course immediately let him out several times
Should take about 2 weeks to acclimate him.
But I have to ask how long have you had this cat - and I kind of disagree with your term "semi feral" he still sounds pretty much like a feral cat living indoors to me if you can't even really touch him. It's not the best environment for either of you to live in. Obviously he doesn't trust you. What would happen if there was a danger in your home? You'd never get him out. It is a situation you really need to work on some how. The last thing I would do is use force - it will cause even less trust. Things have to be on his time.
BUT if he has run of your house and free feeds then the cat does not have any need for you. Feed him then stand next to him without touching him - Sleep in the same room with him (he can check you out while you are non threatening). Sit with him and just be or read out loud or whatever. He'll get used to your voice while you are not engaging him. Try to be on the floor as much as possible instead of standing over him.