I don't think any have made me quit… though a few have come close.
The big boss at the top of the mountain near the start of Lost Odyssey. I'm still convinced that thing is bugged and the devs just wouldn't/couldn't fix it.
Fiery Gates in The Last Remnant. The damned thing cheats getting a free attack at the start and end of every round plus, if any squad is lying dead at the end of a round, it uses that free attack to raise them from the dead to attack my remaining squads! This made for a lot of battles where I nearly won then suddenly got crushed. (Plus I didn't understand that I had to pause a video first in order to skip it so I got stuck watching the nearly-5-minute story video in front of it *several* times.)
The final bosses in Risen and Batman: Arkham Asylum, for similar reasons. The boss battles had completely different fighting systems than the rest of the game. (If you can even call that a fight at the end of Risen.)
They were all bosses, though. I can't remember any typical enemies I hated. Heck, I even liked the sappers in City of Heroes… oh wait!
The Knives of Artemis! Or, more specifically, their thrice damned caltrops. When they throw them, they slow your hero down 50%, all of them have them, and the debuff friggin' stacks. You go up against half a dozen and suddenly you're moving at 5% speed. Even when you beat them it takes forever to walk away!