Non-remappable controls is pretty retarded in this day and age, but I dont think about it much since most of the games I've played suffered from it (and most arent of the twitch type).
I mainly play RPGs and strategy games, and what bugs me the most is unnecessary micromanagement due to a poor interface. Non-sortable journals and inventories are some aspects of it (just a few days with some intro UI course material would help many game developers a lot). Basically anything that force you to use a lot of clicks to perform a mundane task that doesnt advance the plot or game qualifies. I have three "favourites".
Inventories that are arftifically limited, for instance by number of slots rather than weight like 20 rings taking up as much space as 20 sets of armour in unmodded Baldurs gate, would definitely be one of the pet peeves. It wouldnt be so annoying in itself, but it is often coupled with a need to sell a lot of the crap, forcing you to trek back and forth to various traders, often through totally emptied areas.
Which brings up what I might dislike the most. Slow walking through "cleared" or "safe" areas, something that makes a game longer but doesnt add to the length of the actual playing experience. Prime offenders would be Arcanum or Baldurs Gate (when I play the latter with my gf one of goes to make tea or sth while the other does the walking and selling of loot, since it takes quite long, and it adds up during the course of a game). Give me fast travel for gods sake! It would be different if the trip actually was presenting some sort of gameplay experience, like running through a camp of orcs or something, but just marching is so dull.
Gothic III actually solved this pretty well, and it is one reason I dont mind the fast travel system of the otherwise dislikable Oblivion much either.
The third thing that hasnt been mentioned, but I find utterly retarded, is how three-dimensional interfaces in some strategy games clutter things and making it hard to actually select the right item or click on the right spot. Trying to make an agent or worse, an army, reach the intended destination in a cluttered spot can be extremely frustrating, which has resulted in me quick-saving before issuing movement orders on such occasions. The first medieval had similar problems in 2D, with agent icons cluttering small provinces on the map making it incredibly frustrating to select the one you wanted on Malta or some other tiny speck...
I think I pick the moving of merchants and ships to distant locations in that game as the most retarded design issue in the games I play today. The stupid things often have to take multi-turn walks to locations. Walks that can be disrupted if they run into something (which they often do as they tend to walk on roads). This force you to give them new directions. But they are disrupted by "obstacles" even if they can walk around them (as they ignore zones of control)!!! And a second merchant walking the same path in the same round, when the obstacle should be known, will be interrupted in the same way... All agents are like this, but I single out merchants since I feel that they should have been abstracted out of a game where the primary issue is battles!!! As for the ships the reason for my annoyance with that is that the only way to avoid them getting "stuck" on other ships is to move them in steps corresponding to their line of sight. And a stupid ship can move 5-6 times that distance in one turn, so you have to click 5 times rather than 1 to get somewhere, for no other reason than that the programmers didnt bother to code basic evasion, on a static grid... How freaking difficult can it be to find a free path for ONE turn on a grid, in turn-based mode where nothing else is moving???