But such groups are almost invariably coordinated. In a legislature based on sortition, there will be a percentage of busybodies/ assholes/ opportunists but they'll have a coordination problem, opponents, and term limits acting to restrain them.
Term limits incentivize a deep state exactly one layer removed from those to which the limits apply, as a repository of institutional knowledge about how things actually get done.
This seems rational. We on't have term limits int he US Congress and it doesn't seem any the better for it.
Japan, a heavily bureaucratized country, systematically moves junior and mid-tier staff around in some departments to minimize the possibility of nest-feathering and empire-building, although I would not say it's perfect by a long way.
We do have term limits for positions like the presidency, and what we see is a perpetual power structure one layer removed, in the party system, which effectively chooses who we're permitted to vote for.
Introducing term limits only forces the wealth and power to change it's face periodically. It is addressing a symptom, not the cause.