The median is the midpoint where 50% of people are above/below the threshold. If there are outliers, and perhaps we can agree there are outliers on the smart/dumb scale, it can skew the average above or below the median.
And I guess being extremely dumb carries a larger risk of death than being extremely smart, so if people start out symmetrically distributed along this scale, the distribution will skew right over time.
If we assume intelligence is a normal distribution, then we can say half of the population is below average. I think it's fair to categorize half as most.