There are multiple factors at work: There's ongoing maintenance as with everything, and you can probably accumulate its costs into it meaning "you're paying for a new one every 40 years through maintenance alone". Yield is going down over time (solar panels are sold with guarantees like "80% efficiency after 20 years"). And finally, technology is improving: replacing these 2006 solar panels in 2020 not only gives you back the base yield of a 2006 panel, but the base yield of a 2020 panel (see https://upload.wikimedia.org/wikipedia/commons/b/b8/CellPVef... for a research level comparison over time. That won't translate directly into industrial use though, but the direction should be evident)
So the situation is less that is "has to be rebuilt" but that it's economical to rebuild every 20-ish years (factoring in the maintenance costs: a panel that you replaced due to physical damange 10 years in needn't be scrapped just 10 years later) rather than letting the stuff hang around for arbitrarily long times.
With a proper recycling chain that shouldn't be too bad (there's little reason why the silicon can't be purified, and the doting material be refurbished), and ongoing maintenance is a concern with all types of power plant.
So the situation is less that is "has to be rebuilt" but that it's economical to rebuild every 20-ish years (factoring in the maintenance costs: a panel that you replaced due to physical damange 10 years in needn't be scrapped just 10 years later) rather than letting the stuff hang around for arbitrarily long times.
With a proper recycling chain that shouldn't be too bad (there's little reason why the silicon can't be purified, and the doting material be refurbished), and ongoing maintenance is a concern with all types of power plant.