I think you may be forgetting point in your history where you needed deliberate practice to get to where you are now. If you only ever choose between not playing and playing for fun, you will never get better. The transition to playing only for fun is the point where improving is now left to chance—not that you completely stop growing, but you become more entrenched in your existing practices and your skills crystallize at their current level.
This feels alien to me. As I play for fun of course I try new things, and keep expanding what I can do. That's how I've always learned music. I think I've even done my fastest growing while playing live for dances, which is probably the most "fun" and least "deliberate practice" there is!
You're speaking in very strong terms, as if we really understood how people become good musicans, but I think we really don't understand it and people take many different paths.
(My background: I started on guitar at about ten, switched to mandolin at about twenty, have always dabbled in other instruments, and at this point my main one is probably piano. I play in two contra dance bands, https://kingfisherband.com and https://freeraisins.com , and mostly play for dancing.)
I've heard this from a couple of concert pianists in my family, and I've dabbled a bit myself too. Basically lost my 3-4 years of playing after a while. I can still play scales and chords, more or less, but I've tried learning again and struggle even with basic pieces.
The saying goes “practice makes permanent”.