When listed elements(including trees) are very large, there's a reasonable compromise pattern throughout UI in pagination. We've already found that infinite scroll isn't that desirable for productive UX because it eliminates any kind of landmarks, and tree views are the same way: so, my current belief on what to do is to default to expanding breadth-first to some limit, and then bias node expansion in places where the user has clicked, including some history so that the UI can present multiple leaf nodes simultaneously. You can imagine this as having cursors for each explored branch and viewing the area around the cursor, with "18,000 more items..." at the edges of the view.

Re: the layout problem, it seems to be a factor in a lot of real-world apps(buildings, hardware design, etc.) and to the extent that it's "solved" by retained mode, it's a solution based on moving around the processing order so that you deal with a different set of constraint edge cases manually. As such I think it really is just a unsolved class of issues, which wasn't tackled before because it involved a degree of algorithmic complexity and resource use that wasn't on the table in the 1980's when the GUI was first widely adopted.

So I think it's fine to explore having a whole algorithmic path and data structures dedicated to building the bounding targets for both collision and rendering - I have done similar when I've poked around at doing my own framework. That reflects the actual complexity of the solution, and it becomes especially apparent how deep you could go once you allow your targets to be arbitrary shapes with 2D transforms.