I disagree, you'd be surprised how many people have the mental model of various large code bases (think Linux kernel) in their minds. It's not as if you are some savant memorizing all the lines of code and it helps to differentiate between "complexity" and "a mess." :)

Linus refused to merge the kernel debugger patches for a very long time because he felt that if a bug could not be solved by thinking (and logging), then the code was too complicated.

Did he state what changed his mind?

This might be an argument for not going overboard with complexity in the first place, which is probably a view Pike and Thompson would subscribe to.

> Often that point is quite low.

I've found that liberal logging and careful error management has almost replaced the debugger entirely. In the last three months, I've only fired up the debugger twice, and that was when I was working with untyped memory and peeking at memory through the debugger was the most efficient way to get things done.

I'm also one to use logging and error management to help my way through a program. I don't understand how anyone else does it any other way. But surely the debugger helps a lot when no other way to reason about your code exists.

I have a lot of problems explaining this to the people I work with. Most of the errors we make are in some way related to having imperfect information through the debugger.

At some point complexity gets so high the debugger isn't even capable of handling the system under inspection. What are the tools we use in these cases?

>>You are only so smart. Once the complexity of the programming model reaches a certain point a debugger is necessary to validate and discover the true nature of a system.

That is true but if you rely only on the tools, you will never gain a smarter understanding of the systems you are working with.

I just want to point out that Rob designed much of the logging infrastructure at Google. Take from that what you will.

For people who do not work at Google, is there something unusual (positive or negative) about Google's logging infrastructure?

Think about the scale at which it must operate. It is quite literally awesome. I wish I could give numbers. :-(

One should always try to write well modularized code with lots of good abstraction barriers so that you only ever have to have a fraction of the complexity of the system in your head at a time. If you had to understand everything down to the transistor level you could never even understand a Hello World program. That's because the language you use gives you good abstractions so you don't have to worry about linking, operating systems, memory allocation, etc. When writing complex systems you should strive to create such good abstractions yourself.

Not to mention a debugger to try to understand the "outside world", libraries, hardware, most of it with bugs that may affect you.

Yuck, a debugger exposes so little of the world that it's only appropriate in highly targeted approaches.