Ive almost never worked on a project where there was the right number of QAs who were doing the right thing.
Usually there either arent any in which case bugs get missed or there are 5 very cheap ones running mindless scripts who are standing in for the devs' inability or unwillingness to write decent automated tests but dont catch the really deep level thorny stuff.
QA has a reputation problem. Because it is considered as unimportant role, good people don't get attracted to it. The average people who do don't do a good job.
> QA has a reputation problem. Because it is considered as unimportant role, good people don't get attracted to it.
Hard disagree. QAs' main tasks are acceptance testing and V&V. This means their work is to act as independent parties whose role is to verify that the results are indeed being delivered. Their importance is up there next to Product Owners.
The problem is that some QA roles are relegated to running the same old test scripts and do some bullshit manual exploratory tests that catch and assert nothing. It's the ultimate bullshit job as it's performative. This is clear in the way that LLM agents are seen as threatening the very existence of the role.
Really? The best people I worked with were never QA.
Moreover, the best QAs would almost always try to be not QA - to shift into a better respected and better paid field.
I wish it werent so (hence my username) but there is a definite class divide between devs and QA and it shows up not just in terms of the pay packets but also who gets the boot in down times and who gets listened to. This definitely affects the quality of people.
I think it's overdue an overhaul much like the sysadmin->devops transition.
We have differing experiences, which shouldn't be surprising. My example explicitly referred to someone who was a good engineer who enjoyed the QA role.
This might have been an Apple/MS thing, but we always had very technical QA people on the dev tools team. For example, the QA lead for the C++ compiler had written their own compiler from scratch and was an amazing contributor.
In the Windows team (back before the test org was decimated) I saw the described "class divide". Anybody who was good enough would switch from SDET to SDE [disclaimer: obviously there were some isolated exceptions]. The test team produced reams of crappy test frameworks, each of which seemed like a "proving project" for its creators to show they could be competent SDEs. After the Great Decimation my dev team took ownership of many such frameworks and it was a total boondoggle; we wasted years trying (and mostly failing) to sort through the crappy test code.
This was all unfortunate, and I agree in principle with having a separate test org, but in Windows the culture unfortunately seemed to be built around testers as second-class software developers.
I spent most of my time working on Visual Studio (in the Boston time frame) so we got to interact with pretty much every team. I absolutely hated interacting with the Windows team. Everything was a fight for no reason.
As I said above, everyone has their own experiences but the QA folks I worked with at MS were fantastic.
Not sure if you're aware but Dave Plumber now has a really good YT channel [0] where he talks about MS back in those days. It's a fun walk down memory lane.
> Really? The best people I worked with were never QA.
> Moreover, the best QAs would almost always try to be not QA - to shift into a better respected and better paid field.
That sort of seems circular. If they're not respected or paid well, of course most of the talented people would not want to remain in QA, and eventually you'd just have mediocre QA. That doesn't really give you any insight into whether high quality QA would be useful though.
(edit: I see now that's basically the point you're trying to make, so I guess we're in agreement)
This was after seeing those ridiculous PRs where microsoft engineers patiently deconstructed AI slop PRs they were forced to deal with on the open source repos they maintained.
When he was gone a few months later and github was folded into microsoft's org chart the writing was firmly on the wall.
He was never truly independent though. The org structure was such that the GitHub CEO reported up through a Microsoft VP and Satya. He was never really a CEO after the acquisition, it was in name only.
Also of note is that the Microsoft org chart always showed GitHub in that structure while the org chart available to GitHub stopped at their CEO. Its not that they were finally rolled into Microsoft's org chart so much as they lifted the veil and stopped pretending.
I never said he was "truly independent" nor meant to imply it.
Nonetheless it looks like he was both willing and able to push back on a good deal of the AI stupidity raining down from above and then he was removed and then, well, this...
You said he was independent, I didn't include "truly" intending to make a distinction there. How could one be an independent CEO while reporting to a VP who reports to another CEO?
I don't personally know him and wouldn't begin to assume on what, or how, he pushed back. Though Microsoft had AI in the GitHub org well before the leadership change - the AI leader now in charge of GitHub was previously in charge of an AI org that was moved over in the org chart to dotted line report as embedded employees, or whatever they would have been called.
Humans have the ability to retrospect, push back on a faulty spec, push back on an unclarified spec, do experiments, make judgement calls and build tools and processes to account for their own foibles.
Humans also have the ability to introspect. Ultimately, (nearly) every software project is intended to provide a service to humans, and most humans are similar in most ways: "what would I want it to do?" is a surprisingly-reliable heuristic for dealing with ambiguity, especially if you know where you should and shouldn't expect it to be valid.
The best LLMs can manage is "what's statistically-plausible behaviour for descriptions of humans in the corpus", which is not the same thing at all. Sometimes, I imagine, that might be more useful; but for programming (where, assuming you're not reinventing wheels or scrimping on your research, you're often encountering situations that nobody has encountered before), an alien mind's extrapolation of statistically-plausible human behaviour observations is not useful. (I'm using "alien mind" metaphorically, since LLMs do not appear particularly mind-like to me.)
Most companies I've worked for have had 'know the customer' events so that developers learn what the customers really do and in turn even if we are not in their domain we have a good idea what they care about.
Not get stuck on an incorrect train of thought, not ignore core instructions in favour of training data like breaking naming conventions across sessions or long contexts, not confidently state "I completely understand the problem and this will definitely work this time" for the 5th time without actually checking. I could go on.
LLMs by their nature are not goal orientated (this is fundamental difference of reinforcement learning vs neural networks for example). So a human will have the, let's say, the ultimate goal of creating value with a web application they create ("save me time!"). The LLM has no concept of that. It's trying to complete a spec as best it can with no knowledge of the goal. Even if you tell it the goal it has no concept of the process to achieve or confirm the goal was attained - you have to tell it that.
The economics of software development have lowered the bar for software engineers: there simply aren't enough people who are good at it (or even want to be), and the salaries are very high, so plenty of people who shouldn't be SWE's are.
I am a software engineer, and I would absolutely love to see more professional accountability in this field. Unfortunately, it would make the cost of software go up significantly (because many many people writing software would be ejected from the industry)
I've found recent versions of Claude and codex to be reluctant in this regard. They will recognise the problem they created a few minutes ago but often behave as if someone else did it. In many ways that's true though, I suppose.
Does it do this for really cut and dry problems? I’ve noticed that ChatGPT will put a lot of effort into (retroactively) “discovering” a basically-valid alternative interpretation of something it said previously, if you object on good grounds. Like it’s trying to evade admitting that it made a mistake, but also find some say to satisfy your objection. Fair enough, if slightly annoying.
But I have also caught it on straightforward matters of fact and it’ll apologize. Sometimes in an over the top fashion…
LLM based solutions don’t need to stay dry and warm at night, with a full belly, possibly with their sexual partner with whom they have a drive to procreate.
I've been trying to push people to use hitchstory or similar to generate docs from specification tests precisely to avoid that redundancy but most people just look blankly at it and go "why don't you just do that with AI?"
Usually there either arent any in which case bugs get missed or there are 5 very cheap ones running mindless scripts who are standing in for the devs' inability or unwillingness to write decent automated tests but dont catch the really deep level thorny stuff.
reply