I don’t even think humans can “move beyond” their sensory data. They generalize using it, which is amazing, but they are still limited by it.* So why is this a reasonable standard for non-biological intelligence?
We have compelling evidence that both can learn in unsupervised settings. (I grant one has to wrap a transformer model with a training harness, but how can anyone sincerely consider this as a disqualifier while admitting that an infant cannot raise itself from birth!)
I’m happy to discuss nuance like different architectures (carbon versus silicon, neurons versus ANNs, etc), but the human tendency to move the goalposts is not something to be proud of. We really need to stop doing this.
* Jeff Hawkins describes the brain as relentlessly searching for invariants from its sensory data. It finds patterns in them and generalizes.
Human sensory data doesn't correspond -- not neatly, and probably not at all -- to LLM training data.
Human sensory data combines to give you a spatiotemporal sense, which is the overarching sense of being a bounded entity in time and space. From one's perceptions, one can then generalize and make predictions, etc. The stronger one's capacity for cognition, the more accurate and broader these generalizations and predictions become. Every invention, including or perhaps especially the invention of mathematics, is rooted in this.
LLMs have no apparent spatiotemporal sense, are not physically bounded, and don't know how to model the physical world. They're trained on static communications -- though, of course, they can model those, they can predict things like word sequences, and they can produce output that mirrors previously communicated ideas. There's something huge about the fact, staring us right in the face, that they're clearly not capable of producing anything genuinely new of any significance.
SoTA models are at least very close to AGI when it comes to textual and still image inputs for most domains. In many domains, SoTA AI is superhuman both in time and speed. (Not wrt energy efficiency.*)
AI SoTA for video is not at AGI level, clearly.
Many people distinguish intelligence from memory. With this in mind, I think one can argue we’ve reached AGI in terms of “intelligence”; we just haven’t paired it up with enough memory yet.
* Humans have a really compelling advantage in terms of efficiency; brains need something like 20W. But AGI as a threshold has nothing directly to do with power efficiency, does it?
LLMS are terrible at writing in terms of style, and in terms of content or creativity they couldn’t come up with a short story any better than what you’d find at an amateur writer workshop. To declare we have reached AGI in textual media seems premature at best.
> There's precious little training material left that isn't generated by LLMs themselves.
Percentage-wise this is quite exaggerated.
> Consider this to be model collapse (i.e. we might be at the best SOTA possible with the approach we use today - any further training is going to degrade it).
You consider this above factor to lead to model collapse? You’ve only mentioned one factor here; this isn’t enough. I’m aware of the GIGO factor, yes. Still there are at least ~5 other key factors needed to make a halfway decent scaling prediction.
It is worth mentioning one outside view here: any one human technology tends to advance as long as there are incentives and/or enthusiasts that push it. I don’t usually bet against motivated humans eventually getting somewhere, provided they aren’t trying to exceed the actual laws of physics. There are bets I find interesting: future scenarios, rates of change, technological interactions, and new discoveries.
Here are two predictions I have high uncertainty about. First, the transformer as an architectural construct will NOT be tossed out within the next five years because something better at the same level is found. Second, SoTA AI performance advances probably due to better fine-tuning training methods, hybrid architectures, and agent workflows.
> There's precious little training material left that isn't generated by LLMs themselves.
> Percentage-wise this is quite exaggerated.
How exaggerated?
a) The percentage is not static, but continuously increasing.
b) Even if it were static, you only need a few generations for even a small percentage to matter.
> You consider this above factor to lead to model collapse? You’ve only mentioned one factor here; this isn’t enough. I’m aware of the GIGO factor, yes. Still there are at least ~5 other key factors needed to make a halfway decent scaling prediction.
What are those other factors, and why isn't GIGO sufficient for model collapse?
Calling such a concept a 'layer' is quite confusing, given my experience with Photoshop, OmniGraffle, Pixelmator, and most other drawing tools I've seen. Why not just call it an "element" or "shape" or "item"?
When designers started switching from Photoshop to Sketch, "Layer" kinda changed meaning from "a clear sheet with drawings on it" to "any object on the canvas".
In Photoshop a layer could have many shapes on it, if you drew one shape/mark on top of another within one layer, they would be inseparable. If you drew any two things in Sketch, they would always be separate, I suspect this is why they stuck with the term "Layer" even though something like "Object" may have been more accurate.
I think layer is quite correct. Both Photoshop and sketch approach it the right way. Think of a sheet of paper as a layer. In Photoshop layers are sheets of paper that each matches the size of the document. On each paper you can draw whatever you want. They then lay on top of each other and the white parts are transparent.
In sketch, instead of sheets of paper that match the size of the document, you have a perfectly cutout shape for each stroke you draw. You can then move that cutout around within the document.
Both of those have the same principle except one is cutout one is not.
I agree that layer is correct, though I understand where there might be some “intuition clash” with people who are used to the classic paradigm.
When each layer is a transparent sheet it has an absolute Z order that translates 1-1 with the physical thing it’s virtualizing (transparencies, probably from traditional animation).
However if you cut out tiny shapes and arrange them with some on top of others to make a larger composition, there is no longer an absolute Z order to any element because there are a bunch of smaller stacks of various heights.
The virtualization no longer translates to the physical world, and from my experience that disconnect can sour people against a new way of doing things, regardless of whether it’s better.
First, you are not alone. Seek out support of all kinds. Don't blame "yourself" ... remember that the very idea of "self" is always changing and only a small part of it is under short term conscious control. Mental conditions can be very hard in certain environments, but more manageable in others.
Never forget that such conditions also bring advantages! Not the least of which is empathy.
I also have ADHD and anxiety, plus a history of not effectively managing my disappointment when things at work seem batshit crazy. It has taken a long time to recognize that a significant level of organizational dysfunction is very common.
To answer your question: I don't have any ironclad answers. You can gather ideas like you are doing and try experimenting.
Try organizing (grouping) your resume in different ways -- by topic or skill.
You can put the year of the job instead of the range in months. Perhaps even leave out the date.
I don’t even think humans can “move beyond” their sensory data. They generalize using it, which is amazing, but they are still limited by it.* So why is this a reasonable standard for non-biological intelligence?
We have compelling evidence that both can learn in unsupervised settings. (I grant one has to wrap a transformer model with a training harness, but how can anyone sincerely consider this as a disqualifier while admitting that an infant cannot raise itself from birth!)
I’m happy to discuss nuance like different architectures (carbon versus silicon, neurons versus ANNs, etc), but the human tendency to move the goalposts is not something to be proud of. We really need to stop doing this.
* Jeff Hawkins describes the brain as relentlessly searching for invariants from its sensory data. It finds patterns in them and generalizes.