Here is a bit more realistic model of the riffle shuffle. These results show that you need between 3 to 8 riffle shuffles, depending on your skill. Notably, the number of needed shuffles actually does get lower as you get better, i.e., less clumping is better: http://blog.maxshinnpotential.com/2017/11/05/optimality-in-c...
Do you think this also applies to kernel density estimation? I've used the R program and noticed extra density peaks that I wasn't able to explain except as artifacts. I guess it could've been picking up a mixture of distributions... but the densities appeared periodic if that makes any sense?
This is something a bit different, the effect in the paper comes from the eigenvectors of the data matrix. I'm no expert on kernel density estimation, but since it is basically just convolution, I would guess the effect you are describing comes from multiplication in Fourier space.
Thanks! Makes sense.. your paper is very good. It stresses the importance of understanding the mathematical tools we use and if we don’t how it can lead us astray.
ATM I can only read the lead .. and then it's paywall and not yet on sci-hub :-)
Would I be right in guessing that things are only stable until the eigenvalues fall to the floor and any further dimensional reductions will easily perturb?
Sorry, I didn't have funding to pay the open access fees. The published version is very similar to the preprint: https://www.biorxiv.org/content/10.1101/2023.06.20.545619v1 Alternatively, if you want the full version, PNAS gave me some eprints to share so just email me (link on my site in my profile) and I will send you a copy!
Not sure what you mean by "things are stable" in this context, could you elaborate?
I'm really good at typing in the traditional qwerty and йцукен layouts. The problem is that I need two and sometimes the wrong one is active. This is why I envy people who can do everything without needing non-Latin alphabets.
I like the idea of extending Latin with some extra letters as needed to accomodate more languages. Someone would have to make a couple of new fonts probably.
Maybe a nitpick, but Aaron Swartz was probably quoting Richard Hamming:
And I started asking, "What are the important problems of your field?" And after a week or so, "What important problems are you working on?" And after some more time I came in one day and said, "If what you are doing is not important, and if you don't think it is going to lead to something important, why are you at Bell Labs working on it?" ... If you do not work on an important problem, it's unlikely you'll do important work.
Wow you're probably right, I wasn't aware of that paper/talk. I wish discovered this earlier in my career, but I suppose it's not too late for me to work on important problems.
One classic result from biophysics is that, if all of your decisions have a fixed level of difficulty, then what the author suggests is (mathematically) proven to be suboptimal: if a decision feels more difficult, you actually should spend more time on that decision. (Keywords to search for: drift diffusion model[2], sequential probability ratio test[3])
The technical term for what the author is suggesting one should do (not spending so much time on decisions with equal outcomes) is an "urgency signal", or just "urgency" for short. If you are using an urgency signal, then you will spend less time on difficult decisions (i.e. ones with near equal outcomes) than you would without an urgency signal, but still more than you would easy decisions. (For easy decisions, you spend approximately the same amount of time regardless of whether you have an urgency signal.) In the extreme case, for an infinitely strong urgency signal, you will spend equal time on both easy and difficult decisions. (See Paul Cisek's work, e.g., [1].) Conceptually speaking, you need time to detect that the current decision has near-equal outcomes.
It was only recently (mathematically) proven that if you have different levels of difficulty in your decisions, it is optimal to use an urgency signal [4,5]. So since most sequences of decisions aren't all equally difficult (as in the study referenced here), in practice, people will use an urgency signal in decision-making.
Of course, both the desired accuracy and the urgency signal depend on how the decision is being evaluated: if your goal is to make an accurate decision (e.g. buying a house), then you will have a weaker urgency signal and require more evidence before you make a decision. By contrast, if you prioritise decision speed, you will show more urgency signal and require less evidence to make a choice. (The technical term is "speed-accuracy tradeoff".)
Current work suggests that when you are making some decision for the first time, you will not use an urgency signal, but as you become an expert at making those decisions, you will gradually develop an urgency signal [6]. This makes sense conceptually: if you know approximately how hard you can expect these decisions to be, you will recognise the situation where they have approximately the same utility and adjust your strategy accordingly.
