Wanted to share a project I've been working on aimed at helping solve the context problem called Memoryport. It works across LLM providers/apps, stores conversations locally, is fully OSS, and enables anyone to keep track of their memories over time. I tested mine with 500M tokens of context space (note: not the same as a context window) and only added ~300ms of latency to the inference session. I also built out an open spec called AMP that standardizes the communication protocol for how memory systems should interact with LLMs (see repo).
More than happy to answer any questions and hope you all find this as useful as I have.
Thanks for sharing this! If you'd be interested in running the benchmark yourself with Hypura I'd happily merge into our stats. Otherwise will add to my todo list :)
Even if there was a ton of writing, I'm not sure where NVMe even comes in the picture, write durability is about the flash cells on SSDs, nothing to do with the interface, someone correct me if I'm wrong.
I'm referencing it as being possible, however I didn't share benchmarks because candidly the performance would be so slow it would only be useful for very specific tasks over long time horizons. The more practical use cases are less flashy but capable of achieving multiple tokens/sec (ie smaller MoE models where not all experts need to be loaded in memory simultaneously)
Yes, definitely agree. It's more of a POC than a functional use case. However, for many smaller MoE models this method can actually be useful and capable of achieving multiple tokens/sec.
We aren't a package manager. We're a registry and CDN (of sorts). Blockchain is actually a huge solution to this problem for three very notable reasons. The first is that Deno module imports are url based, and we don't want code going off the internet, as this would break the code dependent on it. Blockchain solves this because transactions (module code) are unable to be modified or deleted. This means that import links will never break, thanks to blockchain! In addition, it's unbelievably cheap to permanently store data. For reference, we've stored 17,297 files on the blockchain. For proof, you can see our wallet address and transaction history here: https://viewblock.io/arweave/address/tySYSW93nDky1sbCO56PmyE... This permanent and decentralized data storage has cost us right around 5 cents USD. Thirdly, thanks to the blockchain, the module data is completely decentralized across over 340 nodes and counting around the world. You can see the exact statistic here: https://viewblock.io/arweave Again, thanks for bringing these things up. These are great points for us to address publicly.
You might take a look at how Go implemented their module system and solves the above without blockchain or a lockfile. Simple algo, excellent engineering, built in security and integrity checking.
In the end, you will want to mirror all external dependencies (repos, git, pkg) into your own cloud.
So Demo has us specify versions at the point of import, in how many files? Is this lunacy?
I store more files on GitHub for free, how can blockchain be cheaper than that?
Actually, this raises a very good point. I'm Tate, a co-founder. Our publishing system works in a way that users will be able to publish malicious modules, yes, but our registry is not decentralized up to a certain point; let me elaborate on this. If a user finds that a module is malicious and wants to report it, we can remove it from the registry completely because the registry is centralized. Though this data will still be accessible from the blockchain and the import url will be functional, we're building a system to warn the user whenever the url is imported from a Deno-specific response header. Now, after a certain amount of time has passed and a module isn't reported as malicious, we're building a system to automatically publish the entire registry to the blockchain as well, so that the registry AND the module are immutable. This is called Fossil, our "archiver." You can see its code here: https://github.com/nestdotland/fossil Again, thanks for bringing this up. I hope this explanation helped. Our goal certainly is not to promote or enable malicious code!
Indeed, this is a massive issue with any url based imports. Because Trex supports nest.land, this is not an issue. nest.land is actually a first-of-its-kind blockchain module registry and CDN. Because we use the blockchain for storing modules, they can never be deleted or altered in any way. This also means that they are permanently and indefinitely resolvable from the web. Because of this, module vendoring is no longer an issue!
Wanted to share a project I've been working on aimed at helping solve the context problem called Memoryport. It works across LLM providers/apps, stores conversations locally, is fully OSS, and enables anyone to keep track of their memories over time. I tested mine with 500M tokens of context space (note: not the same as a context window) and only added ~300ms of latency to the inference session. I also built out an open spec called AMP that standardizes the communication protocol for how memory systems should interact with LLMs (see repo).
More than happy to answer any questions and hope you all find this as useful as I have.
reply