You avoid an unnecessary copy. Normal read system call gets the data from disk hardware into the kernel page cache and then copies it into the buffer you provide in your process memory. With mmap, the page cache is mapped directly into your process memory, no copy.
All running processes share the mapped copy of the file.
There are a lot of downsides to mmap: you lose explicit error handling and fine-grained control of when exactly I/O happens. Consult the classic article on why sophisticated systems like DBMSs do not use mmap: https://db.cs.cmu.edu/mmap-cidr2022/
I've never had to use mmap but this is always been the issue in my head. If you're treating I/O as memory pages, what happens when you read a page and it needs to "fault" by reading the backing storage but the storage fails to deliver? What can be said at that point, or does the program crash?
If you fail to load an mmapped page because of an I/O error, Unix-like OSes interrupt your program with SIGBUS/SIGSEGV. It might be technically possible to write a program that would handle those signals and recover, but it seems like a lot more work and complexity than just checking errno after a read system call.
If an I/O error happens with read()/write(), you get back an error code, which SQLite can deal with and pass back up to the application, perhaps accompanied by a reasonable error message. But if you get an I/O error with mmap, you get a signal. SQLite itself ought not be setting signal handlers, as that is the domain of the application and SQLite is just a lowly library. And even if SQLite could set signal handlers, it would be difficult to associate a signal with a particular I/O operation. So there isn't a good way to deal with I/O errors when using mmap(). With mmap(), you just have to assume that the filesystem/mass-storage works flawlessly and never runs out of space.
SQLite can use mmap(). That is a tested and supported capability. But we don't advocate it because of the inability to precisely identify I/O errors and report them back up into the application.
> The operating system must have a unified buffer cache in order for the memory-mapped I/O extension to work correctly, especially in situations where two processes are accessing the same database file and one process is using memory-mapped I/O while the other is not. Not all operating systems have a unified buffer cache. In some operating systems that claim to have a unified buffer cache, the implementation is buggy and can lead to corrupt databases.
What are those OSes with buggy unified buffer caches? More importantly, is there a list of platforms where the use of mmap in sqlite can lead to data loss?
I know that the spirit of HN will strike me down for this, but sqlite is not a "sophisticated system". It assumes the hardware is lawful neutral. Real hardware is chaotic. Sqlite has a good reputation because it is very easy to use. In fact this is the same reason programmers like mmap: it is a hell of a shortcut.
I think the main thing is whether mmap will make sqlite lose data or otherwise corrupt already committed data
... it will if two programs open the same sqlite, one with mmap, and another without https://www.sqlite.org/mmap.html - at least "in some operating systems" (no mention of which ones)
> The operating system must have a unified buffer cache in order for the memory-mapped I/O extension to work correctly, especially in situations where two processes are accessing the same database file and one process is using memory-mapped I/O while the other is not. Not all operating systems have a unified buffer cache. In some operating systems that claim to have a unified buffer cache, the implementation is buggy and can lead to corrupt databases.
Sqlite is otherwise rock solid and won't lose data as easily
* You want your program to crash on any I/O error because you wouldn't handle them anyway
* You value the programming convenience of being able to treat a file on disk as if the entire thing exists in memory
* The performance is good enough for your use. As the article showed, sequential scan performance is as good as direct I/O until the page cache fills up *from a single SSD*, and random access performance is as good as direct I/O until the page cache fills up *if you use MADV_RANDOM*. If your data doesn't fit in memory, or is across multiple storage devices, or you don't correctly advise the OS about your access patterns, mmap will probably be much slower
To be clear, normal I/O still benefits from the OS's shared page cache, where files that other processes have loaded will probably still be in memory, avoiding waiting on the storage device. But each normal I/O process incurs the space and time cost of a copy into its private memory, unlike mmap.
One reason to use shared memory mmap is to ensure that even if your process crashes, the memory stays intact. Another is to communicate between different processes.
Yes. The acronym is “ZTNA” (Zero Trust Network Access).
It is an alternative to a traditional corporate VPN that addresses a few architectural issues; namely:
- L3 connectivity (which allows for lateral movement) to the corporate network.
- Inbound exposure to the VPN gateway (scaling can become a challenge, not to mention continuous vulnerabilities from… certain vendors)
- Policy management can get convoluted if you want to do micro-segmentation properly.
ZTNA is essentially an “inside-out” architecture and acts (kind of) like a L4 proxy. I’m going to butcher this explanation, but:
1. Company installs apps/VMs/containers throughout their network. These must have network reachability to the internal apps/services the company wants to make available to its users.
2. These apps/VMs/containers establish TLS tunnels back to the company’s tenant in the vendor’s cloud.
3. Company rolls out the vendor’s ZTNA client to user devices. This also establishes a TLS tunnel to the vendor’s cloud. Hence the vendor’s cloud is like a MitM gatekeeper.
4. Company creates policies in the vendor’s cloud that says “User A can access App X via app/VM/container Z”
5. Even if App X is on the same LAN segment as App Y, App Y is invisible to User A because connectivity to the internal apps happens at L4.
It is an interesting architecture. That being said, ZTNA solutions have their own issues as well (you can probably already spot some based on my explanation above!)
(Note: I worked for a security vendor that sold a ZTNA solution as part of their ~4-5 years ago. Things could be different now.)
Running LLMs will be slow and training them is basically out of the question. You can get a Framework Desktop with similar bandwidth for less than a third of the price of this thing (though that isn't NVIDIA).
First, if it uses PRNG with a fixed-size state, it isn't accurate to say it never repeats, correct? It will be periodic eventually, even if that takes 2^256 operations or more.
Second, can you go more into the potential practical or theoretical advantages? Your scheme is certainly more complicated, but I don't see how it offers better tamper protection or secrecy than a block cipher operating in an authenticated mode (AES+GCM, for instance). Those have a number of practical advantages, like parallel encryption/decryption and ubiquitous hardware support.
You are correct. The probability of a state collision is cryptographically negligible, on the order of breaking a 256-bit hash function.
You're also right that AES-GCM is faster and has hardware support. Ariadne explores a different trade-off. Its primary advantage is its architectural agility.
Instead of a fixed algorithm, the sequence of operations in Ariadne is dynamic and secret, derived from the key and data history. An attacker doesn't just need to break a key; they have to contend with an unknown, ephemeral algorithm.
This same flexible structure allows the core CVM to be reconfigured into other primitives. We've built concepts for programmable proofs-of-work, verifiable delay functions, and even ring signatures.
FYI your comments seem to be showing up as dead (dead comments don't show up by default, only when people logged into HN have them enabled), I think something may have triggered a shadowban on your account. Might want to send a message to the moderators.
They cannot just relicense the work of all of their public contributors without them agreeing in writing. This is completely illegitimate. (They don't seem to require signing any contributor agreement.)
