←back to thread

283 points ghuntley | 4 comments | | HN request time: 0s | source
Show context
bawolff ◴[] No.45133765[source]
Shouldn't you also compare to mmap with huge page option? My understanding is its presicely meant for this circumstance. I don't think its a fair comparison without it.

Respectfully, the title feels a little clickbaity to me. Both methods are still ultimately reading out of memory, they are just using different i/o methods.

replies(2): >>45134007 #>>45138806 #
jared_hulbert ◴[] No.45134007[source]
The original blog post title is intentionally clickbaity. You know, to bait people into clicking. Also I do want to challenge people to really think here.

Seeing if the cached file data can be accessed quickly is the point of the experiment. I can't get mmap() to open a file with huge pages.

void* buffer = mmap(NULL, size_bytes, PROT_READ, (MAP_HUGETLB | MAP_HUGE_1GB), fd, 0); doesn't work.

You can can see my code here https://github.com/bitflux-ai/blog_notes. Any ideas?

replies(2): >>45134269 #>>45134410 #
mastax ◴[] No.45134269[source]
MAP_HUGETLB can't be used for mmaping files on disk, it can only be used with MAP_ANONYMOUS, with a memfd, or with a file on a hugetlbfs pseudo-filesystem (which is also in memory).
replies(2): >>45134451 #>>45135606 #
inetknght ◴[] No.45134451[source]
> MAP_HUGETLB can't be used for mmaping files on disk

False. I've successfully used it to memory-map networked files.

replies(4): >>45134599 #>>45134638 #>>45135603 #>>45140875 #
1. minitech ◴[] No.45134638[source]
That doesn’t sound like the intended meaning of “on disk”.
replies(1): >>45134653 #
2. inetknght ◴[] No.45134653[source]
Kernel doesn't really care about "on disk", it cares about "on filesystem".

The "on disk" distinction is a simplification.

replies(1): >>45134845 #
3. pclmulqdq ◴[] No.45134845[source]
The kernel absolutely does care about the "on disk" distinction because it determines what driver to use.
replies(1): >>45135217 #
4. ddtaylor ◴[] No.45135217{3}[source]
The interface is handled by the kernel.