←back to thread

283 points ghuntley | 2 comments | | HN request time: 0s | source
Show context
bawolff ◴[] No.45133765[source]
Shouldn't you also compare to mmap with huge page option? My understanding is its presicely meant for this circumstance. I don't think its a fair comparison without it.

Respectfully, the title feels a little clickbaity to me. Both methods are still ultimately reading out of memory, they are just using different i/o methods.

replies(2): >>45134007 #>>45138806 #
jared_hulbert ◴[] No.45134007[source]
The original blog post title is intentionally clickbaity. You know, to bait people into clicking. Also I do want to challenge people to really think here.

Seeing if the cached file data can be accessed quickly is the point of the experiment. I can't get mmap() to open a file with huge pages.

void* buffer = mmap(NULL, size_bytes, PROT_READ, (MAP_HUGETLB | MAP_HUGE_1GB), fd, 0); doesn't work.

You can can see my code here https://github.com/bitflux-ai/blog_notes. Any ideas?

replies(2): >>45134269 #>>45134410 #
mastax ◴[] No.45134269[source]
MAP_HUGETLB can't be used for mmaping files on disk, it can only be used with MAP_ANONYMOUS, with a memfd, or with a file on a hugetlbfs pseudo-filesystem (which is also in memory).
replies(2): >>45134451 #>>45135606 #
inetknght ◴[] No.45134451[source]
> MAP_HUGETLB can't be used for mmaping files on disk

False. I've successfully used it to memory-map networked files.

replies(4): >>45134599 #>>45134638 #>>45135603 #>>45140875 #
loloquwowndueo ◴[] No.45134599[source]
Share your code?
replies(2): >>45134637 #>>45140890 #
inetknght ◴[] No.45134637[source]
I don't work there any more (it was a decade ago) and I'm pretty busy right now with a new job coming up (offered today).

Do you have kernel documentation that says that hugetlb doesn't work for files? I don't see that stated anywhere.

replies(1): >>45136551 #
1. Sesse__ ◴[] No.45136551[source]
It's filesystem-dependent. In particular, tmpfs will work. To the best of my knowledge, no “normal” filesystems (e.g., ext4, xfs) will.
replies(1): >>45145898 #
2. inetknght ◴[] No.45145898[source]
It works fine on my ext4 fs...