Can Varnish automatically determine which storage medium should be used to cache an object?

For instance, I have an incredibly slow and archaic backend provider with requests regularly taking in the tens of seconds. We also have a machine with moderate memory and large storage. Ideally, I'd like to use both memory and storage in order to serve requests.

I found:

https://info.varnish-software.com/blog/partitioning-your-varnish-cache

But that seems to imply I have to manually split the data between the two, which seems challenging?

As indicated in the comments by Gerard H. Pille, the operating system's page cache will make sure the hot data is stored in memory.

File stevedore

In terms of Varnish this means you can use the file stevedore to store the objects.

Here how you would configure this:

varnishd -s file,/var/lib/varnish/storage,250G

I only focused on the -s parameter in the example above. In reality, there will be a lot of parameters used for varnihsd.

Combining malloc & file

If you have a very clear indication what content will only live in memory, you can define multiple stevedors, as illustrated below:

varnishd -s disk=file,/var/lib/varnish/storage,250G -s memory=malloc,32G

In VCL, you can then to the following:

sub backend_response {
    if(beresp.http.Content-Type ~ "^(image|video)/") {
        set beresp.storage = "file";
    } else {
        set beresp.storage = "memory";
    }
}

File stevedore limitations

The file stevedore has the capability of caching a lot of data. The main problem is that it is prone to disk fragmentation over time.

We've seen file-based setups perform really poorly overtime, primarily due to fragmentation.

Another downside is that the operating system's page cache is left guessing on what the right content is that should fit into memory.

And finally, although disks are persistent, the file stevedore isn't: as soon as you restart Varnish the contents of your cache are gone.

Massive Storage Engine

At Varnish Software, we developed Massive Storage Engine, which is an advanced stevedore that tackles the issues that file has.

See https://docs.varnish-software.com/varnish-cache-plus/features/mse/ for more information.

Disclaimer: mse is a proprietary stevedore that is part of Varnish Enterprise. It requires a commercial license, but there are official machine images for AWS, Azure & GCP where you pay by the hour without having to purchase a license key upfront.

MSE is a lot more configurable and was designed with anti-fragmentation in mind. The memory cache doesn't rely on the operating system's page cache and performs better than the standard malloc implementation in Varnish.

The persistence layer can be tailored to your needs, based on multiple storage locations. There is also a module that allows you to select specific storage devices from VCL (see https://docs.varnish-software.com/varnish-cache-plus/vmods/mse/).

And finally, the memory governor feature automatically scales the memory size, based on memory requirements of other parts of Varnish. See https://docs.varnish-software.com/varnish-cache-plus/features/mse/memory_governor/