running out of varnish

just a quick note that might be helpful when running a website through varnish: for at least a year now varnish has had an issue with serving large files. from a given size somewhere around 600mb a "stevedore" cannot be allocated anymore, resulting in an assertion error being raised and a successful, but otherwise empty response. the problem can be worked around using "pipe" to serve the content in question. in the case of plone your vcl might look like:

if (req.url ~ "(/at_download/|@@download$|\.mp4$|\.mov$)") {
    pipe;
}

this works, but is rather crude, of course. you'll need to somehow catch & specify all potentially big content, making it likely to miss some. unfortunately though, varnish only allows to "pipe" from inside its vcl_recv, so it's not possible to check the actual size after fetching the object from the backend, for example...