Beware of Varnish hit_for_pass

Little back story:
I am very much fond of Move Fast and Break Things. I am playing with varnish last 3 months. We let go 2 programmers this month and the work load increase 2x. This should not be an excuse but reason why I did not get into depth for hit_for_pass. And i paid for it. I paid for 2 nights sleep and lot of unnecessary pains.

I thought, hit_for_pass meant varnish will pass the object and will not cache it. I got this worng info from here https://www.varnish-software.com/static/book/VCL_Basics.html#vcl-request-flow. I marked the section in purple color

So I put the following code in vcl_fetch so that logged user’s requests do not get cached.

if (req.http.cookie ~ "session") {
    return (hit_for_pass);
}

After that, the nightmare started. Users started reporting that they could see other users page. They could post as other users. I stop our site and start investigating. I could not figure out why it was happening. Then I figure out varnish was sending cached page to multiple users. The worst part was – varnish was sending session cookie. Fortunately we stopped server with in minutes. I easily reverted the changes done with in those 10 mins.

After much research i figure out what hit_for_pass really means. You can get a good explanation here http://stackoverflow.com/questions/12691489/varnish-hit-for-pass-mean

I removed all the sessions from our memchached server. I Renamed the session cookie, added entropy for session. I also added two extra checks in code so that we always serve correct content for the user. I added extra code so that all the users who were logged in during that time had to reset their password.

This is what I am using right now.

if (req.http.cookie ~ "session") {
    set beresp.ttl = 0s;
}

About Zakir Hyder

This entry was written by .

24. November 2014 by Zakir Hyder
Categories: Linux, Varnish | Comments

Comments

  1. […] Little back story: I am very much fond of Move Fast and Break Things. I am playing with varnish last 3 months. We let go 2 programmers this month and the work load increase 2x. This should not be an…  […]