How to Confirm SNS Subscription and Fix PendingConfirmation Subscription

I have been trying to confirm SNS Subscription. But It just keep showing “PendingConfirmation”.

After searching in aws forum, I found that they send a POST request instantly. So I began searching for “Amazon Simple Notification Service Agent” in access log. But there were no records. I figure out that VSF was blocking the request.

So i create a another subscription with backend. Then I can see the request in access log. But still $_POST array was empty. After a number of trial and error, I found that i can only get the POST data using ‘php://input’. Here is the code that worked for me

$json = file_get_contents('php://input');
file_put_contents("file.json",json_encode($json));
$json = json_decode($json,1);

if (!empty($json)) {
    file_put_contents("file.json", 'not empty',FILE_APPEND);
    if($json['Type'] == 'SubscriptionConfirmation') {
        $ch = curl_init();
        curl_setopt($ch, CURLOPT_URL, $json['SubscribeURL']);
        curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36');
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
        // ENABLE HTTP POST
        $postresponse = curl_exec($ch);
        curl_close($ch);

    } 
}

18. December 2014 by Zakir Hyder
Categories: Server Management, Varnish, Web Development | Tags: , , , , , | Comments

Beware of Varnish hit_for_pass

Little back story:
I am very much fond of Move Fast and Break Things. I am playing with varnish last 3 months. We let go 2 programmers this month and the work load increase 2x. This should not be an excuse but reason why I did not get into depth for hit_for_pass. And i paid for it. I paid for 2 nights sleep and lot of unnecessary pains.

I thought, hit_for_pass meant varnish will pass the object and will not cache it. I got this worng info from here https://www.varnish-software.com/static/book/VCL_Basics.html#vcl-request-flow. I marked the section in purple color

So I put the following code in vcl_fetch so that logged user’s requests do not get cached.

if (req.http.cookie ~ "session") {
    return (hit_for_pass);
}

After that, the nightmare started. Users started reporting that they could see other users page. They could post as other users. I stop our site and start investigating. I could not figure out why it was happening. Then I figure out varnish was sending cached page to multiple users. The worst part was – varnish was sending session cookie. Fortunately we stopped server with in minutes. I easily reverted the changes done with in those 10 mins.

After much research i figure out what hit_for_pass really means. You can get a good explanation here http://stackoverflow.com/questions/12691489/varnish-hit-for-pass-mean

I removed all the sessions from our memchached server. I Renamed the session cookie, added entropy for session. I also added two extra checks in code so that we always serve correct content for the user. I added extra code so that all the users who were logged in during that time had to reset their password.

This is what I am using right now.

if (req.http.cookie ~ "session") {
    set beresp.ttl = 0s;
}

24. November 2014 by Zakir Hyder
Categories: Linux, Varnish | Comments

How to Recover SSh Access to Amazon EC2 Instance After UFW firewall Activation But Forgot to Allow SSH

ufw is very good tool. But if you forget to turn on ssh then you would not able to log in to your server. But you forget to enable the ssh in ec2 instance then you have a way to recover it.

  • Stop your instance
  • Attach your EBS volume to another instance. if you don’t have one, create a instance.
  • Create a folder
  • mkdir  recover
    

    Check if the EBS attached.

    sudo fdisk -l
    

    If you see /dev/xvdf then EBS successfully attached.

  • Mount your EBS volume in to recover folder
  • mount /dev/xvdf recover
    
  • Edit recover/etc/ufw/ufw.conf and change enabled=yes to enabled=no
  • Umount the EBS
  • umount -d /dev/xvdf
    
  • Detach from the temp instance
  • Reattach to the original instance. But remember to name it /dev/sda1
  • Restart the instance
  • Login to server and allow ssh
  • ufw allow ssh
    

There you have it. Nice and simple

04. October 2014 by Zakir Hyder
Categories: Linux, Ubuntu | Tags: , , , , , , | Comments

Setting Up Varnish Security Firewall

Varnish can add a extra layer of security as well as HTTP accelerator. https://github.com/comotion/VSF is a Web Application Firewall (WAF) written using the Varnish Control Language (VCL) and a sprinkling of Varnish Modules (vmods). If you want to know how to install vmod go to http://blog.jambura.com/2014/09/28/building-a-varnish-vmod/

To install Varnish Security Firewall we need to install 4 different

If you varnish version is 3.x.x then use https://github.com/fastly/libvmod-urlcode/tree/varnish-3 for libvmod-urlcode.

After you setup all 4 vmods then download the https://github.com/comotion/VSF. I am assuming you are following http://blog.jambura.com/2014/09/28/building-a-varnish-vmod/.

git clone https://github.com/comotion/VSF.git

So the path to VSF’s vcl is /root/varnish/VSF/vcl/. now symlink the vcl directory into /etc/varnish/security

cd /etc/varnish && ln -s /root/varnish/VSF/vcl security

then you edit your default.vcl and add this line near the top:

include "/etc/varnish/security/vsf.vcl";

after that restart varnish.

service varnish restart

01. October 2014 by Zakir Hyder
Categories: Ubuntu, Varnish | Tags: , , , , , , | Comments

Building a Varnish VMOD

Building a Varnish VMOD in ubuntu is bit complicated process. we are going to use varnish-3.0.5. For VMOD we are going to use https://github.com/varnish/libvmod-shield.

Lets create a folder and get the varnish’s source code.

mkdir varnish
cd varnish
apt-get source varnish 

Then we will create a make file from the source

cd varnish-3.0.5
./autogen.sh
./configure
make 

Then we will get code from https://github.com/varnish/libvmod-shield and install the VMOD

cd ..
git clone https://github.com/varnish/libvmod-shield.git 
cd libvmod-shield/
./autogen.sh 
./configure VARNISHSRC=../varnish-3.0.5
make
make install

This will install VMOD shield.

If you see the following error

/usr/lib/x86_64-linux-gnu/varnish/vmods/libvmod_shield.so: 
cannot open shared object file: No such file or directory

then you need to symlink the VMOD in the /usr/local/lib/varnish/vmods/ to /usr/lib/x86_64-linux-gnu/varnish/vmods/libvmod_shield.so

ln -s /usr/local/lib/varnish/vmods/libvmod_shield.so /usr/lib/x86_64-linux-gnu/varnish/vmods/libvmod_shield.so

Now you can use shield in you vcl.

import shield

28. September 2014 by Zakir Hyder
Categories: Linux, Ubuntu, Varnish | Tags: , , , , , , | Comments

Varnish BackendPolling

BackendPolling can reduce Varnish 503 errors. Varnish poll by opening a new TCP connection to the backend on which we send a preconfigured request, wait for the answer and the connection to be closed by the backend. Only if Varnish get a ‘200’ reply back do we consider the probe good. Any thing other than ‘200’ means backend sick.

While configuring probe be careful of timeout. If backend do not return 200 with in “timeout” then the probe will be marked as sick. Same for .window & .threshold. For example if 4 probe out of 10 marked as sick then the backed will be marked as “Sick“. Then varnish will show 503 error. So its better to keep the .threshold as small as possible.

backend web {
  .host = "www.website.com";  # IP address of your backend (Apache, nginx, etc.)
  .port = "8000";       # Port your backend is listening on
  .connect_timeout = 600s;
  .first_byte_timeout = 600s;
  .between_bytes_timeout = 600s;
  .probe = {
     .url = "/";
     .timeout = 30s;
     .interval = 20s;
     .window = 10;
     .threshold = 4;
  }
}

you can check backend health using this commad

varnishadm debug.health

null

16. September 2014 by Zakir Hyder
Categories: Linux, Varnish | Tags: , , , , | Comments

Be careful of MySQL Query Cache

Configuring MySQL server for high traffic site is very hard. You have to consider many things. One of the pitfall for unweary person like me is query_cache_size. I must confess I thought it would make my mysql server would perform better with query_cache_size. So I gave it 400M.

query_cache_limit = 400M

But I was dead wrong.

The problem was when ever you run INSERT/UPDATE sql, MySQL gets rid of all of the cache. It also locks the Table. This is one of the main features which limits query cache effectiveness.

If you see “Waiting for query cache lock” when you run SHOW ENGINE INNODB STATUS\G. Then you know you have to set it 0.

For more details http://www.percona.com/blog/2006/07/27/mysql-query-cache/

11. September 2014 by Zakir Hyder
Categories: Linux, MySql, Server Management | Tags: , , | Comments

← Older posts