Varnish BackendPolling

BackendPolling can reduce Varnish 503 errors. Varnish poll by opening a new TCP connection to the backend on which we send a preconfigured request, wait for the answer and the connection to be closed by the backend. Only if Varnish get a ‘200’ reply back do we consider the probe good. Any thing other than ‘200’ means backend sick.

While configuring probe be careful of timeout. If backend do not return 200 with in “timeout” then the probe will be marked as sick. Same for .window & .threshold. For example if 4 probe out of 10 marked as sick then the backed will be marked as “Sick“. Then varnish will show 503 error. So its better to keep the .threshold as small as possible.

backend web {
  .host = "www.website.com";  # IP address of your backend (Apache, nginx, etc.)
  .port = "8000";       # Port your backend is listening on
  .connect_timeout = 600s;
  .first_byte_timeout = 600s;
  .between_bytes_timeout = 600s;
  .probe = {
     .url = "/";
     .timeout = 30s;
     .interval = 20s;
     .window = 10;
     .threshold = 4;
  }
}

you can check backend health using this commad

varnishadm debug.health

null

16. September 2014 by Zakir Hyder
Categories: Linux, Varnish | Tags: , , , , | Comments

Be careful of MySQL Query Cache

Configuring MySQL server for high traffic site is very hard. You have to consider many things. One of the pitfall for unweary person like me is query_cache_size. I must confess I thought it would make my mysql server would perform better with query_cache_size. So I gave it 400M.

query_cache_limit = 400M

But I was dead wrong.

The problem was when ever you run INSERT/UPDATE sql, MySQL gets rid of all of the cache. It also locks the Table. This is one of the main features which limits query cache effectiveness.

If you see “Waiting for query cache lock” when you run SHOW ENGINE INNODB STATUS\G. Then you know you have to set it 0.

For more details http://www.percona.com/blog/2006/07/27/mysql-query-cache/

11. September 2014 by Zakir Hyder
Categories: Linux, MySql, Server Management | Tags: , , | Comments

Prevent cronjobs from overlapping in Linux

Recently while fixing a bug, I found out that there are multiple copies of same cronjob running at the same time. Normally, the task completes in just a few seconds. If the task takes longer than a minute, however, we end up with multiple copies running at once. This situation can make havoc in you app. Its better to introduce some kind of locking mechanism. flock(1) is the right tool for that.

*/1 * * * * /usr/bin/flock -n /tmp/cronjob.lockfile /usr/bin/php /var/www/some.php >> /root/cronjob.txt 

The -n option tells flock not to wait for the lock, but to exit. Simply this one liner made my life way easier.

14. August 2014 by Zakir Hyder
Categories: Linux | Tags: , , , | Comments

Time Synchronisation with NTP in Ubuntu Server

Recently during some bug fixing, I found out that our database server’s time is 5 minutes off. First I thought probably the timezone was correct. Then I found out that for Ubuntu server you have to synchronisation the time. Every time server restarts, it update it’s time using ntpdate. You can do it using following code

ntpdate ntp.ubuntu.com

You can setup ntp, which will automatically update server’s time.

sudo apt-get install ntp

Then edit /etc/ntp.conf to add/remove server lines. By default these servers are configured:

# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
server 0.ubuntu.pool.ntp.org
server 1.ubuntu.pool.ntp.org
server 2.ubuntu.pool.ntp.org
server 3.ubuntu.pool.ntp.org

You can update these server according to your time zone from http://www.pool.ntp.org/zone/@. After you changing the conf file, reload the ntp

sudo /etc/init.d/ntp reload

11. August 2014 by Zakir Hyder
Categories: Linux, Ubuntu | Tags: , , , , | Comments

Fixing Varnish’s 503 Error

I have been trying to speed up somewhereinblog.com. Among many other things, I am trying to add Varnish cache. While setting Varnish on our test server, I have been facing 503 error. Basically this means that backend not responding correctly or taking more time.

This is what I got from varnishlog -c -m TxStatus:503

varnishlog -c -m TxStatus:503
...
3 VCL_call     c miss fetch
3 Backend      c 12 default default
3 FetchError   c http format error
3 VCL_call     c error deliver
3 VCL_call     c deliver deliver
3 TxProtocol   c HTTP/1.1
3 TxStatus     c 503

After many trail and error, I finally figure out the problem. Here is the solution worked for me.

First I tried to add .first_byte_timeout, .connect_timeout & .between_bytes_timeout in /etc/varnish/default.vcl because I know backend takes 1-2 secs to load.

backend default {
	.host = "127.0.0.1";  # IP address of your backend (Apache, nginx, etc.)
	.port = "8080";       # Port your backend is listening on
	.connect_timeout = 10s;
	.first_byte_timeout = 20s;
	.between_bytes_timeout = 10s;
}

Still the problem was not solved. Then I tried to add http_max_hdr. Default value http_max_hdr is 64. I increased it to 4 folds and added to /etc/default/varnish

DAEMON_OPTS="-a :80 \
    -T localhost:6082 \
    -f /etc/varnish/default.vcl \
    -u www-data -g www-data \
    -S /etc/varnish/secret \
    -p http_max_hdr=256 \
    -s malloc,2G"

After that 503 went away :D.

12. May 2014 by Zakir Hyder
Categories: Caching, Varnish | Tags: , , , , , , | Comments

Setup Sub Domain in OSX 10.9 Mavericks

First of all, setup Apache, MySQL, PHP in you mac. You can do that by following http://blog.jambura.com/2014/01/17/get-apache-mysql-php-with-mcrypt-and-phpmyadmin-working-on-osx-10-9-mavericks/.

Edit your /private/etc/hosts file

sudo nano /private/etc/hosts

It should look like this
null

If you put swi.dev in you browser you will see “It works”. We have to update our /private/etc/apache2/httpd.conf to allow subdomains

sudo nano +477 /private/etc/apache2/httpd.conf

Uncomment Include /private/etc/apache2/extra/httpd-vhosts.conf line. It should look like this
null

Now we have to edit httpd-vhosts.conf to point document root for swi.dev.

sudo nano  /private/etc/apache2/extra/httpd-vhosts.conf

then add the following

<VirtualHost *:80>
    DocumentRoot "/Users/zakirhyder/Sites/swi"
    ServerName swi.dev
    ServerAlias www.swi.dev
    ErrorLog "/private/var/log/apache2/swi.dev-error_log"
    UseCanonicalName Off
</VirtualHost>

Now restart apache and your subdomain is ready.

sudo apachectl restart

06. May 2014 by Zakir Hyder
Categories: Apple, Server Management | Tags: , , , , | Comments

Implement Twitter OAuth 1.0A in Laravel using OAuth 1 Lib

If you want to integrate twitter API in your web app, you will find a lot of php libraries which can help you do that. You can see 11 of them in https://dev.twitter.com/docs/twitter-libraries. All the libraries uses OAuth. Some of the libraries are specifically developed for twitter only. If you want to add other OAuth enabled API sites like Xing, tumblr, Pocket, Scoop.it, skyrock in your web app then have to add one library for each of them of atleast update the libraries so that it can work with multiple networks. Instead of doing that you can use OAuth 1 Lib and Guzzle to implement different networks in you web app.

We are going to use https://github.com/zakir-hyder/oauth-1-lib and https://github.com/guzzle/guzzle. In this post I am gong to integrate Twitter. I am going to add more networks in coming weeks. So keep https://github.com/zakir-hyder/laravel-oauth-less-than-two in your watch.

First we need to add OAuth 1 Lib in our Laravel app. If you go to http://blog.jambura.com/2014/04/26/add-your-own-github-library-in-laravel-using-composer/ then you can see how easy it is to add github in library Laravel app. But If you do not want to read whole post then just add following in composer.json

"repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/zakir-hyder/oauth-1-lib"
        }
    ],  
"require": {
		"laravel/framework": "4.1.*",
		"zakirhyder/oauth-1-lib": "dev-master"
	}

then

composer dump-autoload

Before we start, Lets me show you how I configured the Laravel app. First I have created an environment for my MBP according to http://laravel.com/docs/configuration#environment-configuration. You have to changed it according to your computer name. The full code base is here https://github.com/zakir-hyder/laravel-oauth-less-than-two/blob/master/bootstrap/start.php. Then I have added env.local.php file where The Twitter app’s key and secret is saved.

return array(
    'twitter_client_key'     => '',
    'twitter_client_secret' => '',
);

Then these values are added credential.php

return array(
    'twitter_client_key'     => $_ENV['twitter_client_key'],
    'twitter_client_secret' => $_ENV['twitter_client_secret'],
    'twitter_base_url' => 'https://api.twitter.com/oauth/',
);

I added Bootstrap 3.0 in this laravel app. For layout I used http://getbootstrap.com/examples/jumbotron/. I added main.blade.php file in app/views/layouts folder. I added hello.blade.php. Its look like this




Now the interesting parts so when you click the “Click here To Auth & Tweet” button – it will redirect you to TwitterProfileAccountsController‘s getCreate function. I am not going to post the whole code here but you if you want you can open it in another tab https://github.com/zakir-hyder/laravel-oauth-less-than-two/blob/master/app/controllers/TwitterProfileAccountsController.php and go through as I explain every thing.

First I removed twitter_oauth_token,twitter_oauth_token_secret and twitter_request_token from session. First two are need to get access token from twitter. I am saving the access_token in session using twitter_request_token name. The $redirect_url is the url twitter sends the user after the user authorise the app. Then app will redirect the user to getSaveTwitterProfile function. $request_token_url is the url where app will send a get request to get temporary oauth_token and oauth_token_secret.

$consumer = new OAuth\OAuthConsumer($consumer_key, $consumer_secret);
$request = OAuth\OAuthRequest::from_consumer_and_token($consumer, NULL,"GET", $request_token_url, $args);
$request->sign_request(new OAuth\OAuthSignatureMethodHMACSHA1(), $consumer, NULL);
$url = $request->to_url();

The purpose of these lines is to build a url to send request to twitter server. After The url is build, app sends a get request using Guzzle.

$client = new Guzzle\Http\Client($url, array(
	'request.options' => array(
		'verify' => false
	)
));
$response = $client->get()->send();

If the request is successful, we will get oauth_token and oauth_token_secret from the response. We will save them in session. Then app will redirect user to twitter with oauth_token

Session::put('twitter_oauth_token', $request_token['oauth_token']);
Session::put('twitter_oauth_token_secret', $request_token['oauth_token_secret']);
return Redirect::to("https://api.twitter.com/oauth/authorize?oauth_token={$request_token['oauth_token']}");

After user authorise the app Twitter will redirect the user to getSaveTwitterProfile() function which is the $redirect_url in getCreate function. First app will check the oauth_token value with session’s oauth_token. The app will build another url to get the access token

$consumer = new OAuth\OAuthConsumer($consumer_key, $consumer_secret);
$token = new OAuth\OAuthConsumer($oauth_token, $oauth_token_secret);
$request = OAuth\OAuthRequest::from_consumer_and_token($consumer, $token,"GET", $access_token_url, $args);
$request->sign_request(new OAuth\OAuthSignatureMethodHMACSHA1(), $consumer, $token);
$url = $request->to_url();

Then app sends a get request to get the access tokens

$client = new Guzzle\Http\Client($url, array(
	'request.options' => array(
		'verify' => false
	)
));
$response = $client->get()->send();

If the request was successful we will get json encoded access token and we will save it session as twitter_request_token.

$response_body = $response->getBody();
$request_token = OAuth\OAuthUtil::parse_parameters($response_body);
Session::put('twitter_request_token', $request_token);

Now app has the user’s access token, so app can post a tweet in user’s profile. The pricedure is same as before. App Will build a url. But this time app will add a parameter which is our tweet and app will send request as POST.

$args['status'] = "#laravel #oauth #Guzzle https://github.com/zakir-hyder/laravel-oauth-less-than-two";
$consumer = new OAuth\OAuthConsumer($consumer_key, $consumer_secret);
$token = new OAuth\OAuthConsumer($request_token['oauth_token'], $request_token['oauth_token_secret']);
$request = OAuth\OAuthRequest::from_consumer_and_token($consumer, $token,"POST", $post_url, $args);
$request->sign_request(new OAuth\OAuthSignatureMethodHMACSHA1(), $consumer, $token);
$url = $request->get_normalized_http_url();
parse_str($request->to_postdata(), $post_data);

Now app will post the request

$client = new Guzzle\Http\Client();
$request = $client->post($url, false, $post_data);
$request->getCurlOptions()->set(CURLOPT_SSL_VERIFYPEER, false);
$response = $request->send();

BTW you should not use CURLOPT_SSL_VERIFYPEER=false in production server. See http://snippets.webaware.com.au/howto/stop-turning-off-curlopt_ssl_verifypeer-and-fix-your-php-config/ for reason.

Now on you twitter profile you will see “#laravel #oauth #Guzzle https://github.com/zakir-hyder/laravel-oauth-less-than-two” tweet.

01. May 2014 by Zakir Hyder
Categories: Laravel, Library, PHP, Web Development | Tags: , , , , , , , , , , | Comments

← Older posts