Prevent cronjobs from overlapping in Linux

Recently while fixing a bug, I found out that there are multiple copies of same cronjob running at the same time. Normally, the task completes in just a few seconds. If the task takes longer than a minute, however, we end up with multiple copies running at once. This situation can make havoc in you app. Its better to introduce some kind of locking mechanism. flock(1) is the right tool for that.

*/1 * * * * /usr/bin/flock -n /tmp/cronjob.lockfile /usr/bin/php /var/www/some.php >> /root/cronjob.txt 

The -n option tells flock not to wait for the lock, but to exit. Simply this one liner made my life way easier.

14. August 2014 by Zakir Hyder
Categories: Linux | Tags: , , , | Comments

Time Synchronisation with NTP in Ubuntu Server

Recently during some bug fixing, I found out that our database server’s time is 5 minutes off. First I thought probably the timezone was correct. Then I found out that for Ubuntu server you have to synchronisation the time. Every time server restarts, it update it’s time using ntpdate. You can do it using following code

ntpdate ntp.ubuntu.com

You can setup ntp, which will automatically update server’s time.

sudo apt-get install ntp

Then edit /etc/ntp.conf to add/remove server lines. By default these servers are configured:

# Use servers from the NTP Pool Project. Approved by Ubuntu Technical Board
# on 2011-02-08 (LP: #104525). See http://www.pool.ntp.org/join.html for
# more information.
server 0.ubuntu.pool.ntp.org
server 1.ubuntu.pool.ntp.org
server 2.ubuntu.pool.ntp.org
server 3.ubuntu.pool.ntp.org

You can update these server according to your time zone from http://www.pool.ntp.org/zone/@. After you changing the conf file, reload the ntp

sudo /etc/init.d/ntp reload

11. August 2014 by Zakir Hyder
Categories: Linux, Ubuntu | Tags: , , , , | Comments

Fixing Varnish’s 503 Error

I have been trying to speed up somewhereinblog.com. Among many other things, I am trying to add Varnish cache. While setting Varnish on our test server, I have been facing 503 error. Basically this means that backend not responding correctly or taking more time.

This is what I got from varnishlog -c -m TxStatus:503

varnishlog -c -m TxStatus:503
...
3 VCL_call     c miss fetch
3 Backend      c 12 default default
3 FetchError   c http format error
3 VCL_call     c error deliver
3 VCL_call     c deliver deliver
3 TxProtocol   c HTTP/1.1
3 TxStatus     c 503

After many trail and error, I finally figure out the problem. Here is the solution worked for me.

First I tried to add .first_byte_timeout, .connect_timeout & .between_bytes_timeout in /etc/varnish/default.vcl because I know backend takes 1-2 secs to load.

backend default {
	.host = "127.0.0.1";  # IP address of your backend (Apache, nginx, etc.)
	.port = "8080";       # Port your backend is listening on
	.connect_timeout = 10s;
	.first_byte_timeout = 20s;
	.between_bytes_timeout = 10s;
}

Still the problem was not solved. Then I tried to add http_max_hdr. Default value http_max_hdr is 64. I increased it to 4 folds and added to /etc/default/varnish

DAEMON_OPTS="-a :80 \
    -T localhost:6082 \
    -f /etc/varnish/default.vcl \
    -u www-data -g www-data \
    -S /etc/varnish/secret \
    -p http_max_hdr=256 \
    -s malloc,2G"

After that 503 went away :D.

12. May 2014 by Zakir Hyder
Categories: Caching, Varnish | Tags: , , , , , , | Comments

Setup Sub Domain in OSX 10.9 Mavericks

First of all, setup Apache, MySQL, PHP in you mac. You can do that by following http://blog.jambura.com/2014/01/17/get-apache-mysql-php-with-mcrypt-and-phpmyadmin-working-on-osx-10-9-mavericks/.

Edit your /private/etc/hosts file

sudo nano /private/etc/hosts

It should look like this
null

If you put swi.dev in you browser you will see “It works”. We have to update our /private/etc/apache2/httpd.conf to allow subdomains

sudo nano +477 /private/etc/apache2/httpd.conf

Uncomment Include /private/etc/apache2/extra/httpd-vhosts.conf line. It should look like this
null

Now we have to edit httpd-vhosts.conf to point document root for swi.dev.

sudo nano  /private/etc/apache2/extra/httpd-vhosts.conf

then add the following

<VirtualHost *:80>
    DocumentRoot "/Users/zakirhyder/Sites/swi"
    ServerName swi.dev
    ServerAlias www.swi.dev
    ErrorLog "/private/var/log/apache2/swi.dev-error_log"
    UseCanonicalName Off
</VirtualHost>

Now restart apache and your subdomain is ready.

sudo apachectl restart

06. May 2014 by Zakir Hyder
Categories: Apple, Server Management | Tags: , , , , | Comments

Implement Twitter OAuth 1.0A in Laravel using OAuth 1 Lib

If you want to integrate twitter API in your web app, you will find a lot of php libraries which can help you do that. You can see 11 of them in https://dev.twitter.com/docs/twitter-libraries. All the libraries uses OAuth. Some of the libraries are specifically developed for twitter only. If you want to add other OAuth enabled API sites like Xing, tumblr, Pocket, Scoop.it, skyrock in your web app then have to add one library for each of them of atleast update the libraries so that it can work with multiple networks. Instead of doing that you can use OAuth 1 Lib and Guzzle to implement different networks in you web app.

We are going to use https://github.com/zakir-hyder/oauth-1-lib and https://github.com/guzzle/guzzle. In this post I am gong to integrate Twitter. I am going to add more networks in coming weeks. So keep https://github.com/zakir-hyder/laravel-oauth-less-than-two in your watch.

First we need to add OAuth 1 Lib in our Laravel app. If you go to http://blog.jambura.com/2014/04/26/add-your-own-github-library-in-laravel-using-composer/ then you can see how easy it is to add github in library Laravel app. But If you do not want to read whole post then just add following in composer.json

"repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/zakir-hyder/oauth-1-lib"
        }
    ],  
"require": {
		"laravel/framework": "4.1.*",
		"zakirhyder/oauth-1-lib": "dev-master"
	}

then

composer dump-autoload

Before we start, Lets me show you how I configured the Laravel app. First I have created an environment for my MBP according to http://laravel.com/docs/configuration#environment-configuration. You have to changed it according to your computer name. The full code base is here https://github.com/zakir-hyder/laravel-oauth-less-than-two/blob/master/bootstrap/start.php. Then I have added env.local.php file where The Twitter app’s key and secret is saved.

return array(
    'twitter_client_key'     => '',
    'twitter_client_secret' => '',
);

Then these values are added credential.php

return array(
    'twitter_client_key'     => $_ENV['twitter_client_key'],
    'twitter_client_secret' => $_ENV['twitter_client_secret'],
    'twitter_base_url' => 'https://api.twitter.com/oauth/',
);

I added Bootstrap 3.0 in this laravel app. For layout I used http://getbootstrap.com/examples/jumbotron/. I added main.blade.php file in app/views/layouts folder. I added hello.blade.php. Its look like this




Now the interesting parts so when you click the “Click here To Auth & Tweet” button – it will redirect you to TwitterProfileAccountsController‘s getCreate function. I am not going to post the whole code here but you if you want you can open it in another tab https://github.com/zakir-hyder/laravel-oauth-less-than-two/blob/master/app/controllers/TwitterProfileAccountsController.php and go through as I explain every thing.

First I removed twitter_oauth_token,twitter_oauth_token_secret and twitter_request_token from session. First two are need to get access token from twitter. I am saving the access_token in session using twitter_request_token name. The $redirect_url is the url twitter sends the user after the user authorise the app. Then app will redirect the user to getSaveTwitterProfile function. $request_token_url is the url where app will send a get request to get temporary oauth_token and oauth_token_secret.

$consumer = new OAuth\OAuthConsumer($consumer_key, $consumer_secret);
$request = OAuth\OAuthRequest::from_consumer_and_token($consumer, NULL,"GET", $request_token_url, $args);
$request->sign_request(new OAuth\OAuthSignatureMethodHMACSHA1(), $consumer, NULL);
$url = $request->to_url();

The purpose of these lines is to build a url to send request to twitter server. After The url is build, app sends a get request using Guzzle.

$client = new Guzzle\Http\Client($url, array(
	'request.options' => array(
		'verify' => false
	)
));
$response = $client->get()->send();

If the request is successful, we will get oauth_token and oauth_token_secret from the response. We will save them in session. Then app will redirect user to twitter with oauth_token

Session::put('twitter_oauth_token', $request_token['oauth_token']);
Session::put('twitter_oauth_token_secret', $request_token['oauth_token_secret']);
return Redirect::to("https://api.twitter.com/oauth/authorize?oauth_token={$request_token['oauth_token']}");

After user authorise the app Twitter will redirect the user to getSaveTwitterProfile() function which is the $redirect_url in getCreate function. First app will check the oauth_token value with session’s oauth_token. The app will build another url to get the access token

$consumer = new OAuth\OAuthConsumer($consumer_key, $consumer_secret);
$token = new OAuth\OAuthConsumer($oauth_token, $oauth_token_secret);
$request = OAuth\OAuthRequest::from_consumer_and_token($consumer, $token,"GET", $access_token_url, $args);
$request->sign_request(new OAuth\OAuthSignatureMethodHMACSHA1(), $consumer, $token);
$url = $request->to_url();

Then app sends a get request to get the access tokens

$client = new Guzzle\Http\Client($url, array(
	'request.options' => array(
		'verify' => false
	)
));
$response = $client->get()->send();

If the request was successful we will get json encoded access token and we will save it session as twitter_request_token.

$response_body = $response->getBody();
$request_token = OAuth\OAuthUtil::parse_parameters($response_body);
Session::put('twitter_request_token', $request_token);

Now app has the user’s access token, so app can post a tweet in user’s profile. The pricedure is same as before. App Will build a url. But this time app will add a parameter which is our tweet and app will send request as POST.

$args['status'] = "#laravel #oauth #Guzzle https://github.com/zakir-hyder/laravel-oauth-less-than-two";
$consumer = new OAuth\OAuthConsumer($consumer_key, $consumer_secret);
$token = new OAuth\OAuthConsumer($request_token['oauth_token'], $request_token['oauth_token_secret']);
$request = OAuth\OAuthRequest::from_consumer_and_token($consumer, $token,"POST", $post_url, $args);
$request->sign_request(new OAuth\OAuthSignatureMethodHMACSHA1(), $consumer, $token);
$url = $request->get_normalized_http_url();
parse_str($request->to_postdata(), $post_data);

Now app will post the request

$client = new Guzzle\Http\Client();
$request = $client->post($url, false, $post_data);
$request->getCurlOptions()->set(CURLOPT_SSL_VERIFYPEER, false);
$response = $request->send();

BTW you should not use CURLOPT_SSL_VERIFYPEER=false in production server. See http://snippets.webaware.com.au/howto/stop-turning-off-curlopt_ssl_verifypeer-and-fix-your-php-config/ for reason.

Now on you twitter profile you will see “#laravel #oauth #Guzzle https://github.com/zakir-hyder/laravel-oauth-less-than-two” tweet.

01. May 2014 by Zakir Hyder
Categories: Laravel, Library, PHP, Web Development | Tags: , , , , , , , , , , | Comments

Add your own github library in Laravel using composer

Adding a random library is quite easy in Laravel. There are two ways you can add an library. Both ways use composer. I am going to use https://github.com/zakir-hyder/oauth-1-lib as our example library. It is basically https://oauth.googlecode.com/svn/code/php/OAuth.php with OAuth namespace and some other changes.

The Easiest Way

Download the oauth-1-lib. Then create folder in app/. Lets name it “library”. So the path to library folder is app/library. Put the src folder of oauth-1-lib in library folder. Rename src folder to “OAuth”. If you want can rename to whatever you want. Then add the library path in the autoload section of composer.json file.

"autoload": {
	"classmap": [
		"app/commands",
		"app/controllers",
		"app/models",
		"app/library",
		"app/database/migrations",
		"app/database/seeds",
		"app/tests/TestCase.php"
	]
},

Then run the dump-autoload command

composer dump-autoload

Now you can use it in you project.

$consumer = new OAuth\OAuthConsumer($consumer_key, $consumer_secret);

The Cool Way

We will composer’s VCS. But before that we have to add a composer.json in the library. We create a composer.json file and add the following.

{
    "name": "zakirhyder/oauth-1-lib",
    "description": "This library includes all the classes needed for OAuth 1/1.0a",
    "license": "MIT",
    "authors": [
        {
            "name": "Zakir Hyder",
            "email": "zakir.hyder@gmail.com"
        }
    ],
    "minimum-stability": "dev",
    "require": {
        "php": ">=5.3.0"
    },
    "autoload": {
        "psr-0": {
            "OAuth": "src/"
        }
    }
}

Now lets add our library in a laravel project. First we will add repository laravel’s composer.json

"repositories": [
        {
            "type": "vcs",
            "url": "https://github.com/zakir-hyder/oauth-1-lib"
        }
    ],  

Then we add our library in require key

"require": {
		"laravel/framework": "4.1.*",
		"zakirhyder/oauth-1-lib": "dev-master"
	}

One thing about name -I made mistake thinking “zakir-hyder/oauth-1-lib(from the url of https://github.com/zakir-hyder/oauth-1-lib)” is the name I have use. But No I have to use name I used in library’s composer.json.

26. April 2014 by Zakir Hyder
Categories: Composer, Laravel, PHP | Tags: , , , | Comments

Get Apache, MySQL, PHP With mcrypt and phpMyAdmin working on OSX 10.9 Mavericks

One of the best thing about OSX 10.9 Mavericks for web developer is Apache and PHP already installed. You do not have to start the Apache – its automatically starts when mac starts.

Apache
Even though you want to start manually

sudo apachectl start

to stop it

sudo apachectl stop

to restart it

sudo apachectl restart

To find the Apache version

httpd -v

Now you can access the it by http://localhost.

The document root is /Library/WebServer/Documents/. But I would suggest you use User Level Root. First create folder “Sites” in your home folder. Now go to /etc/apache2/users/

cd /etc/apache2/users/

Suppose you username is “zakir”. You can found your name

whoami

Now create a conf file in /etc/apache2/users/

sudo nano zakir.conf

add the following to zakir.conf

<Directory "/Users/zakir/Sites/">
Options Indexes FollowSymLinks MultiViews
AllowOverride all
Order allow,deny
Allow from all
</Directory>

Change permission of the zakir.conf

sudo chmod 644 zakir.conf

Now restart the apache

sudo apachectl restart

Now you access it via http://localhost/~zakir/. Now you need change httpd.conf file to allow .htaccess file.

sudo nano /etc/apache2/httpd.conf

Find the line . Change value of AllowOverride To All.

Permission of the Sites folder

sudo chmod go-rwx Sites
sudo chmod go+x Sites
sudo chgrp -R _www Sites
sudo chmod -R go-rwx Sites
sudo chmod -R g+rx Site

If you have mutiple website ie mutiple folders then do the following after creating each folder

sudo chown -R username ~/Sites/folder
sudo chmod go-rwx ~/Sites/folder
sudo chmod go+x ~/Sites/folder
sudo chgrp -R _www ~/Sites/folder
sudo chmod -R go-rwx ~/Sites/folder
sudo chmod -R g+rx ~/Sites/folder

In this way only you can edit the files and others can not edit files. Apache can also run files. To check permission of a file use following line

stat -f %Mp%Lp file

PHP
Open httpd.conf file

sudo nano /etc/apache2/httpd.conf

Uncomment “LoadModule php5_module libexec/apache2/libphp5.so” line.
Now restart the apache

sudo apachectl restart

Install mcrypt for PHP
First install xcode and xcode for command line see http://blog.jambura.com/2013/02/19/setup-homebrew-perlbrew-ruby-rvm-perl-cpanm-nginx-in-mountain-lion/ for details.

Create mcrypt folder in your home folder.

cd ~
mkdir mcrypt
cd mcrypt

Download libmcrypt from http://sourceforge.net/projects/mcrypt/files/Libmcrypt/2.5.8/libmcrypt-2.5.8.tar.gz/download and download php source http://museum.php.net/php5/php-5.4.17.tar.gz. Copy the files to mcrypt folder.
Untar the files

tar -zxvf libmcrypt-2.5.8.tar.gz
tar -zxvf php-5.4.17.tar.gz

Configuring libmcrypt

cd libmcrypt-2.5.8
./configure
make
sudo make install

Compile mcrypt php Extension

cd ../php-5.4.17/ext/mcrypt/
make
sudo make install

You will see this msg

Installing shared extensions:     /usr/lib/php/extensions/no-debug-non-zts-20100525/

Copy php.ini from /etc

sudo cp /etc/php.ini.default /etc/php.ini
sudo chmod u+w  /etc/php.ini

Open /etc/php.ini and add the line below at the end

extension=mcrypt.so

Now restart the apache

sudo apachectl restart

MySQL
First download dmg file from http://dev.mysql.com/downloads/mysql/. Install Install all 3…

mysql5.6.xxx.pkg
MySQLstartupitem.pkg
MySQLPrefPane

Now got System Preference and then got mysql then start mysql
null

now set mysql password for mysql – I liked to set password blank

/usr/local/mysql/bin/mysqladmin -u root password ''

Fix the 2002 socket error

sudo mkdir /var/mysql
sudo ln -s /tmp/mysql.sock /var/mysql/mysql.sock

Download phpMyAdmin, the zip package and move the folder with its contents into the ~/Sites/phpmyadmin folder.

Make the config folder and Change the permissions

mkdir ~/Sites/phpmyadmin/config
chmod o+w ~/Sites/phpmyadmin/config

Go to http://localhost/~zakir/phpmyadmin/setup/. Click new server, Switch to the Authentication tab and set the local mysql root user and the password. Make sure you click on save, then a config.inc.php is now in the /config directory of phpmyadmin directory, move this file to the root level of /phpmyadmin and then remove the now empty /config directory.

17. January 2014 by Zakir Hyder
Categories: Apple, Server Management, Web Development | Tags: , , , , , , , , , , , , | Comments

← Older posts