Bash | Jason's Web & IT Blog

Archive for the ‘ Bash ’ Category

Checking a URL redirection via its headers

curl --verbose --head --location windeurope.org/dailywind

Gets you:


* About to connect() to windeurope.org port 80 (#0)
*   Trying 213.167.242.87... connected
* Connected to windeurope.org (213.167.242.87) port 80 (#0)
> HEAD /dailywind HTTP/1.1
> User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: windeurope.org
> Accept: */*
>
< HTTP/1.1 302 Found
HTTP/1.1 302 Found
< Date: Wed, 11 Apr 2018 10:18:46 GMT
Date: Wed, 11 Apr 2018 10:18:46 GMT
< Server: Apache
Server: Apache
< Location: https://windeurope.org/dailywind
Location: https://windeurope.org/dailywind
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1

<
* Connection #0 to host windeurope.org left intact
First redirect is from HTTP to HTTPS
* Issue another request to this URL: 'https://windeurope.org/dailywind'
* About to connect() to windeurope.org port 443 (#1)
*   Trying 213.167.242.87... connected
* Connected to windeurope.org (213.167.242.87) port 443 (#1)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=windeurope.org,OU=Gandi Standard SSL,OU=Domain Control Validated
*       start date: Jan 03 00:00:00 2017 GMT
*       expire date: Feb 18 23:59:59 2020 GMT
*       common name: windeurope.org
*       issuer: CN=Gandi Standard SSL CA 2,O=Gandi,L=Paris,ST=Paris,C=FR
> HEAD /dailywind HTTP/1.1
> User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: windeurope.org
> Accept: */*
>
< HTTP/1.1 302 Found
HTTP/1.1 302 Found
< Date: Wed, 11 Apr 2018 10:18:46 GMT
Date: Wed, 11 Apr 2018 10:18:46 GMT
< Server: Apache
Server: Apache
Second redirect to the Daily Wind sub-page
< Location: https://windeurope.org/about-wind/daily-wind/
Location: https://windeurope.org/about-wind/daily-wind/
< Content-Type: text/html; charset=iso-8859-1
Content-Type: text/html; charset=iso-8859-1

<
* Connection #1 to host windeurope.org left intact
* Issue another request to this URL: 'https://windeurope.org/about-wind/daily-wind/'
* Re-using existing connection! (#1) with host windeurope.org
* Connected to windeurope.org (213.167.242.87) port 443 (#1)
> HEAD /about-wind/daily-wind/ HTTP/1.1
> User-Agent: curl/7.19.7 (i386-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: windeurope.org
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Wed, 11 Apr 2018 10:18:46 GMT
Date: Wed, 11 Apr 2018 10:18:46 GMT
< Server: Apache
Server: Apache
< Set-Cookie: wfvt_1398057934=5acde1079f955; expires=Wed, 11-Apr-2018 10:48:47 GMT; path=/; secure; httponly
Set-Cookie: wfvt_1398057934=5acde1079f955; expires=Wed, 11-Apr-2018 10:48:47 GMT; path=/; secure; httponly
< Set-Cookie: PHPSESSID=v62a1us6nrh5u85vl92mdkb551; path=/
Set-Cookie: PHPSESSID=v62a1us6nrh5u85vl92mdkb551; path=/
< Expires: Thu, 19 Nov 1981 08:52:00 GMT
Expires: Thu, 19 Nov 1981 08:52:00 GMT
< Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Pragma: no-cache
Pragma: no-cache
< Link: ; rel="https://api.w.org/"
Link: ; rel="https://api.w.org/"
< Link: ; rel=shortlink
Link: ; rel=shortlink
< Content-Type: text/html; charset=UTF-8
Content-Type: text/html; charset=UTF-8

<
* Connection #1 to host windeurope.org left intact
* Closing connection #0
* Closing connection #1

2018: 04/11
CATEGORY: Apache
Bash
Commands
Linux
Write comment

WindEurope DB backup script

This is the WindEurope database backup script. It’s modified from the standard EWEA script because the DB server is a) not local and b) running on a non-standard port. Therefore we need to make a few changes to the mysql query. We run the script in the same way as usual.

#!/bin/bash
# db.sh
# Backup of database to offsite server
# Jason Bickley, Senior Web Manager
# Wind Europe
# 23 Jan 2016

#==== RECEIVE VARIABLES FROM COMMAND LINE ====#
FREQ=$1
SERVER=$2

#==== SCRIPT OPTIONS ====#
USER=backup
PASS="mwbubCEsxCU6XVsW"
DIR=/root/backup/mysql/
FILE=localhost.sql.gz
DEST=backup.ewea.org
DATE=$(date +"%Y%m%d %T")
LOG=/var/log/backupDB_log

#==== EXCUTION OF COMMANDS ====#
# Change operating directory
cd $DIR

#==== EXPORT DATABASE ====#
mysqldump -h192.168.0.113 --port=13306 -u$USER -p$PASS --all-databases --lock-all-tables | gzip > $SERVER.$FREQ.$FILE
chmod 600 $SERVER.$FREQ.$FILE

#==== RSYNC THE EXPORTED FILE TO OFFSITE SERVER ====#
rsync -aze "ssh -p 10022 -i /root/.ssh/"$SERVER"_prv_key" $SERVER.$FREQ.$FILE root@$DEST:/backups/$FREQ/$SERVER/db/

#==== DELETE THE LOCAL FILE ====#
rm -f $SERVER.$FREQ.$FILE

#==== CONFIRM SUCCESS IN LOG FILE ====#
echo $DATE Backup successful! $DEST:/backups/$FREQ/$SERVER/db/$SERVER.$FREQ.$FILE >> $LOG

To use it, you just have to run: db.sh {freq} {servername} {option}

cd /root/Scripts/backup/
./db.sh daily main

2016: 05/26
CATEGORY: Bash
Commands
Linux
MySQL
TAGS: backup
database
db
export
mysql
rsync
Write comment

EWEA navigation script

This solution is to create a local HTML file from the dynamic www.ewea.org/navigation-file generated by Typo3, that can be rsync’d to other servers so that they may include it in their templates as local files, speeding up the page creation process (and bypassing a huge latency issue we had with the EWEA 2015 programme pages).

#!/bin/sh
curl -L -o /var/www/html/sites/www.ewea.org/navigation http://www.ewea.org/navigation-file
rsync -avze "ssh -p 10022 -i /root/.ssh/main_prv_key" /var/www/html/sites/www.ewea.org/navigation [email protected]:/var/www/html/
rsync -avze "ssh -p 10022 -i /root/.ssh/main_prv_key" /var/www/html/sites/www.ewea.org/navigation [email protected]:/var/www/html/
rsync -avze "ssh -p 10022 -i /root/.ssh/main_prv_key" /var/www/html/sites/www.ewea.org/navigation [email protected]:/var/www/html/

This is scheduled to run daily at midnight via cron

On the receiving server, the following code is then used to display the navigation on the website.

<div id="navigation">
	<?php // INCLUDE EWEA MENU
	include("/var/www/html/navigation");
	?>
</div><!-- end #navigation -->

2015: 11/10
CATEGORY: Bash
Coding
HTML
PHP
Templates
Wordpress
Write comment

Empty swap and clear memory cache

Just because I was curious, I looked up how to empty swap and memory cache today.

To clear the memory cache:

echo 1 > /proc/sys/vm/drop_caches

To empty the swap:

swapoff -a
swapon -a

2015: 10/20
CATEGORY: Bash
Linux
Write comment

Update Gandi Modules

A couple of my Gandi servers refused to restart the firewall due to missing modules (modules didn’t match the kernel version).
Here is a list of commands to rectify this problem.

Update modules:

Change directory to root: cd /
Download the modules’ archive: wget http://mirrors.gandi.net/kernel/$(uname -r)-modules.tar.gz
Move the existing modules: mv -f /lib/modules/$(uname -r) /lib/modules/$(uname -r).old
Uncompress the archive: tar xzf $(uname -r)-modules.tar.gz
Then: depmod $(uname -r)
You can list loaded modules: lsmod
You can check all the previous version of kernel for your server here.

Referenced from here: http://wiki.gandi.net/en/iaas/references/server/kernel_modules?s[]=modules

2015: 09/30
CATEGORY: Bash
iptables
Linux
Write comment

Find original process in Linux

When you need to find the original thread in a list of a hundred child threads, you can use “ps auxc | grep ”

[root@main www.ewea.org]# ps auxc | grep httpd
root      1989  0.0  0.2 319260 12916 ?        Ss   09:24   0:01 httpd
apache    6341  1.4  1.1 483420 69864 ?        S    13:18   3:10 httpd
apache   12844  1.5  1.1 472324 68452 ?        S    14:16   2:32 httpd
apache   12854  1.5  1.0 466824 62652 ?        S    14:16   2:26 httpd
apache   22137  1.6  1.0 465680 61540 ?        S    15:30   1:25 httpd
apache   22145  1.5  1.0 466868 61920 ?        S    15:30   1:17 httpd
apache   22501  1.5  1.0 466620 62464 ?        S    15:32   1:19 httpd
apache   22508  1.5  1.0 369720 62528 ?        S    15:32   1:18 httpd
apache   22509  1.5  1.1 480484 67440 ?        S    15:32   1:17 httpd

2015: 05/06
CATEGORY: Bash
Linux
Write comment

Jason’s default .bashrc file

alias mv='mv -i'
alias df='df -h'
alias du='du -sh'
alias vi=vim
alias scp='scp -P 10022'
alias dir='ls -lah'
alias clearall='history -c;clear'
alias cleard='clear;dir'
alias servicerestart='service httpd restart && service mysqld restart && service vsftpd restart'
alias networkrestart='service iptables restart && service network restart'
alias ipconfig=ifconfig
alias version='cat /etc/redhat-release && uname -a'
alias cdh='cd /var/www/html/sites/www.ewea.org'
alias apachify='chown -R apache:apache . ; find ./ -type d -print -exec chmod 775 {} \;; find ./ -type f -print -exec chmod 664 {} \;'
alias send='rsync -avze "ssh -p 10022 -i /root/.ssh/main_prv_key"'
alias wgett='wget --page-requisites'

2013: 10/16
CATEGORY: Bash
Linux
TAGS: alias bash default
Write comment

Create a local copy of a web page

Using the following command on linux, you can create a local copy of a web page with all images and files saved:

wget --page-requisites http://example.com/

2013: 07/10
CATEGORY: Bash
Linux
Write comment

MySQL commands

2013: 07/10
CATEGORY: Bash
Linux
MySQL
Write comment

Backup server data and rsync to offsite server

This is the current backup script used on all the web servers.
(Last updated 26 May, 2016)

#!/bin/bash
# srv.sh
# Backup of files to offsite server
# Jason Bickley, Senior Web Manager, WindEurope
# 26 MAY 2016

#==== RECEIVE VARIABLES FROM COMMAND LINE ====#
FREQ=$1
SERVER=$2
EXCLUDE="--exclude-from=/root/backup/excludelist"
OPTION=$3
DEST=backup.ewea.org
DATE=$(date +"%Y%m%d %T")
LOG=/var/log/backupSRV_log

#==== SET TIMESTAMP =============#
touch /root/.timestamp /etc/.timestamp /var/.timestamp

#==== DAILY BACKUP OF SERVER ====#
rsync -avzhe "ssh -p 10022 -i /root/.ssh/"$SERVER"_prv_key" $EXCLUDE $OPTION / root@$DEST:/backups/$FREQ/$SERVER/

#=== CONFIRM SUCCESS IN LOG FILE ====#
echo $DATE Backup successful! $DEST:/backups/$FREQ/$SERVER/ >> $LOG

To use it, you just have to run: srv.sh {freq} {servername} {option}
For example:

cd /root/Scripts/backup/
./srv.sh daily web1 --delete

Contents of /root/backup/excludelist

/bin /boot /cgroup /db /dev /lib /lib64 /lost+found /media /mnt /opt /proc /run /sbin /selinux /srv /sys /tmp /usr /var/cache /var/db /var/empty /var/ftp /var/games /var/gandi /var/local /var/lock /var/lost+found /var/mail /var/nis /var/opt /var/lib/php/session /var/preserve /var/run /var/tmp /var/www/html/sites/events /var/www/html/sites/globalwindday.org /var/www/html/sites/solutionwind.com /var/www/html/sites/stats.ewea.org /var/www/html/sites/testing.ewea.org /var/www/html/sites/www.ewea.org /var/www/html/sites/www.ewea.org/fileadmin/videos/health-and-safety /var/yp

2013: 07/09
CATEGORY: Bash
Linux
TAGS: backup
offsite
rsync
script
ssh
Write comment

Information

Change this sentence and title from admin Theme option page.

RSS FEED