This post about Linux df command opens series of articles for Linux newbies where you’ll find description and usage examples of major Linux commands like df, top, fsck, mount and so on.
Linux df command can be used to display disk usage statistics for the file systems present on the Linux system. It’s handy tool to know which filesystem is consuming how much memory. Also, if a particular filename is picked up and supplied as argument to df command then it displays the disk usage statistics for the file system on which the file resides. This command can be used by the system administrators to know the disk usage status of various file systems on Linux so that proper clean-up and maintenance of the Linux system can be performed. The df command provides various options through which the output can be customized in a way that is most suited to the user.
In this article, we will discuss the df command through practical examples.
Before jumping on to the examples, lets first take a look on how to use the df command. Here is the syntax information of df command from the man page:
df [OPTION]... [FILE]...
So we see that the df command does not require any mandatory argument. The OPTION and FILE arguments are non-mandatory. While the OPTION argument tells the df command to act in a way as specified by the definition of that OPTION, the FILE argument tells the df command to print disk usage of only that file system on which the FILE resides.
NOTE: for those who are new to this type of syntax information, any argument specified in square brackets  are non-mandatory.
1. Basic example
Here is how the df command can be used in its most basic form.
# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda6 29640780 4320704 23814388 16% / udev 1536756 4 1536752 1% /dev tmpfs 617620 888 616732 1% /run none 5120 0 5120 0% /run/lock none 1544044 156 1543888 1% /run/shm
In the output above, the disk usage statistics of all the file systems were displayed when the df command was run without any argument.
The first column specifies the file system name, the second column specifies the total memory for a particular file system in units of 1k-blocks where 1k is 1024 bytes. Used and available columns specify the amount of memory that is in use and is free respectively. The use column specifies the used memory in percentage while the final column ‘Mounted on’ specifies the mount point of a file system.
2. Get the disk usage of file system through a file
As already discussed in the introduction, df can display the disk usage information of a file system if any file residing on that file system is supplied as an argument to it.
Here is an example:
# df test Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda6 29640780 4320600 23814492 16% /
Here is another example:
# df groff.txt Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda6 29640780 4320600 23814492 16% /
We used two different files (residing on same file system) as argument to df command. The output confirms that the df command displays the disk usage of file system on which a file resides.
3. Display inode information
There exists an option -i through which the output of the df command displays the inode information instead of block usage.
# df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda6 1884160 261964 1622196 14% / udev 212748 560 212188 1% /dev tmpfs 216392 477 215915 1% /run none 216392 3 216389 1% /run/lock none 216392 8 216384 1% /run/shm
As we can see in the output above, the inode related information was displayed for each filesystem.
4. Produce a grand total
There exists an option –total through which the output displays an additional row at the end of the output which produces a total for every column.
Here is an example:
# df --total Filesystem 1K-blocks Used Available Use% Mounted on /dev/sda6 29640780 4320720 23814372 16% / udev 1536756 4 1536752 1% /dev tmpfs 617620 892 616728 1% /run none 5120 0 5120 0% /run/lock none 1544044 156 1543888 1% /run/shm total 33344320 4321772 27516860 14%
So we see that the output contains an extra row towards the end of the output and displays total for each column.
5. Produce output in human readable format
There exists an option -h through which the output of df command can be produced in a human readable format.
Here is an example:
# df -h Filesystem Size Used Avail Use% Mounted on /dev/sda6 29G 4.2G 23G 16% / udev 1.5G 4.0K 1.5G 1% /dev tmpfs 604M 892K 603M 1% /run none 5.0M 0 5.0M 0% /run/lock none 1.5G 156K 1.5G 1% /run/shm
So we can see that the output displays the figures in form of ‘G’ (gigabytes), ‘M’ (megabytes) and ‘K’ (kilobytes). This makes the output easy to read and comprehend and thus makes is human readable. Note that the name of the second column is also changed to ‘size’ in order to make it human readable.
Most of Linux distributions comes with Linux Auditing System that makes it possible to track file changes, file accesses as well as system calls. It’s pretty useful functionality for sysadmins who wish to know who and when accessed and/or changed sensitive files like /etc/passwd, /etc/sudoers or others.
Daemon auditd that usually runs in background and starts after reboot by default logs those events into /var/log/audit.log file (or into other file if different syslog facility is specified). The common usage is to list all files which should be watched and search auditd’s logs from time to time. For example, I prefer to track any file changes into /etc/passwd, reading/writing of /etc/sudoers, executing of /bin/some/binary or just everything (read, write, attributes changes, executing) for my /very/important/file.
In order to configure that you’ll need two commands: auditctl and ausearch. First one is for configuring auditd daemon (e.g. setting a watch on a file), second one is for searching auditd logs (it’s possible to use grep against /var/log/audit.log too but ausearch command makes this task easier).
Install and start Linux Auditing System
If it happened that auditd daemon isn’t installed in your system then you can fix this by one of below commands:
sudo apt-get install audit
sudo yum install audit
The next step is to make sure that auditd is running, if command ps ax | grep [a]udit shows nothing then start auditd using command:
As soon as auditd daemon is started we can start configuring it for tracking file changes using auditctl command.
Make auditd to log all file changes
auditctl -w /etc/passwd -k passwd-ra -p ra
This command will add a rule for auditd daemon to monitor file /etc/passwd file (see option -w /etc/passwd) for reading or changing the atributes (see option -p ra, where r is for read, a is for attribute). Also this command specifies filter key (-k passwd-ra) that will uniquely identify auditd records in its logs files.
Now let’s test this rule: optput the last 20 lines of /etc/passwd file and then search audit log for corresponding records
[root@test artemn]# ausearch -k passwd-ra ---- time->Wed Jul 4 15:17:14 2012 type=CONFIG_CHANGE msg=audit(1341407834.821:207310): auid=500 ses=23783 op="add rule" key="passwd-ra" list=4 res=1 ---- time->Wed Jul 4 15:17:20 2012 type=PATH msg=audit(1341407840.181:207311): item=0 name="/etc/passwd" inode=31982841 dev=09:02 mode=0100644 ouid=0 ogid=0 rdev=00:00 type=CWD msg=audit(1341407840.181:207311): cwd="/home/artemn" type=SYSCALL msg=audit(1341407840.181:207311): arch=c000003e syscall=2 success=yes exit=3 a0=7fffecd41817 a1=0 a2=0 a3=7fffecd40b40 items=1 ppid=642502 pid=521288 auid=500 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=23783 comm="tail" exe="/usr/bin/tail" key="passwd-ra"
As you can see the output of second command shows that auditd has one record for filter key ‘passwd-ra’, it shows that root user (uid=0 gid=0) has read file /etc/passwd using command tail (comm=”tail” exe=”/usr/bin/tail”) at July 4, 2012 (time->Wed Jul 4 15:17:20 2012).
Utility ausearch is pretty powerful so I recommend to read output of man ausearch, in the meantime here are some useful examples:
ausearch -x /bin/grep ausearch -x rm
This approach allows to scan auditd records for certain executable, e.g. if you’d like to see if any of watched files was deleted (or not) using command rm then you should use second command of above two.
This one will show you all records for certain UID (username).
ausearch -ui 1000
cpulimit is a small program written in C that allows to limit CPU usage by Linux process. Limit is specified in percentage so it’s possible to prevent high CPU load generated by scripts, programs or processes.
I found cpulimit pretty useful for the scripts running from cron, for example I can do overnight backups and be sure that compression of 50GB file via gzip won’t eat all CPU resources and all other system processes will have enough CPU time.
In most of Linux distributions cpulimit is available from binary repositories so you can install it using commands:
sudo apt-get install cpulimit
sudo yum install cpulimit
If it’s not possible in your distro then it’s extremely easy to compile it:
cd /usr/src/ wget --no-check-certificate https://github.com/opsengine/cpulimit/tarball/master -O cpulimit.tar tar -xvf cpulimit.tar cd opsengine-cpulimit-9df7758 make ln -s cpulimit /usr/sbin/cpulimit
From that moment you can run commands limited by CPU percentage, e.g. below command executes gzip compression so that gzip process will never step over 10% of CPU limit:
/usr/sbin/cpulimit --limit=10 /bin/gzip vzdump-openvz-102-2012_06_26-19_01_11.tar
You can check actual CPU usage by gzip using commands:
ps axu | grep [g]zip
Btw, the first command contains ‘grep [g]zip’ to avoid the last line in common output:
root 896448 10.0 3.1 159524 3528 ? S 13:12 0:00 /usr/sbin/cpulimit --limit=10 /bin/gzip vzdump-openvz-102-2012_06_26-19_01_11.tar root 26490 0.0 0.0 6364 708 pts/0 S+ 15:24 0:00 grep gzip
Using cpulimit you can also allocate CPU limit to already running processes, e.g. below command will allocate 20% CPU limit to process with PID 2342:
/usr/sbin/cpulimit -p 2342 -l 20
It’s possible to specify process by its executable file instead of PID:
/usr/sbin/cpulimit -P /usr/sbin/nginx -l 30
Cold backup is done when database is not open. While performing cold backup using RMAN the database need to be in MOUNT mode. This is because RMAN needs the data file details which are available while database is in MOUNT mode.
Cold backup is also called consistent backup. This is because before bringing the database in MOUNT mode, the database is first shutdown with IMMEDIATE or TRANSACTIONAL option. This means that database sequence number are all synchronised.
Below are the steps to do RMAN cold backup to an external drive in Linux.
Step1: Mount the external drive
Step2: Run rman_cold_backup.sh in nohup.
The content of rman_cold_backup.sh is below:
# Set Oracle Environment for DB
echo “`date` – Started `basename $0`”
export NLS_DATE_FORMAT=’DD-MON-YY HH24:MI:SS’
rman target /
allocate channel prmy1 type disk format ‘/media/full_backup_%d_%s_%p’;
allocate channel prmy2 type disk format ‘/media/full_backup_%d_%s_%p’;
allocate channel prmy3 type disk format ‘/media/full_backup_%d_%s_%p’;
allocate channel prmy4 type disk format ‘/media/full_backup_%d_%s_%p’;
BACKUP CURRENT CONTROLFILE format ‘/media/ctrl_file_%d_%s_%p’;
BACKUP AS COMPRESSED BACKUPSET DATABASE;
release channel prmy1;
release channel prmy2;
release channel prmy3;
release channel prmy4;
alter database open;
if [ $? -eq 0 ]
echo “RMAN Backup Completed”
echo “RMAN Backup Failed”
echo “`date` – Finished `basename $0`”
As this backup is a cold backup the, the archive log files will be not required to restore and recover the database. Hence archive log files are not added.
However if you wish to take backup of arrive logs as well you can change
BACKUP AS COMPRESSED BACKUPSET DATABASE;
BACKUP AS COMPRESSED BACKUPSET DATABASE PLUS ARCHIVELOG;
In the above script.
Remember to run the script in nohup.
Also remember to check there is enough space in the external drive.
Restricted shells like rssh and scponly give sysadmin the possibility to limit the operations that Linux user can do, for example you can create user that will be allowed to copy files via scp but won’t be permitted to login into system’s command line. This is quite important security feature that should be considered by every sysadmin to prevent unauthorized activity by users for example over SSH.
If you have some online storage that is used for uploading backup data over scp or rsync/ssh from remote hosts then it is highly recommended to use restricted shells for those incoming connections and make sure that even if the attacker has got username/password (or key) then he (or she!) won’t be able to break into your system.
scponly is extremely simple restricted shell, user account that has scponly binary as its shell won’t be able to do anything except transfer data from remote host via scp protocol or via rsync/scp. rssh provides little bit more features: you can limit users to use selected protocols like scp, sftp, rsync, cvs or rdist either in chroot environment or not.
I prefer using yum or aptitude to install such kind of software like rssh or scponly so the fastest way is to try one of below commands depending on your needs:
apt-get install rssh apt-get install scponly yum install rssh yum install scponly
If there are problems to find desired restricted shell in your Linux distro’s repository then you should download sources and do some ./configure, make and make install. Here are the links: latest rssh .tar.gz, latest scponly .tgz.
scponly doesn’t need any configuration and works out of the box so you just should set it as a shell for user account. Here are some examples.
Create new user account with scponly as shell:
useradd -s /usr/sbin/scponly user1
Modify user account to set rssh as a shell:
usermod -s /usr/sbin/rssh user2
Where /usr/sbin/scponly is binary executable of scponly.
rssh comes with text configuration file usually stored in /etc/rssh.conf. You can either setup per-user settings there or configure global restrictions for all accounts which are using rssh. Default rssh.conf file is well commented so there shouldn’t be any problems to configure rssh as you needs. At the same time, here are some examples.
If you wish to restrict all users to scp and rsync only then you should uncomment lines in rssh.conf like below:
allowscp #allowsftp #allowcvs #allowrdist allowrsync
Now coming to per-user examples. User peter is allowed to use scp protocol only, the following line in rssh.conf will do that:
User ann is allowed to scp, rsync only:
As you can see enabled protocols in per-user setup are specified as 11000 (scp, sftp), 11111 (scp, sftp, cvs, rdist, rsync) or 00000 (no protocols enabled). 022 in above examples specifies umask.
Let’s assume you’ve created user1 and enabled only scp and rsync using rssh. An attempt to access the server via SSH under user1 account will end with the following output:
artiomix$ ssh firstname.lastname@example.org email@example.com's password: This account is restricted by rssh. Allowed commands: scp rsync If you believe this is in error, please contact your system administrator. Connection to 22.214.171.124 closed.
At the same time scp transfers will work without problems:
artiomix$ scp -P 23451 /etc/test.file firstname.lastname@example.org:/tmp email@example.com's password: test.file 100% 983 1.0KB/s 00:00
rssh support chroot environments for scp, rsync and other transfer protocols. It means that you can restrict users not only by command they can use but also by filesystems they reach. For example, user1 can be chrooted to /chroot_user1 so it can’t be used to copy something from /etc or /var/www directories of the server. Here is nice manual about chroot in rssh.
HAProxy is open source proxy that can be used to enable high availability and load balancing for web applications. It was designed especially for high load projects so it is very fast and predictable, HAProxy is based on single-process model.
In this post I’ll describe sample setup of HAProxy: users’ requests are load balanced between two web servers Web1 and Web1, if one of them goes down then all the request are processed by alive server, once dead servers recovers load balancing enables again. See topology to the right.
yum install haproxy
apt-get install haproxy
As soon as HAProxy is installed it’s time to edit its configuration file, usually it’s placed in /etc/haproxy/haproxy.cfg. Official documentation for HAProxy 1.4 (stable) is here.
Here is configuration file to implement setup shown at the diagram and described above:
global user daemon group daemon daemon log 127.0.0.1 daemon listen http bind 126.96.36.199:80 mode http option tcplog log global option dontlognull balance roundrobin clitimeout 60000 srvtimeout 60000 contimeout 5000 retries 3 server web1 web1.example.com:80 check server web2 web2.example.com:80 check cookie web1 insert nocache cookie web2 insert nocache
Let’s stop on most important parts of this configuration file. Section global specifies user and group which will be used to run haproxy process (daemon in our example). Line daemon tells HAProxy to run in background, log 127.0.0.1 daemon specifies syslog facility for sending logs from HAProxy.
Section listen http contains line bind 188.8.131.52:80 that specifies IP address and port that will be used to accept users’ requests (they will be load balanced between Web1 and Web2). Line mode http means that HAProxy will filter all requests different from HTTP and will do load balancing over HTTP protocol.
Line balance roundrobin specifies load balancing algorithm according to which each web server (Web1 and Web2) will be used in turns according to their weights. In our example weights for both servers are the same so load balancing is fair.
Lines server web1 … and server web2 … specify web servers for load balancing and failover, in our case they are load balanced according to round robin algorithm and have the same priority/weight.
The last two lines in configuration files are optional, they makes it possible to preserve cookies, it means for example that if you logged in to web application hosted at Web1 and then HAProxy forwarded your next request to Web2 you will still have logged in session opened as cookies with session id from Web1 will be sent to you from Web2 as well.
In this post you will find set of password managers for Linux which provides secure storage for your passwords for sensitive data. If you still keep the passwords in plain text then you must consider one of available password managers so this article is for you.
KeePassX has been a very popular and famous password manager for Linux for a very long time and still trusted by pretty big number of users. When user launches the KeePassX password manager first it requires to set up of a master password to add an extra layer of security to password storage. As an option you can use a file with encryption key instead of the password. This key file can be used along with the master password to provide stronger security. KeePassX application is rather simple so you can easily create one or more databases which will have a master password and will contain all the login credentials stored encrypted. This manager is considered to be one of the most secure managers. If you’re Ubuntu user just type in terminal the following command:
sudo apt-get install keepassx
Gpassword Manager (GPM) is also one of the most secure and highly rated password managers which have more friendly and easy to use interface that KeePassX. This utility has many features that make it to be a good choice for most of the high level computer users. This password manager allows to set and add favorites into system-tray that is one of the unique features of this application. GPM utility uses the crypto++ method for encryption which can be used in Windows and Linux hence it enables the same database to be used on different platforms without the need to convert anything.
My Passwords is a simple and easy to use utility that allows you to store all your login credentials in an encrypted manner within a file. The most exciting feature of this utility are its speed and no requirement of an installation. Encryption algorithm that is used there is AES. Storage in Derby Database format along with AES encryption gives the user the power to create secure and fast password repository. The interface for this utility is fairly simple.
Fiagaro’s Password Manager 2
Fiagaro’s Password Manager 2 is another powerful tool with strong encryption methods that makes it one of the most secure utility for managing passwords in Linux. Fiagaro’s Password Manager 2 uses the AES-256 encryption of the database files which hold all your login credentials (it uses master password that should be set up once you started the program first).
Gringotts is rather old project: its application for Linux/Unix provides the user the possibility to store his or her notes in secure storage encrypted by symmetrical ciphers. Gringotts has a set of eight different algorithms that can be used to encrypt the desired data. This utility also provides different methods for hashing as well as compression. The interface of Gringotts is not as simple as of other password Managers but still easy to use and most effective for old school bearded Unix users.
It’s hard to believe but I didn’t know about Grub fallback feature. So every time when I needed to reboot remote server into a new kernel I had to test it on local server to make sure it won’t panic on remote unit. And if kernel panic still happened I had to ask somebody who has physical access to the server to reboot the hardware choose proper kernel in Grub. It’s all boring and not healthful – it’s much better to use Grub’s native fallback feature.
Grub is default boot loader in most Linux distributions today, at least major distros like Centos/Fedora/RedHat, Debian/Ubuntu/Mint, Arch use Grub. This makes it possible to use Grub fallback feature just out of the box. Here is example scenario.
There is remote server hosted in New Zealand and you (sitting in Denmark) have access to it over the network only (no console server). In this case you cannot afford that the new kernel makes server unreachable, e.g. if new kernel crash during boot it won’t load network interface drivers so your Linux box won’t appear online until somebody reboots it into workable kernel. Thankfully Grub can be configured to try loading new kernel once and if it fails Grub will load another kernel according to configuration. You can see my example grub.conf below:
default=saved timeout=5 splashimage=(hd0,1)/boot/grub/splash.xpm.gz hiddenmenu fallback 0 1 title Fedora OpenVZ (2.6.32-042stab053.5) root (hd0,1) kernel /boot/vmlinuz-2.6.32-042stab053.5 ro root=UUID=6fbdddf9-307c-49eb-83f5-ca1a4a63f584 rd_MD_UUID=1b9dc11a:d5a084b5:83f6d993:3366bbe4 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=sv-latin1 rhgb quiet crashkernel=auto initrd /boot/initramfs-2.6.32-042stab053.5.img savedefault fallback title Fedora (184.108.40.206-88.fc14.i686) root (hd0,1) kernel /boot/vmlinuz-220.127.116.11-88.fc14.i686 ro root=UUID=6fbdddf9-307c-49eb-83f5-ca1a4a63f584 rd_MD_UUID=1b9dc11a:d5a084b5:83f6d993:3366bbe4 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=sv-latin1 rhgb quiet initrd /boot/initramfs-18.104.22.168-88.fc14.i686.img savedefault fallback
According to this configuration Grub will try to load ‘Fedora OpenVZ’ kernel once and if it fails system will be loaded into good ‘Fedora’ kernel. If ‘Fedora OpenVZ’ loads well you’ll be able to reach the server over the network after reboot. Notice lines ‘default=saved’ and ‘savedefault fallback’ which are mandatory to make fallback feature working.
I’ve heard that official Grub fallback feature may work incorrectly on RHEL5 (and Centos 5) so there is elegant workaround (found here):
1. Add param ‘panic=5′ to your new kernel line so it looks like below:
title Fedora OpenVZ (2.6.32-042stab053.5) root (hd0,1) kernel /boot/vmlinuz-2.6.32-042stab053.5 ro root=UUID=6fbdddf9-307c-49eb-83f5-ca1a4a63f584 rd_MD_UUID=1b9dc11a:d5a084b5:83f6d993:3366bbe4 rd_NO_LUKS rd_NO_LVM rd_NO_DM LANG=en_US.UTF-8 SYSFONT=latarcyrheb-sun16 KEYTABLE=sv-latin1 rhgb quiet crashkernel=auto panic=5 initrd /boot/initramfs-2.6.32-042stab053.5.img
This param will make crashed kernel to reboot itself in 5 seconds.
2. Point default Grub param to good kernel, e.g. ‘default=0′.
3. Type in the following commands (good kernel appears in grub.conf first and new kernel is second one):
# grub grub> savedefault --default=1 --once savedefault --default=1 --once grub> quit
This will make Grub to boot into new kernel once and if it fails it will load good kernel. Now you can reboot the server and make sure it will 100% appear online in a few minutes. I usually prefer native Grub fallback feature but if you see it doesn’t work for you it makes sense to try above mentioned workaround.
Mosh (stands for Mobile Shell) is replacement of SSH for remote connections to Unix/Linux systems. It brings a few noticeable advantages over well known SSH connections. In brief, it’s faster and more responsive, especially on long delay and/or unreliable links.
Key benefits of Mosh
- Stays connected if your IP is changed. Roaming feature of Mosh allows you to move between Internet connections and keep Mosh session online. For example, if your wifi connection changes IP you don’t need to reconnect.
- Keeps session after loosing connection. For example, if you lost Internet connection for some time, or your laptop went offline due to exhausted battery – you’ll be able to pick up previously opened Mosh session easily.
- No root rights needed to use Mosh. Unlike SSH Mosh server is not a daemon that needs to listen on specific port to accept incoming connections from clients. Mosh server and client are executables that could be run by ordinary user.
- The same credentials for remote login. Mosh uses SSH for authorization so in order to open connection you need the same credentials as before.
- Responsive Ctrl+C combination. Unlike SSH Mosh doesn’t fill up network buffers so even if you accidentally requested to output 100 MB file you’ll be able to hit Ctrl+C and stop it immediately.
- Better for slow or lagged links. Have you ever tried to use SSH on satellite link where average RTT is 600 ms or more? Wish Mosh you don’t need to wait until server replies to see your typing. It works in CLI and such programs as vi or emacs so on it makes it possible to do the job slow connections more comfortably.
Well, there are some disadvantages too:
- No IPv6 support.
- UTF-8 only.
Mosh is available for all major Linux distributions, FreeBSD and Mac OS X systems:
Ubuntu (12.04 LTS) or Debian (testing/unstable):
Mac OS X:
P.S. It’s better that combination of SSH and GNU Screen.