How much RAM does my Ubuntu / Fedora Linux desktop PC have?

Here is one of the most frequently asked questions from my mailbag:

Hey I need to know how much ram memory I have in my Ubuntu Linux computer. Under Windows XP I can find out memory by visiting Start > Control Panels > System in control panel. So how do I find out RAM information under Linux PC?

The best way to find out RAM information is open a terminal and type the following command:

$ free -m
OR
$ free -mt

Output:

             total       used       free     shared    buffers     cached

Mem:          2008       1837        170          0         50        963

-/+ buffers/cache:        822       1185

Swap:         1027          0       1027

free command print the total amount of free and used physical and swap memory in the system, as well as the buffers used by the Linux kernel. You can pass following options

• -b switch displays the amount of memory in bytes
• -k switch (set by default) displays it in kilobytes
• -m switch displays it in megabytes.
• -t switch displays a line containing the totals.
• -o switch disables the display of a "buffer adjusted" line. If the -o option is not specified, free subtracts buffer memory from the used memory and adds it to the free memory reported.
• -s switch activates continuous polling delay seconds apart. You may actually specify any floating point number for delay, usleep(3) is used for microsecond resolution delay times.

GUI tool - gnome-system-monitor

The gnome-system-monitor provides an overall view of the resource usage on your system, including memory and CPU allocation. Start gnome-system-monitor from Menus or just type:
$ gnome-system-monitor &



Another Good system monitor - Conky

Conky is a system monitor for X originally based on the torsmo code but kickass :). Run apt-get command to install conky:
$ sudo apt-get install conky
Conky can display just about anything, either on your root desktop or in it’s own window. Conky has many built-in objects, as well as the ability to execute programs and scripts, then display the output from stdout.
To start conky copy default config file and type conky at shell prompt:
$ zcat /usr/share/doc/conky/examples/conkyrc.sample.gz > .conkyrc
$ conky &
OR
$ conky -a left_top &






Read man page of conky for more customization options.

phpSysInfo - PHP script that displays information such as RAM, CPU et all

You can obtained pretty much same information over Internet using webbased php tool called phpSysInfo.

I'd love to hear if anyone has used additional cool GUI tool like conky for this kind of system analysis/monitoring.

How do I find out Linux Resource utilization to detect system bottlenecks?

Q. How can I find out Linux Resource utilization using vmstat command? How do I get information about high disk I/O and memory usage?

A. vmstat command reports information about processes, memory, paging, block IO, traps, and cpu activity. However, a real advantage of vmstat command output - is to the point and (concise) easy to read/understand. The output of vmstat command use to help identify system bottlenecks. Please note that Linux vmstat does not count itself as a running process.

Here is an output of vmstat command from my enterprise grade system:
$ vmstat -S M
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa

3  0      0   1963    607   2359    0    0     0     0    0     1 32  0 68  0

Where,

• The fist line is nothing but six different categories. The second line gives more information about each category. This second line gives all data you need.
• -S M: vmstat lets you choose units (k, K, m, M) default is K (1024 bytes) in the default mode. I am using M since this system has over 4 GB memory. Without -M option it will use K as unit

$ vmstat
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa

3  0      0 2485120 621952 2415368  0    0     0     0    0     1 32  0 68  0

Field Description For Vm Mode

(a) procs is the process-related fields are:

• r: The number of processes waiting for run time.
• b: The number of processes in uninterruptible sleep.

(b) memory is the memory-related fields are:

• swpd: the amount of virtual memory used.
• free: the amount of idle memory.
• buff: the amount of memory used as buffers.
• cache: the amount of memory used as cache.

(c) swap is swap-related fields are:

• si: Amount of memory swapped in from disk (/s).
• so: Amount of memory swapped to disk (/s).

(d) io is the I/O-related fields are:

• bi: Blocks received from a block device (blocks/s).
• bo: Blocks sent to a block device (blocks/s).

(e) system is the system-related fields are:

• in: The number of interrupts per second, including the clock.
• cs: The number of context switches per second.

(f) cpu is the CPU-related fields are:

These are percentages of total CPU time.

• us: Time spent running non-kernel code. (user time, including nice time)
• sy: Time spent running kernel code. (system time)
• id: Time spent idle. Prior to Linux 2.5.41, this includes IO-wait time.
• wa: Time spent waiting for IO. Prior to Linux 2.5.41, shown as zero.

As you see the first output produced gives averages data since the last reboot. Additional reports give information on a sampling period of length delay. You need to sample data using delays i.e. collect data by setting intervals. For example collect data every 2 seconds (or collect data every 2 second 5 times only):

$ vmstat -S M 2
OR
$ vmstat -S M 2 5
Output:

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----

r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa

3  0      0   1756    607   2359    0    0     0     0    0     1 32  0 68  0

3  0      0   1756    607   2359    0    0     0     0 1018    65 38  0 62  0

3  0      0   1756    607   2359    0    0     0     0 1011    64 37  0 63  0

3  0      0   1756    607   2359    0    0     0    20 1018    72 37  0 63  0

3  0      0   1756    607   2359    0    0     0     0 1012    64 37  0 62  0

3  0      0   1756    607   2359    0    0     0     0 1011    65 38  0 63  0

3  0      0   1995    607   2359    0    0     0     0 1012    62 35  2 63  0

3  0      0   1731    607   2359    0    0     0     0 1012    64 34  3 62  0

3  0      0   1731    607   2359    0    0     0     0 1013    72 38  0 62  0

3  0      0   1731    607   2359    0    0     0     0 1013    63 37  0 63  0

This is what most system administrators do to identify system bottlenecks. I hope all of you find vmstat data is concise and easy to read.

How do I Find Out Linux CPU Utilization?

Whenever a Linux system CPU is occupied by a process, it is unavailable for processing other requests. Rest of pending requests must wait till CPU is free. This becomes a bottleneck in the system. Following command will help you to identify CPU utilization, so that you can troubleshoot CPU related performance problems.

Finding CPU utilization is one of the important tasks. Linux comes with various utilities to report CPU utilization. With these commands, you will be able to find out:

* CPU utilization
* Display the utilization of each CPU individually (SMP cpu)
* Find out your system's average CPU utilization since the last reboot etc
* Determine which process is eating the CPU(s)

Old good top command to find out Linux cpu load

The top program provides a dynamic real-time view of a running system. It can display system summary information as well as a list of tasks currently being managed by the Linux kernel.

The top command monitors CPU utilization, process statistics, and memory utilization. The top section contains information related to overall system status - uptime, load average, process counts, CPU status, and utilization statistics for both memory and swap space.

Top command to find out Linux cpu usage

Type the top command:
$ top
Output:

You can see Linux CPU utilization under CPU stats. The task’s share of the elapsed CPU time since the last screen update, expressed as a percentage of total CPU time. In a true SMP environment (multiple CPUS), top will operate in number of CPUs. Please note that you need to type q key to exit the top command display.

The top command produces a frequently-updated list of processes. By default, the processes are ordered by percentage of CPU usage, with only the "top" CPU consumers shown. The top command shows how much processing power and memory are being used, as well as other information about the running processes.

Find Linux CPU utilization using mpstat and other tools

Please note that you need to install special package called sysstat to take advantage of following commands. This package includes system performance tools for Linux (Red Hat Linux / RHEL includes these tools by default).

# apt-get install sysstat
Use up2date command if you are using RHEL:
# up2date sysstat



Display the utilization of each CPU individually using mpstat

If you are using SMP (Multiple CPU) system, use mpstat command to display the utilization of each CPU individually. It report processors related statistics. For example, type command:

# mpstat Output:

Linux 2.6.15.4 (debian)         Thursday 06 April 2006

05:13:05  IST  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s

05:13:05  IST  all   16.52    0.00    2.87    1.09    0.07    0.02    0.00   79.42    830.06

The mpstat command display activities for each available processor, processor 0 being the first one. Global average activities among all processors are also reported. The mpstat command can be used both on SMP and UP machines, but in the latter, only global average activities will be printed.:

# mpstat -P ALL

Output:

Linux 2.6.15.4 (wwwportal1.xxxx.co.in)         Thursday 06 April 2006

05:14:58  IST  CPU   %user   %nice    %sys %iowait    %irq   %soft  %steal   %idle    intr/s

05:14:58  IST  all   16.46    0.00    2.88    1.08    0.07    0.02    0.00   79.48    835.96

05:14:58  IST    0   16.46    0.00    2.88    1.08    0.07    0.02    0.00   79.48    835.96

05:14:58  IST    1   15.77    2.70    3.17    2.01    0.05    0.03    0.00   81.44    822.54

Another output from my HP Dual Opteron 64 bit server:# mpstat -P ALLOutput:

Linux 2.6.5-7.252-smp (ora9.xxx.in)   04/07/06

07:44:18     CPU   %user   %nice %system %iowait    %irq   %soft   %idle    intr/s

07:44:18     all    3.01   57.31    0.36    0.13    0.01    0.00   39.19   1063.46

07:44:18       0    5.87   69.47    0.44    0.05    0.01    0.01   24.16    262.11

07:44:18       1    1.79   48.59    0.36    0.23    0.00    0.00   49.02    268.92

07:44:18       2    2.19   42.63    0.28    0.16    0.01    0.00   54.73    260.96

07:44:18       3    2.17   68.56    0.34    0.06    0.03    0.00   28.83    271.47

Report CPU utilization using sar command

You can display today’s CPU activity, with sar command:
# sar
Output:

Linux 2.6.9-42.0.3.ELsmp (dellbox.xyz.co.in)         01/13/2007

12:00:02 AM       CPU     %user     %nice   %system   %iowait     %idle

12:10:01 AM       all      1.05      0.00      0.28      0.04     98.64

12:20:01 AM       all      0.74      0.00      0.34      0.38     98.54

12:30:02 AM       all      1.09      0.00      0.28      0.10     98.53

12:40:01 AM       all      0.76      0.00      0.21      0.03     99.00

12:50:01 AM       all      1.25      0.00      0.32      0.03     98.40

01:00:01 AM       all      0.80      0.00      0.24      0.03     98.92

...

.....

..

04:40:01 AM       all      8.39      0.00     33.17      0.06     58.38

04:50:01 AM       all      8.68      0.00     37.51      0.04     53.78

05:00:01 AM       all      7.10      0.00     30.48      0.04     62.39

05:10:01 AM       all      8.78      0.00     37.74      0.03     53.44

05:20:02 AM       all      8.30      0.00     35.45      0.06     56.18

Average:          all      3.09      0.00      9.14      0.09     87.68

Comparison of CPU utilization

The sar command writes to standard output the contents of selected cumulative activity counters in the operating system. The accounting system, based on the values in the count and interval parameters. For example display comparison of CPU utilization; 2 seconds apart; 5 times, use:

# sar -u 2 5
Output (for each 2 seconds. 5 lines are displayed):

Linux 2.6.9-42.0.3.ELsmp (www1lab2.xyz.ac.in)         01/13/2007

05:33:24 AM       CPU     %user     %nice   %system   %iowait     %idle

05:33:26 AM       all      9.50      0.00     49.00      0.00     41.50

05:33:28 AM       all     16.79      0.00     74.69      0.00      8.52

05:33:30 AM       all     17.21      0.00     80.30      0.00      2.49

05:33:32 AM       all     16.75      0.00     81.00      0.00      2.25

05:33:34 AM       all     14.29      0.00     72.43      0.00     13.28

Average:          all     14.91      0.00     71.49      0.00     13.61

Where,

• -u 12 5 : Report CPU utilization. The following values are displayed:
• %user: Percentage of CPU utilization that occurred while executing at the user level (application).
• %nice: Percentage of CPU utilization that occurred while executing at the user level with nice priority.
• %system: Percentage of CPU utilization that occurred while executing at the system level (kernel).
• %iowait: Percentage of time that the CPU or CPUs were idle during which the system had an outstanding disk I/O request.
• %idle: Percentage of time that the CPU or CPUs were idle and the system did not have an outstanding disk I/O request.

To get multiple samples and multiple reports set an output file for the sar command. Run the sar command as a background process using.

# sar -o output.file 12 8 >/dev/null 2>&1 &
Better use nohup command so that you can logout and check back report later on:
# nohup sar -o output.file 12 8 >/dev/null 2>&1 &
All data is captured in binary form and saved to a file (data.file). The data can then be selectively displayed ith the sar command using the -f option.
# sar -f data.file



Task: Find out who is monopolizing or eating the CPUs

Finally, you need to determine which process is monopolizing or eating the CPUs. Following command will displays the top 10 CPU users on the Linux system.

# ps -eo pcpu,pid,user,args | sort -k 1 -r | head -10

OR

# ps -eo pcpu,pid,user,args | sort -r -k1 | less 

Output:

%CPU   PID USER     COMMAND

  96  2148 vivek    /usr/lib/vmware/bin/vmware-vmx -C /var/lib/vmware/Virtual Machines/Ubuntu 64-bit/Ubuntu 64-bit.vmx -@ ""

0.7  3358 mysql    /usr/libexec/mysqld --defaults-file=/etc/my.cnf --basedir=/usr --datadir=/var/lib/mysql --user=mysql --pid-file=/var/run/mysqld/mysqld.pid --skip-locking --socket=/var/lib/mysql/mysql.sock

0.4 29129 lighttpd /usr/bin/php

0.4 29128 lighttpd /usr/bin/php

0.4 29127 lighttpd /usr/bin/php

0.4 29126 lighttpd /usr/bin/php

0.2  2177 vivek    [vmware-rtc]

0.0     9 root     [kacpid]

0.0     8 root     [khelper]

Now you know vmware-vmx process is eating up lots of CPU power. ps command displays every process (-e) with a user-defined format (-o pcpu). First field is pcpu (cpu utilization). It is sorted in reverse order to display top 10 CPU eating process.

iostat command

You can also use iostat command which report Central Processing Unit (CPU) statistics and input/output statistics for devices and partitions. It can be use to find out your system's average CPU utilization since the last reboot.

# iostatOutput:

Linux 2.6.15.4 (debian)         Thursday 06 April 2006

avg-cpu:  %user   %nice %system %iowait  %steal   %idle

     16.36    0.00    2.99    1.06    0.00   79.59

Device:            tps   Blk_read/s   Blk_wrtn/s   Blk_read   Blk_wrtn

hda               0.00         0.00         0.00         16          0

hdb               6.43        85.57       166.74     875340    1705664

hdc               0.03         0.16         0.00       1644          0

sda               0.00         0.00         0.00         24          0 

You may want to use following command, which gives you three outputs every 5 seconds (as previous command gives information since the last reboot):$ iostat -xtc 5 3



GUI tools for your laptops/desktops

Above tools/commands are quite useful on remote server. For local system with X GUI installed you can try out gnome-system-monitor. It allows you to view and control the processes running on your system. You can access detailed memory maps, send signals, and terminate the processes.
$ gnome-system-monitor



In addition, the gnome-system-monitor provides an overall view of the resource usage on your system, including memory and CPU allocation.

Get Detailed Information About Particular IP address Connections Using netstat Command

netstat command and shell pipe feature can be used to dig out more information about particular IP address connection. You can find out total established connections, closing connection, SYN and FIN bits and much more. You can also display summary statistics for each protocol using netstat.

This is useful to find out if your server is under attack or not. You can also list abusive IP address using this method.

# netstat -nat | awk '{print $6}' | sort | uniq -c | sort -n

Output:

      1 CLOSE_WAIT

      1 established)

      1 Foreign

      3 FIN_WAIT1

      3 LAST_ACK

     13 ESTABLISHED

     17 LISTEN

    154 FIN_WAIT2

    327 TIME_WAIT

Dig out more information about a specific ip address:
# netstat -nat |grep {IP-address} | awk '{print $6}' | sort | uniq -c | sort -n

      2 LAST_ACK

      2 LISTEN

      4 FIN_WAIT1

     14 ESTABLISHED

     91 TIME_WAIT

    130 FIN_WAIT2

Busy server can give out more information:

# netstat -nat |grep 202.54.1.10 | awk '{print $6}' | sort | uniq -c | sort -n

Output:

  15 CLOSE_WAIT

  37 LAST_ACK

  64 FIN_WAIT_1

  65 FIN_WAIT_2

1251 TIME_WAIT

3597 SYN_SENT

5124 ESTABLISHED

Get List Of All Unique IP Address

To print list of all unique IP address connected to server, enter:
# netstat -nat | awk '{ print $5}' | cut -d: -f1 | sed -e '/^$/d' | uniq
To print total of all unique IP address, enter:
# netstat -nat | awk '{ print $5}' | cut -d: -f1 | sed -e '/^$/d' | uniq | wc -l

Output:

449

Find Out If Box is Under DoS Attack or Not

If you think your Linux box is under attack, print out a list of open connections on your box and sorts them by according to IP address, enter:

# netstat -atun | awk '{print $5}' | cut -d: -f1 | sed -e '/^$/d' |sort | uniq -c | sort -n

Output:

    1 10.0.77.52

      2 10.1.11.3

      4 12.109.42.21

      6 12.191.136.3

.....

...

....

    13 202.155.209.202

     18 208.67.222.222

     28 0.0.0.0

    233 127.0.0.1

You can simply block all abusive IPs using iptables or just null route them.

Get Live View of TCP Connections

You can use tcptrack command to display the status of TCP connections that it sees on a given network interface. tcptrack monitors their state and displays information such as state, source/destination addresses and bandwidth usage in a sorted, updated list very much like the top command.

Display Summary Statistics for Each Protocol

Simply use netstat -s:
# netstat -s | less
# netstat -t -s | less
# netstat -u -s | less
# netstat -w -s | less
# netstat -s

Output:

Ip:

    88354557 total packets received

    0 forwarded

    0 incoming packets discarded

    88104061 incoming packets delivered

    96037391 requests sent out

    13 outgoing packets dropped

    66 fragments dropped after timeout

    295 reassemblies required

    106 packets reassembled ok

    66 packet reassembles failed

    34 fragments failed

Icmp:

    18108 ICMP messages received

    58 input ICMP message failed.

    ICMP input histogram:

        destination unreachable: 7173

        timeout in transit: 472

        redirects: 353

        echo requests: 10096

    28977 ICMP messages sent

    0 ICMP messages failed

    ICMP output histogram:

        destination unreachable: 18881

        echo replies: 10096

Tcp:

    1202226 active connections openings

    2706802 passive connection openings

    7394 failed connection attempts

    47018 connection resets received

    23 connections established

    87975383 segments received

    95235730 segments send out

    681174 segments retransmited

    2044 bad segments received.

    80805 resets sent

Udp:

    92689 packets received

    14611 packets to unknown port received.

    0 packet receive errors

    96755 packets sent

TcpExt:

    48452 invalid SYN cookies received

    7357 resets received for embryonic SYN_RECV sockets

    43 ICMP packets dropped because they were out-of-window

    5 ICMP packets dropped because socket was locked

    2672073 TCP sockets finished time wait in fast timer

    441 time wait sockets recycled by time stamp

    368562 delayed acks sent

    430 delayed acks further delayed because of locked socket

    Quick ack mode was activated 36127 times

    32318597 packets directly queued to recvmsg prequeue.

    741479256 packets directly received from backlog

    1502338990 packets directly received from prequeue

    18343750 packets header predicted

    10220683 packets header predicted and directly queued to user

    17516622 acknowledgments not containing data received

    36549771 predicted acknowledgments

    102672 times recovered from packet loss due to fast retransmit

    Detected reordering 1596 times using reno fast retransmit

    Detected reordering 1 times using time stamp

    8 congestion windows fully recovered

    32 congestion windows partially recovered using Hoe heuristic

    19 congestion windows recovered after partial ack

    0 TCP data loss events

    39951 timeouts after reno fast retransmit

    29653 timeouts in loss state

    197005 fast retransmits

    186937 retransmits in slow start

    131433 other TCP timeouts

    TCPRenoRecoveryFail: 20217

    147 times receiver scheduled too late for direct processing

    29010 connections reset due to unexpected data

    365 connections reset due to early user close

    6979 connections aborted due to timeout

Display Interface Table

You can easily display dropped and total transmitted packets with netstat for eth0:
# netstat --interfaces eth0

Output:

Kernel Interface table

Iface       MTU Met    RX-OK RX-ERR RX-DRP RX-OVR    TX-OK TX-ERR TX-DRP TX-OVR Flg

eth0       1500   0  2040929      0      0      0  3850539      0      0      0 BMRU