Optimizing Kernel 2.6 and SLES9 for the best possible performance

Chris Mason

SUSE Labs Kernel Team Leader

Novell, Inc.

Introducing the 2.6 Kernel

We will focus on

Identifying problems

Block Layer

VM

Filesystems

SUSE performance

Finding the Problem

Start by finding the bottleneck

CPU

IO

Memory

vmstat for basic system usage

vmstat 1

r b swpd free buff cache si so bi bo in cs us sy id wa

0 0 76804 8268 14996 167396 1 1 36 64 132 197 4 1 92 3

0 0 76804 8268 14996 167396 0 0 0 0 1023 879 3 0 97 0

0 0 76804 8300 14996 167396 0 0 0 0 1158 1134 2 0 98 0

Filesystem Tuning

Pick the right filesystem for the task

Indexed metadata

Small files

Large files

Database and Mail server workloads

Filesystem Tuning

atime updates on files and directories

mount -o noatime,nodiratime

O_DIRECT

Not specific to database workloads

Best with preallocated files

Cache pollution benefits

asynchronous IO

Entirely new model for concurrency

Logging Modes

ReiserFS, ext3

mount -o data=ordered,writeback,journal

ordered mode does not risk exposing old data

writeback mode is fastest in many workloads

journal mode can improve mail server workloads

Dedicated Logging Devices

ReiserFS

mkreiserfs -j /dev/xxx -s 8193 /dev/xxy

reiserfstune –journal-new-device /dev/xxx -s 8193

Ext3

mke2fs -O journal_dev /dev/xxx

mkfs -j -J device=/dev/xxx,size=8193 /dev/xxy

tune2fs -J device=/dev/xxx,size=8193 /dev/xxy

Filesystem Tuning

Split filesystems based on data access patterns

Keep commit heavy data away from data that does not have to be synchronous

Keep streaming writes and reads on different spindles then random IO

iostat -x 1 to identify overloaded drives

Filesystem Tuning

Optimize directory layout for the filesystem

Keep data that will be accessed together in the same subdirectories

Spread data out into different subdirectories to increase large file concurrency

reiserfs – mount -o alloc=oid_groups,skip_busy for filesystem with only a few huge files

Different filesystems order directories differently

Block Layer Tuning

Use the right elevator

CFQ – default in SLES9

AS – deafult in mainline kernels

deadline noop

Latency vs throughput

http://www.finux.org/Reprints/Reprint-Pratt OLS2004.pdf

Block Layer Tuning

Spreading the load across controllers

Per-target locking for scsi

Software raid bandwidth

Battery backed caching

VM Tuning

I386 Systems

Limited to 32-48GB of ram

Consider 64bit

/proc tunables

/proc/sys/vm/swappiness

/proc/dirty_ratio

/proc/dirty_background_ratio

Huge Pages

Huge pages allow more ram to be referenced at a time

Reduces cpu and memory overhead

Used heavily by databases

/proc/sys/vm/nr_hugepages

mount -t hugetlbfs none /mnt_point

shmget(SHM_HUGETLB)

Profiling

oprofile

Powerful tool for profiling the entire system

Can identify CPU hogs both in the kernel and userspace

opcontrol --vmlinux=/boot/vmlinux-2.6.5-7.139-smp

opcontrol --start

<run test>

opreport --symbols > output

Binding Processes to CPUs

CPU Affinity

Context switching costs

Binding CPUs to a specific process can improve performance

taskset 0x3 <-p pid> <command>

Binding Interrupts

cat /proc/interrupts

echo 3 > /prop/irq/0/smp_affinity

Network Improvements

Gibabit ethernet

Significant interrupt overhead reduction

ifconfig dev mtu 9000

NFS modes

asynchronous vs synchronous operation

filesystem interaction

TCP vs UDP

Conclusions

Novell/SUSE provides excellent baseline performance

Internal R&D

Partner testing and benchmarks

2004-01-28T12:03:58