LINUX-使用fio工具测试磁盘性能

linux使用fio测试磁盘性能

1、安装fio

centos7使用yum方式安装fio

sudo yum install epel-release -y
sudo yum install fio -y

#安装完成之后，查看fio版本
opt]# fio -v
fio-3.7

ubuntu20.04使用二进制方式安装fio（写这篇文章的时候，fio的版本为rev.3.36）

在fio官网(可能需要使用科学上网)下载fio-3.36.tar.gz文件放到/opt目录下，解压后./configure、make、make install之后就可以使用fio了。

最新版的fio编译需要使用高于11版本的gcc。所以

#ubuntu换源
cp /etc/apt/sources.list /etc/apt/sources.list.bakcup

cat > /etc/apt/sources.list <<EOF
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse
deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse
EOF

apt update

#命令参考
cd /opt
sudo apt install git gcc wget tar
wget https://brick.kernel.dk/snaps/fio-3.36.tar.gz
tar -zxvf fio-3.36.tar.gz
cd /opt/fio-3.36/
./configure #在这一步出错了，等有时间再来研究ubuntu安装fio
make
make install

2、fio参数解释

可以使用fio -help查看每个参数，具体的参数左右可以在官网查看how to文档，如下为几个常见的参数描述

filename=/dev/emcpowerb　支持文件系统或者裸设备，-filename=/dev/sda2或-filename=/dev/sdb
direct=1 测试过程绕过机器自带的buffer，使测试结果更真实
rw=randwread 测试随机读的I/O
rw=randwrite 测试随机写的I/O
rw=randrw 测试随机混合写和读的I/O
rw=read 测试顺序读的I/O
rw=write 测试顺序写的I/O
rw=rw 测试顺序混合写和读的I/O
bs=4k 单次io的块文件大小为4k
bsrange=512-2048 同上，提定数据块的大小范围
size=5g 本次的测试文件大小为5g，以每次4k的io进行测试
numjobs=30 本次的测试线程为30
runtime=1000 测试时间为1000秒，如果不写则一直将5g文件分4k每次写完为止
ioengine=psync io引擎使用pync方式，如果要使用libaio引擎，需要yum install libaio-devel包
rwmixwrite=30 在混合读写的模式下，写占30%
group_reporting 关于显示结果的，汇总每个进程的信息
此外
lockmem=1g 只使用1g内存进行测试
zero_buffers 用0初始化系统buffer
nrfiles=8 每个进程生成文件的数量

网宿平台磁盘性能测试

#100%随机，100%读， 4K
fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=randread -ioengine=psync -bs=4k -size=1000G -numjobs=50 -runtime=180 -group_reporting -name=rand_100read_4k

rand_100read_4k: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 50 threads
Jobs: 50 (f=50): [r(50)][100.0%][r=13.3MiB/s,w=0KiB/s][r=3402,w=0 IOPS][eta 00m:00s]
rand_100read_4k: (groupid=0, jobs=50): err= 0: pid=2095: Tue Jan  2 10:55:32 2024
   read: IOPS=3399, BW=13.3MiB/s (13.9MB/s)(2393MiB/180177msec)
    clat (usec): min=209, max=1248.6k, avg=14689.20, stdev=14291.18
     lat (usec): min=209, max=1248.6k, avg=14690.27, stdev=14291.18
    clat percentiles (msec):
     |  1.00th=[   10],  5.00th=[   12], 10.00th=[   13], 20.00th=[   14],
     | 30.00th=[   14], 40.00th=[   15], 50.00th=[   15], 60.00th=[   15],
     | 70.00th=[   15], 80.00th=[   15], 90.00th=[   15], 95.00th=[   16],
     | 99.00th=[   16], 99.50th=[   26], 99.90th=[  230], 99.95th=[  342],
     | 99.99th=[  592]
   bw (  KiB/s): min=    7, max=  488, per=2.01%, avg=272.73, stdev=38.42, samples=17935
   iops        : min=    1, max=  122, avg=68.13, stdev= 9.61, samples=17935
  lat (usec)   : 250=0.01%, 500=0.04%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.01%, 4=0.01%, 10=1.13%, 20=98.23%, 50=0.22%
  lat (msec)   : 100=0.12%, 250=0.15%, 500=0.07%, 750=0.01%, 1000=0.01%
  cpu          : usr=0.04%, sys=0.12%, ctx=612495, majf=0, minf=59
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=612487,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

#100%随机，100%读， 4K
Run status group 0 (all jobs):
   READ: bw=13.3MiB/s (13.9MB/s), 13.3MiB/s-13.3MiB/s (13.9MB/s-13.9MB/s), io=2393MiB (2509MB), run=180177-180177msec

Disk stats (read/write):
  vdb: ios=612486/0, merge=0/0, ticks=8986690/0, in_queue=8987576, util=100.00%



#100%随机，100%写， 4K
fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=randwrite -ioengine=psync -bs=4k -size=1000G -numjobs=50 -runtime=180 -group_reporting -name=rand_100write_4k
rand_100write_4k: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 50 threads

Jobs: 50 (f=50): [w(50)][100.0%][r=0KiB/s,w=13.3MiB/s][r=0,w=3404 IOPS][eta 00m:00s]
rand_100write_4k: (groupid=0, jobs=50): err= 0: pid=2170: Tue Jan  2 10:59:12 2024
  write: IOPS=3392, BW=13.3MiB/s (13.9MB/s)(2386MiB/180057msec)
    clat (usec): min=616, max=1327.4k, avg=14728.48, stdev=30322.20
     lat (usec): min=617, max=1327.5k, avg=14729.92, stdev=30322.20
    clat percentiles (usec):
     |  1.00th=[  1893],  5.00th=[  5800], 10.00th=[  7635], 20.00th=[  9634],
     | 30.00th=[ 10814], 40.00th=[ 11731], 50.00th=[ 12518], 60.00th=[ 13173],
     | 70.00th=[ 13829], 80.00th=[ 14615], 90.00th=[ 15926], 95.00th=[ 17171],
     | 99.00th=[ 85459], 99.50th=[219153], 99.90th=[484443], 99.95th=[591397],
     | 99.99th=[859833]
   bw (  KiB/s): min=    7, max= 1880, per=2.02%, avg=273.61, stdev=92.74, samples=17826
   iops        : min=    1, max=  470, avg=68.35, stdev=23.18, samples=17826
  lat (usec)   : 750=0.03%, 1000=0.41%
  lat (msec)   : 2=0.60%, 4=1.47%, 10=20.63%, 20=74.80%, 50=0.75%
  lat (msec)   : 100=0.39%, 250=0.50%, 500=0.33%, 750=0.07%, 1000=0.02%
  cpu          : usr=0.05%, sys=0.14%, ctx=610874, majf=0, minf=61
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,610815,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

#100%随机，100%写， 4K
Run status group 0 (all jobs):
  WRITE: bw=13.3MiB/s (13.9MB/s), 13.3MiB/s-13.3MiB/s (13.9MB/s-13.9MB/s), io=2386MiB (2502MB), run=180057-180057msec

Disk stats (read/write):
  vdb: ios=42/610252, merge=0/0, ticks=18/8972595, in_queue=8977620, util=100.00%


#100%顺序，100%读 ，4K
fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=read -ioengine=psync -bs=4k -size=1000G -numjobs=50 -runtime=180 -group_reporting -name=sqe_100read_4k
sqe_100read_4k: (g=0): rw=read, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 50 threads
Jobs: 50 (f=50): [R(50)][100.0%][r=15.6MiB/s,w=0KiB/s][r=4001,w=0 IOPS][eta 00m:00s]
sqe_100read_4k: (groupid=0, jobs=50): err= 0: pid=2233: Tue Jan  2 11:03:01 2024
   read: IOPS=4015, BW=15.7MiB/s (16.4MB/s)(2824MiB/180012msec)
    clat (usec): min=358, max=205087, avg=12446.83, stdev=2301.98
     lat (usec): min=358, max=205088, avg=12447.84, stdev=2301.99
    clat percentiles (usec):
     |  1.00th=[ 9110],  5.00th=[10290], 10.00th=[10814], 20.00th=[11469],
     | 30.00th=[11863], 40.00th=[12256], 50.00th=[12518], 60.00th=[12780],
     | 70.00th=[13173], 80.00th=[13566], 90.00th=[13960], 95.00th=[14353],
     | 99.00th=[15008], 99.50th=[15270], 99.90th=[20841], 99.95th=[27657],
     | 99.99th=[77071]
   bw (  KiB/s): min=  272, max=  408, per=2.00%, avg=321.16, stdev=22.75, samples=17994
   iops        : min=   68, max=  102, avg=80.24, stdev= 5.69, samples=17994
  lat (usec)   : 500=0.01%, 750=0.16%, 1000=0.08%
  lat (msec)   : 2=0.06%, 4=0.04%, 10=3.01%, 20=96.53%, 50=0.07%
  lat (msec)   : 100=0.03%, 250=0.01%
  cpu          : usr=0.05%, sys=0.12%, ctx=722877, majf=0, minf=70
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=722824,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

#100%顺序，100%读 ，4K
Run status group 0 (all jobs):
   READ: bw=15.7MiB/s (16.4MB/s), 15.7MiB/s-15.7MiB/s (16.4MB/s-16.4MB/s), io=2824MiB (2961MB), run=180012-180012msec

Disk stats (read/write):
  vdb: ios=719760/0, merge=1569/0, ticks=8946324/0, in_queue=8952553, util=100.00%

#100%顺序，100%写 ，4K
fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=write -ioengine=psync -bs=4k -size=1000G -numjobs=50 -runtime=180 -group_reporting -name=sqe_100write_4k
sqe_100write_4k: (g=0): rw=write, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 50 threads
Jobs: 50 (f=50): [W(50)][100.0%][r=0KiB/s,w=16.9MiB/s][r=0,w=4334 IOPS][eta 00m:00s]
sqe_100write_4k: (groupid=0, jobs=50): err= 0: pid=2306: Tue Jan  2 11:06:31 2024
  write: IOPS=4111, BW=16.1MiB/s (16.8MB/s)(2891MiB/180012msec)
    clat (usec): min=555, max=2306.0k, avg=12154.13, stdev=30277.32
     lat (usec): min=556, max=2306.0k, avg=12155.45, stdev=30277.32
    clat percentiles (msec):
     |  1.00th=[    4],  5.00th=[    9], 10.00th=[   10], 20.00th=[   11],
     | 30.00th=[   11], 40.00th=[   12], 50.00th=[   12], 60.00th=[   12],
     | 70.00th=[   13], 80.00th=[   13], 90.00th=[   14], 95.00th=[   14],
     | 99.00th=[   18], 99.50th=[   22], 99.90th=[  115], 99.95th=[  300],
     | 99.99th=[ 1905]
   bw (  KiB/s): min=    7, max=  576, per=2.06%, avg=338.78, stdev=49.43, samples=17473
   iops        : min=    1, max=  144, avg=84.65, stdev=12.36, samples=17473
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=0.11%, 4=1.23%, 10=13.20%, 20=84.88%, 50=0.41%
  lat (msec)   : 100=0.05%, 250=0.06%, 500=0.03%, 750=0.01%, 1000=0.01%
  cpu          : usr=0.05%, sys=0.14%, ctx=740186, majf=0, minf=11
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=0,740179,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

#100%顺序，100%写 ，4K
Run status group 0 (all jobs):
  WRITE: bw=16.1MiB/s (16.8MB/s), 16.1MiB/s-16.1MiB/s (16.8MB/s-16.8MB/s), io=2891MiB (3032MB), run=180012-180012msec

Disk stats (read/write):
  vdb: ios=83/736081, merge=0/2169, ticks=24/8934022, in_queue=8939697, util=100.00%

#100%随机，70%读，30%写 4K
fio -filename=/dev/vdb -direct=1 -iodepth 1 -thread -rw=randrw -rwmixread=70 -ioengine=psync -bs=4k -size=1000G -numjobs=50 -runtime=180 -group_reporting -name=randrw_70read_4k
randrw_70read_4k: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=1
...
fio-3.7
Starting 50 threads
Jobs: 50 (f=50): [m(50)][100.0%][r=13.3MiB/s,w=5909KiB/s][r=3407,w=1477 IOPS][eta 00m:00s]
randrw_70read_4k: (groupid=0, jobs=50): err= 0: pid=2368: Tue Jan  2 11:12:45 2024
   read: IOPS=3385, BW=13.2MiB/s (13.9MB/s)(2385MiB/180309msec)
    clat (usec): min=236, max=1150.4k, avg=12791.97, stdev=18310.71
     lat (usec): min=237, max=1150.4k, avg=12793.01, stdev=18310.71
    clat percentiles (usec):
     |  1.00th=[   502],  5.00th=[  5407], 10.00th=[  7701], 20.00th=[ 10028],
     | 30.00th=[ 11076], 40.00th=[ 11731], 50.00th=[ 12387], 60.00th=[ 12911],
     | 70.00th=[ 13304], 80.00th=[ 13829], 90.00th=[ 14222], 95.00th=[ 14484],
     | 99.00th=[ 29754], 99.50th=[ 79168], 99.90th=[304088], 99.95th=[408945],
     | 99.99th=[650118]
   bw (  KiB/s): min=    7, max= 1128, per=2.02%, avg=273.23, stdev=84.06, samples=17844
   iops        : min=    1, max=  282, avg=68.26, stdev=21.02, samples=17844
  write: IOPS=1452, BW=5809KiB/s (5949kB/s)(1023MiB/180309msec)
    clat (usec): min=583, max=1230.8k, avg=4534.97, stdev=33049.41
     lat (usec): min=584, max=1230.8k, avg=4536.15, stdev=33049.42
    clat percentiles (usec):
     |  1.00th=[   693],  5.00th=[   725], 10.00th=[   742], 20.00th=[   775],
     | 30.00th=[   799], 40.00th=[   832], 50.00th=[   865], 60.00th=[   906],
     | 70.00th=[   971], 80.00th=[  1123], 90.00th=[  1713], 95.00th=[  2933],
     | 99.00th=[116917], 99.50th=[242222], 99.90th=[492831], 99.95th=[583009],
     | 99.99th=[876610]
   bw (  KiB/s): min=    7, max=  576, per=2.02%, avg=117.39, stdev=48.38, samples=17811
   iops        : min=    1, max=  144, avg=29.30, stdev=12.10, samples=17811
  lat (usec)   : 250=0.01%, 500=0.70%, 750=3.91%, 1000=18.22%
  lat (msec)   : 2=6.06%, 4=2.45%, 10=12.12%, 20=54.94%, 50=0.68%
  lat (msec)   : 100=0.30%, 250=0.39%, 500=0.20%, 750=0.04%, 1000=0.01%
  cpu          : usr=0.06%, sys=0.15%, ctx=872395, majf=0, minf=15
  IO depths    : 1=100.0%, 2=0.0%, 4=0.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=610525,261861,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=1

#100%随机，70%读，30%写 4K
Run status group 0 (all jobs):
   READ: bw=13.2MiB/s (13.9MB/s), 13.2MiB/s-13.2MiB/s (13.9MB/s-13.9MB/s), io=2385MiB (2501MB), run=180309-180309msec
  WRITE: bw=5809KiB/s (5949kB/s), 5809KiB/s-5809KiB/s (5949kB/s-5949kB/s), io=1023MiB (1073MB), run=180309-180309msec

Disk stats (read/write):
  vdb: ios=610692/261857, merge=0/0, ticks=7799320/1184229, in_queue=8987079, util=100.00%1

#结果参数解读
io=执行了多少M的IO

bw=平均IO带宽
iops=IOPS
runt=线程运行时间
slat=提交延迟
clat=完成延迟
lat=响应时间
bw=带宽
cpu=利用率
IO depths=io队列
IO submit=单个IO提交要提交的IO数
IO complete=Like the above submit number, but for completions instead.
IO issued=The number of read/write requests issued, and how many of them were short.
IO latencies=IO完延迟的分布

io=总共执行了多少size的IO
aggrb=group总带宽
minb=最小.平均带宽.
maxb=最大平均带宽.
mint=group中线程的最短运行时间.
maxt=group中线程的最长运行时间.

ios=所有group总共执行的IO数.
merge=总共发生的IO合并数.
ticks=Number of ticks we kept the disk busy.
io_queue=花费在队列上的总共时间.
util=磁盘利用率

赏

谢谢你请我吃糖果

支付宝

微信

• 本文作者： IYUYI
• 本文链接： https://iyuyi.github.io/2024/01/02/LINUX-使用fio工具测试磁盘性能/
• 版权声明： IYUYI. All rights reserved.