疑惑
最近在反复搭建ceph集群过程中，总是遇到osd创建不成功的问题，疑似硬盘残留信息，排查中引出了很多陌生的命令，比如vgremove等，于是打算重新了解这部分。
LVM是什么？
逻辑卷管理器（LVM，Logical Volume Manager）是一种把硬盘空间分配成逻辑卷的方法。
看到定义可能还比较懵，不妨结合场景：
有一块系统盘空间随着时间推移容量需要扩容该怎么做？
这时候使用LVM就比较方便了，它可以弹性的调整文件系统的容量，可以整合多个物理分区在一起，让这些分区看起来像是一个磁盘一样。通俗理解就是它可以将很多硬盘/分区全部拿过来作为一个资源池，然后自己再随意划分成逻辑层面的分区，那么这个分区后期进行扩容缩容删除就比较方便了！LVM相关概念1、Physical Volume，PV，物理卷
[root@node3 ~]# fdisk /dev/sdbWelcome to fdisk (util-linux 2.23.2).Changes will remain in memory only, until you decide to write them.Be careful before using the write command.Device does not contain a recognized partition tableBuilding a new DOS disklabel with disk identifier 0x172f2548.Command (m for help): p ###输入 p 查看分区情况Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0x172f2548 Device Boot Start End Blocks Id SystemCommand (m for help): n ###输入 n 创建新分区Partition type: p primary (0 primary, 0 extended, 4 free) e extendedSelect (default p): p ### 默认是 p ，创建主分区，视情况而定Partition number (1-4, default 1): 1 ###这里做第一块主分区First sector (2048-20971519, default 2048):Using default value 2048Last sector, +sectors or +size{K,M,G} (2048-20971519, default 20971519):Using default value 20971519Partition 1 of type Linux and of size 10 GiB is setCommand (m for help): p ###再次输入 p 查看分区情况Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0x172f2548 Device Boot Start End Blocks Id System/dev/sdb1 2048 20971519 10484736 83 LinuxCommand (m for help): w ###最后输入 w 保存从sdb分出来的sdb1 The partition table has been altered!Calling ioctl() to re-read partition table.Syncing disks.fdisk -l查看是否分区成功
[root@node3 ~]# fdisk -lDisk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0x172f2548 Device Boot Start End Blocks Id System/dev/sdb1 2048 20971519 10484736 83 Linux可以看到实际分区Id字段为83，使用时需要通过 fdisk 命令将Id字段调整为8e(LVM 的标识符)，再经过 pvcreate 命令将它转为 LVM 最底层的物理卷(PV)，之后这些PV才能够被利用
[root@node3 ~]# fdisk /dev/sdbWelcome to fdisk (util-linux 2.23.2).Changes will remain in memory only, until you decide to write them.Be careful before using the write command.Command (m for help): t ###输入 t 修改分区类型Selected partition 1Hex code (type L to list all codes): L ###输入 L 列出所有分区类型 0 Empty 24 NEC DOS 81 Minix / old Lin bf Solaris 1 FAT12 27 Hidden NTFS Win 82 Linux swap / So c1 DRDOS/sec (FAT- 2 XENIX root 39 Plan 9 83 Linux c4 DRDOS/sec (FAT- 3 XENIX usr 3c PartitionMagic 84 OS/2 hidden C: c6 DRDOS/sec (FAT- 4 FAT16 <32M 40 Venix 80286 85 Linux extended c7 Syrinx 5 Extended 41 PPC PReP Boot 86 NTFS volume set da Non-FS data 6 FAT16 42 SFS 87 NTFS volume set db CP/M / CTOS / . 7 HPFS/NTFS/exFAT 4d QNX4.x 88 Linux plaintext de Dell Utility 8 AIX 4e QNX4.x 2nd part 8e Linux LVM df BootIt 9 AIX bootable 4f QNX4.x 3rd part 93 Amoeba e1 DOS access a OS/2 Boot Manag 50 OnTrack DM 94 Amoeba BBT e3 DOS R/O b W95 FAT32 51 OnTrack DM6 Aux 9f BSD/OS e4 SpeedStor c W95 FAT32 (LBA) 52 CP/M a0 IBM Thinkpad hi eb BeOS fs e W95 FAT16 (LBA) 53 OnTrack DM6 Aux a5 FreeBSD ee GPT f W95 Ext'd (LBA) 54 OnTrackDM6 a6 OpenBSD ef EFI (FAT-12/16/10 OPUS 55 EZ-Drive a7 NeXTSTEP f0 Linux/PA-RISC b11 Hidden FAT12 56 Golden Bow a8 Darwin UFS f1 SpeedStor12 Compaq diagnost 5c Priam Edisk a9 NetBSD f4 SpeedStor14 Hidden FAT16 <3 61 SpeedStor ab Darwin boot f2 DOS secondary16 Hidden FAT16 63 GNU HURD or Sys af HFS / HFS+ fb VMware VMFS17 Hidden HPFS/NTF 64 Novell Netware b7 BSDI fs fc VMware VMKCORE18 AST SmartSleep 65 Novell Netware b8 BSDI swap fd Linux raid auto1b Hidden W95 FAT3 70 DiskSecure Mult bb Boot Wizard hid fe LANstep1c Hidden W95 FAT3 75 PC/IX be Solaris boot ff BBT1e Hidden W95 FAT1 80 Old MinixHex code (type L to list all codes): 8e ###输入 8e 修改为指定分区类型Changed type of partition 'Linux' to 'Linux LVM'Command (m for help): p ######输入 p 查看分区情况Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectorsUnits = sectors of 1 * 512 = 512 bytesSector size (logical/physical): 512 bytes / 512 bytesI/O size (minimum/optimal): 512 bytes / 512 bytesDisk label type: dosDisk identifier: 0x172f2548 Device Boot Start End Blocks Id System/dev/sdb1 2048 20971519 10484736 8e Linux LVMCommand (m for help): w ###最后输入 w 保存修改的分区类型The partition table has been altered!Calling ioctl() to re-read partition table.Syncing disks.使用pvcreate创建pv
[root@node3 ~]# pvcreate /dev/sdb1 Physical volume "/dev/sdb1" successfully created.2、Volume Group，VG，卷组
VG可以理解为很多物理卷（PV）所组成的组
[root@node3 ~]# vgcreate storage /dev/sdb1 Volume group "storage" successfully created[root@node3 ~]# vgdisplay --- Volume group --- VG Name storage System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 1 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size <10.00 GiB PE Size 4.00 MiB Total PE 2559 Alloc PE / Size 0 / 0 Free PE / Size 2559 / <10.00 GiB VG UUID fK1Wcs-B1BL-TY3G-IRiJ-7JnV-wd0T-CRFx95使用vgextend添加 pv 到 vg
[root@node3 ~]# vgextend storage /dev/sdcsdc sdc1[root@node3 ~]# vgextend storage /dev/sdc1 Physical volume "/dev/sdc1" successfully created. Volume group "storage" successfully extended[root@node3 ~]# vgdisplay --- Volume group --- VG Name storage System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 2 VG Access read/write VG Status resizable MAX LV 0 Cur LV 0 Open LV 0 Max PV 0 Cur PV 2 Act PV 2 VG Size 19.99 GiB PE Size 4.00 MiB Total PE 5118 Alloc PE / Size 0 / 0 Free PE / Size 5118 / 19.99 GiB VG UUID fK1Wcs-B1BL-TY3G-IRiJ-7JnV-wd0T-CRFx95可以看到两个 pv 已经成功添加到 vg 中
3、Logical Volume，LV，逻辑卷
LV 是由 VG 切分而来的，此时 LV 就可以当成一个分区格式化来使用了，
lvcreate 参数
-L 是直接制定容量，基础单位是MB，如果要切割出1G空间，则 -L 1G，MB以此类推。
-l 是以基本单元的个数为单位，每个基本单元的大小是4MB，-l 20，则是切割出4*20=80M空间。
-n 是指定生成逻辑卷的名称。
在卷组中切割出一个：名称为test1，大小为5G的空间：
[root@node3 ~]# lvcreate -n test1 -L 5G storage Logical volume "test1" created.[root@node3 ~]# lvdisplay ###查看lv --- Logical volume --- LV Path /dev/storage/test1 LV Name test1 VG Name storage LV UUID WkSHKt-2GPL-sZX0-ZZeu-KGZB-9Gao-vufFmC LV Write Access read/write LV Creation host, time node3, 2023-02-01 15:22:25 +0800 LV Status available # open 0 LV Size 5.00 GiB Current LE 1280 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 8192 Block device 253:0格式化、挂载使用
[root@node3 ~]# mkfs.ext4 /dev/storage/test1mke2fs 1.42.9 (28-Dec-2013)Filesystem label=OS type: LinuxBlock size=4096 (log=2)Fragment size=4096 (log=2)Stride=0 blocks, Stripe width=0 blocks327680 inodes, 1310720 blocks65536 blocks (5.00%) reserved for the super userFirst data block=0Maximum filesystem blocks=134217728040 block groups32768 blocks per group, 32768 fragments per group8192 inodes per groupSuperblock backups stored on blocks:32768, 98304, 163840, 229376, 294912, 819200, 884736Allocating group tables: doneWriting inode tables: doneCreating journal (32768 blocks): doneWriting superblocks and filesystem accounting information: done[root@node3 ~]# mkdir /mnt/dir1[root@node3 ~]# mount /dev/storage/test1 /mnt/dir1/[root@node3 ~]# df -hFilesystem Size Used Avail Use% Mounted ondevtmpfs 1.9G 0 1.9G 0% /devtmpfs 1.9G 0 1.9G 0% /dev/shmtmpfs 1.9G 20M 1.9G 2% /runtmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup/dev/sda2 19G 2.2G 17G 12% //dev/sda1 1014M 142M 873M 14% /boottmpfs 378M 0 378M 0% /run/user/0/dev/mapper/storage-test1 4.8G 20M 4.6G 1% /mnt/dir14、Physical Extend，PE，物理扩展块
PE 是整个 LVM 最小的存储块，LVM 的 VG 最多仅能含有 65534 个 PE，一个 PE 默认的大小时 4M。
所以可以通过调整 PE 来调整 VG最大容量。
# vgdisplay --- Volume group --- VG Name ceph-77651246-631b-44d8-9ece-1cc0af8e0b68 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 17 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 1 Max PV 0 Cur PV 1 Act PV 1 VG Size 9.00 GiB PE Size 1.00 GiB Total PE 9 Alloc PE / Size 9 / 9.00 GiB Free PE / Size 0 / 0 VG UUID GdTfRy-au4B-DgAN-Ej2X-XgdK-hFNz-wDrLm1LV 扩容、缩容扩容
[root@node3 ~]# umount /mnt/dir1/ ###解除挂载[root@node3 ~]# lvextend -L 15G /dev/storage/test1 ###vg容量不够 Insufficient free space: 2560 extents needed, but only 2558 available[root@node3 ~]# lvextend -L 10G /dev/storage/test1 ###扩展后的分区容量，必须大于原容量 Size of logical volume storage/test1 changed from 5.00 GiB (1280 extents) to 10.00 GiB (2560 extents). Logical volume storage/test1 successfully resized.[root@node3 ~]# e2fsck -f /dev/storage/test1 ###检查硬盘的完整性e2fsck 1.42.9 (28-Dec-2013)Pass 1: Checking inodes, blocks, and sizesPass 2: Checking directory structurePass 3: Checking directory connectivityPass 4: Checking reference countsPass 5: Checking group summary information/dev/storage/test1: 11/327680 files (0.0% non-contiguous), 58462/1310720 blocks[root@node3 ~]# resize2fs /dev/storage/test1 ###重置硬盘容量resize2fs 1.42.9 (28-Dec-2013)Resizing the filesystem on /dev/storage/test1 to 2621440 (4k) blocks.The filesystem on /dev/storage/test1 is now 2621440 blocks long.[root@node3 ~]# mount /dev/storage/test1 /mnt/dir1/ ###重新挂载使用[root@node3 ~]# df -hFilesystem Size Used Avail Use% Mounted ondevtmpfs 1.9G 0 1.9G 0% /devtmpfs 1.9G 0 1.9G 0% /dev/shmtmpfs 1.9G 20M 1.9G 2% /runtmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup/dev/sda2 19G 2.2G 17G 12% //dev/sda1 1014M 142M 873M 14% /boottmpfs 378M 0 378M 0% /run/user/0/dev/mapper/storage-test1 9.8G 23M 9.2G 1% /mnt/dir1缩容
lvextend -L 3G /dev/storage/test1 ###操作参考扩容LV删除、卸载
[root@node3 ~]# umount /mnt/dir1/ ###解除挂载[root@node3 ~]# lvremove /dev/storage/test1 ###删除lvDo you really want to remove active logical volume storage/test1? [y/n]: y Logical volume "test1" successfully removed[root@node3 ~]# vgremove storage ###删除vgDo you really want to remove volume group "storage" containing 1 logical volumes? [y/n]: yDo you really want to remove active logical volume storage/test2? [y/n]: y Logical volume "test2" successfully removed Volume group "storage" successfully removed[root@node3 ~]# pvremove /dev/sdb1 ###删除pv Labels on physical volume "/dev/sdb1" successfully wiped.over!!!