LVM With Integration of Hadoop.

3 min readMar 15, 2021

What is LVM?

LVM

LVM stands for Logical Volume Management. It is use to combine multiple individual hard drives or disk partitions (Physical volumes) into a single volume group (VG). That volume group can then be subdivided into logical volumes (LV) or used as a single large volume

What is Hadoop?

Hadoop

Hadoop is an open-source software framework for storing data and running applications on clusters of commodity hardware. It provides massive storage for any kind of data, enormous processing power and the ability to handle virtually limitless concurrent tasks.

Integrating LVM With Hadoop

Integrating LVM with Hadoop to providing Elasticity to DataNode Storage

Hadoop cluster created without using LVM

Hadoop cluster report without LVM

Attaching External Hard disk to OS

5GiB External Volume attached

Creating Physical Volume from External Harddisk

pvcreate <Hard disk name>

Physical volume created

Creating Volume Group

vgcreate <name_for_vg> <hard disk name>

Volume group (vgForDN) created using physical volume

Creating Logical Volume

lvcreate --size <size> --name <name_for_lv> <vg_name>

Logical volume (lvForDN) created from volume group (vgForDN)

Formating Logical volume

mkfs.ext4 <LV Path>

Logical volume (lvForDN) formatted

Mounting logical volume to hadoop datanode directory

mount <LV Path> <datanode_directory_location>

Logical volume (lvForDN) mounted to hadoop datanode shared directory

Hadoop Cluster report

hadoop dfsadmin -report

Hadoop storage comes from LVM

Increasing hadoop storage by increasing logical volume size

Command for increasing logical volume size -

lvextend --size +<size_to_be_increased> <LV Path>

We need to format the increased volume size in Logical volume
Command to format increased volume size -

resize2fs <LV Path>

Hadoop storage size increased by increasing logical volume size

Hence We are able to increase or decrease hadoop storage capacity using LVM