ZFS provides several features for Storage Pools that can improve reliability of the storage system and increase the overall performance of reading and writing data. An additional level of reliability can be attained through the use of spare devices that can replace failed storage devices in a zpool. Data read performance can be increased through the use of cache devices. Data writes can be improved with the use of log devices. This article provides a description of each type of device and the commands through which they are configured.
Learning Objectives
- You will learn about hot spare devices and how to configure them.
- You will learn about ARC and L2ARC and how to configure cache devices.
- You will learn about the ZFS Intent Log and how to configure log devices.
You should review the previous article “Building a ZFS Playground” for details on creating a virtual playground to “play” with ZFS. With this playground and information from the “Playing in the ZFS Pool” article, you can use the commands in this article to add and remove spare, cache, and log devices for ZFS Storage Pools.
ZFS Hot Spare Devices
To prevent device failures from negatively impacting performance or corrupting data before a storage administrator can replace the failed device, most storage services have the concept of a “hot spare” device. These devices are powered on but remain unused until another device in the storage system fails. Then, the storage system uses the hot spare to replace the failed device. This replacement can be manual or automatically handled by the storage system. The advantage of having spare devices is improved reliability of the storage system even when devices fail. One disadvantage is that the overall size of the storage array is reduced because the spare devices are not available as part of the overall storage space.
ZFS also supports hot spare devices. Spare devices in ZFS can be shared across multiple storage pools. They can also be easily added and removed after a pool has been created. Once a spare device is needed, it will operate as a temporary replacement device until the original failed device is replaced. Then, the spare device is returned to be a spare once again.
Replacing a failed device with a spare can be managed manually or automatically. The default configuration is manual. A storage administrator must manually issue the commands needed to put the spare device into use. In the automatic configuration, ZFS will handle the transition to the spare device when a failed device is detected. To initiate a manual replacement of a failed device, the zpool replace command is used.
The following command is used to create a new zpool with a spare device.
This command shows how to add a spare device to an existing zpool.
This command is used to remove a spare from an existing zpool.
You can share spare devices across more than one pool. The spare is available to the pool that needs it first. The following example shows how to create two pools with one shared spare device.
By default, automatic replacement of failed devices is disabled. This command enables automatic replacement of failed devices.
ZFS Caching and Cache Devices
ZFS uses caching to improve read operations. By default, ZFS uses system memory to create an Adaptive Replacement Cache (ARC). With ZFS, more server memory is better because it will use a majority of the memory to build the ARC and provide more caching. In fact, the default FreeBSD configuration is to use all but 1GB or half of RAM, which ever is more, for the ARC. ZFS can also use storage devices for an extension of the in-memory ARC. This is known as Level-2 ARC, or L2ARC, and consist of additional disks in the storage system. In ZFS terminology, L2ARC devices are referred to as just “cache” devices.
Choosing the right kind of storage device to use for a ZFS cache is important. For better performance consider using solid-state drives (SSD). Larger, rotational storage devices can provide bulk data storage at a lower price point. Faster and more expensive SSDs should be preferred for caching. If the majority of the storage system access is random reads consisting of mostly static data, then additional memory and faster cache devices should be a consideration.
Devices used in a L2ARC configuration can be easily added and removed from a storage pool. Cache devices cannot be mirrored or part of RAID-Z configuration. Since cache data is considered volatile, a cache device can fail without data integrity issues. If the cache device fails, ZFS will issue a read from the base storage devices.
The following command show how to create a cache device when a zpool is created.
This command shows how to add a cache device to an existing zpool.
This command is used to remove a cache device from an existing pool.
ZFS Intent Log and Log Devices
The ZFS Intent Log (ZIL) is a storage logging technique where data that will be written to disk is stored. The log is later written to storage devices as a transaction. The point of having the ZIL is to provide quicker writes for applications that need it. The ZIL is kept in memory, and ZFS can be configured to use storage devices to support the ZIL as secondary log devices. Adding separate log devices adds a location for the ZIL to store specific types of data before it is written to disk. The overall amount of storage needed for the separate log device is quite small, but the devices must be fast with low latency.
Not every application can take advantage of the ZIL. If you have applications that have a large number of asynchronous writes, then the ZIL is not used at all. Applications that use synchronous writes, such as databases, virtual machines, iSCSI targets, and NFS servers (with clients using the ‘sync’ option), uses the ZIL and can benefit from a separate log device.
Note: In our examples, whole virtual disks are used. In reality, you will use smaller, faster disks for separate log devices.
The following command is used to create a log device when a zpool is created.
This command is used to create a mirrored log device when a zpool is created.
This command adds a log device to an existing zpool.
This command removes a log device from an existing zpool
Next Time
In the previous article, I mentioned deduplication and compression would be in this article. I decided to save those topics for another article.
Replacing failed devices with a spare device will be covered in a later article on ZFS maintenance.
References
- Aaron Toponce, Visualizing the ZFS Intent Log
- FreeBSD Handbook, The Z Filesystem (ZFS), Advanced Topics
- Solaris Internals, ZFS Best Practices Guide
- ZFS Build, Explanation of ARC and L2ARC
ikawnoclastic thoughts 