[This is another essay-length post about O/Ses and ZFS...] Arthur Krewat <krewat(a)kilonet.net> asks on Thu, 30 May 2019 20:42:33 -0400 about my TUHS list posting about running a large computing facility without user disk quotas, and our experiences with ZFS: First, ZFS on the Solaris family (including DilOS, Dyson, Hipster, Illumian, Omnios, Omnitribblix, OpenIndiana, Tribblix, Unleashed, and XStreamOS), the FreeBSD family (including ClonOS, FreeNAS, GhostBSD, HardenedBSD, MidnightBSD, PCBSD, Trident, and TrueOS), and on GNU/Linux (1000+ distributions, due to theological differences) offers important data safety features, and ease of management. There are lots of details about ZFS that you can find in the slides of a talk that we have given several times: http://www.math.utah.edu/~beebe/talks/2017/zfs/zfs.pdf The slides at the end of that file contain pointers to ZFS resources, including recent books. Some of the key ZFS features are: * all disks form a dynamic shared pool from which space can be drawn for datasets, on top of which filesystems can be created; * the pool can exploit data redundancy via various RAID Zn choices to survive loss of individual disks, and optionally, provide hot spares shared across the pool, and available to all datasets; * hardware RAID controllers are unneeded, and discouraged --- a JBOD (just a bunch of disks) array is quite satisfactory * all metadata, and all file data blocks, have checksums that are replicated elsewhere in the pool, and checked on EVERY read and write, allowing automatic silent recovery (via data redundancy) from transient or permanent errors in disk blocks --- ZFS is self healing; * ZFS filesystems can have unlimited numbers of snapshots; * snapshots are extremely fast, typically less than one second, even in multi-terabyte filesystems; * snapshots are readonly, and thus, immune to ransomware attacks; * ZFS send and receive operations allow propagation of copies of filesystems by transferring only data blocks that have changed since the last send operation; * the ZFS copy-on-write policy means that in-use blocks are never changed, and that block updates are guaranteed to be atomic; * quotas can optionally be enabled on datasets, and grown as needed (quota shrink is not yet possible, but is in ZFS development plans). * ZFS optionally supports encryption, data compression, block deduplication, and n-way disk replication; * Unlike traditional fsck, which requires disks to be offline during the checks, ZFS scrub operations can be run (usually by cron jobs, and at lower priority) to go through datasets to verify data integrity and filesystem sanity while normal services continue. ZFS likes to cache metadata, and active data blocks, in memory. Most of our VMs that have other filesystems, like EXT{2,3,4}, FFS, JFS, MFS, ReiserFS, UFS, and XFS, run quite happily with 1GB of DRAM. The ZFS, DragonFly BSD Hammer, and BTRFS ones are happier with 2GB to 4GB of DRAM. Our central fileservers have 256GB to 768GB of DRAM. The major drawback of copy-on-write and snapshots is that once a snapshot has been taken, a filesystem-full condition cannot be ameliorated by removing a few large files. Instead, you have to either increase the dataset quota (our normal practice), or you have to free older snapshots. Our view is that the benefits of snapshots for recovery of earlier file versions far outweigh that one drawback: I myself did such a recovery yesterday when I accidentally clobbered a critical file full of digital signature keys. On Solaris and FreeBSD families, snapshots are visible to users as read-only filesystems, like this (for ftp://ftp.math.utah.edu/pub/texlive and http://www.math.utah.edu/pub/texlive): % df /u/ftp/pub/texlive Filesystem 1K-blocks Used Available Use% Mounted on tank:/export/home/2001 518120448 410762240 107358208 80% /home/2001 % ls /home/2001/.zfs/snapshot AMANDA auto-2019-05-21 auto-2019-05-25 auto-2019-05-29 auto-2019-05-18 auto-2019-05-22 auto-2019-05-26 auto-2019-05-30 auto-2019-05-19 auto-2019-05-23 auto-2019-05-27 auto-2019-05-31 auto-2019-05-20 auto-2019-05-24 auto-2019-05-28 % ls /home/2001/.zfs/snapshot/auto-2019-05-21/ftp/pub/texlive Contents Images Source historic protext tlcritical tldump tlnet tlpretest That is, you first use the df command to find the source of the current mount point, then use ls to examine the contents of .zfs/snapshot under that source, and finally follow your pathname downward to locate a file that you want to recover, or compare with a current copy, or another snapshot copy. On Network Appliance systems with the WAFL filesystem design (see https://en.wikipedia.org/wiki/Write_Anywhere_File_Layout ), snapshots are instead mapped to hidden directories inside each directory, which is more convenient for human users, and is a feature that we would really like to see on ZFS. A nuisance for us is that the current ZFS implementation on CentOS 7 (a subset of the pay-for-service Red Hat Enterprise Linux 7) does not show any files under the .zfs/snapshot/auto-YYYY-MM-DD directories, except on the fileserver itself. When we used Solaris ZFS for 15+ years, our users could themselves recover previous file versions following instructions at http://www.math.utah.edu/faq/files/files.html#FAQ-8 Since our move to a GNU/Linux fileserver, they no longer can; instead, they have to contact systems management to access such files. We sincerely hope that CentOS 8 will resolve that serious deficiency: see http://www.math.utah.edu/pub/texlive-utah/README.html#rhel-8 for comments on the production of that O/S release from the recent major new Red Hat EL8 release. We have a large machine-room UPS, and outside diesel generator, so our physical servers are immune to power outages and power surges, the latter being a common problem in Utah during summer lightning storms. Thus, unplanned fileserver outages should never happen. A second issue for us is that on Solaris and FreeBSD, we have never seen a fileserver crash due to ZFS issues, and on Solaris, our servers have sometimes been up for one to three years before we took them down for software updates. However, with ZFS on CentOS 7, we have seen 13 unexplained reboots in the last year. Each has happened late at night, or in the early morning, while backups to our tape robot, and ZFS send/receive operations to a remote datacenter, are in progress. The crash times suggest to us that heavy ZFS activity is exposing a kernel or Linux ZFS bug. We hope that CentOS 8 will resolve that issue. We have ZFS on about 70 physical and virtual machines, and GNU/Linux BTRFS on about 30 systems. With ZFS, freeing a snapshot moves its blocks to the free list within seconds. With BTRFS, freeing snapshots often takes tens of minutes, and sometimes, hours, before space recovery is complete. That can be aggravating when it stops your work on that system. By contrast, snapshots on both BTRFS and ZFS are fast. However, they appear to be far smaller on ZFS than on BTRFS. We have VMs and physical machines with ZFS that have 300 to 1000 daily snapshots with little noticeable reduction in free space, whereas those with BTRFS seem to lose about a gigabyte a day. My home TrueOS system has sufficient space for about 25 years of ZFS dailies. Consequently, I run nightly reports of free space on all of our systems, and manually intervene on the BTRFS ones when space hits a critical level (I try to keep 10GB free). On both ZFS and BTRFS, packages are available to trim old snapshots, and we run the ZFS trimmer via cron jobs on our main fileservers. In the GNU/Linux world, however, only openSUSE comes by default with a cron-enabled BTRFS snapshot trimmer, so intervention is unnecessary on that O/S flavor. I have never installed snapshot trimmer packages on any of our other VMs, because it just means more management work to deal with variants in trimmer packages, configuration files, and cron jobs. Teams of ZFS developers from FreeBSD and GNU/Linux are working on merging divergent features back into a common OpenZFS code base that all O/Ses that support ZFS can use; that merger is expected to happen within the next few months. ZFS has been ported by third parties to Apple macOS and Microsoft Windows, so it has the potential of becoming a universal filesystem available on all common desktop environments. Then we could use ZFS send/receive instead of .iso, .dmg, and .img files to copy entire filesystems between different O/Ses. ------------------------------------------------------------------------------- - Nelson H. F. Beebe Tel: +1 801 581 5254 - - University of Utah FAX: +1 801 581 4148 - - Department of Mathematics, 110 LCB Internet e-mail: beebe(a)math.utah.edu - - 155 S 1400 E RM 233 beebe(a)acm.org beebe(a)computer.org - - Salt Lake City, UT 84112-0090, USA URL: http://www.math.utah.edu/~beebe/ - -------------------------------------------------------------------------------

On 5/31/19 10:06 AM, Michael Kjörling wrote: (Baring any local privilege escalation....) I think that ZFS would protect (snapshots) against ransomware running as an unprivileged user that can't run zfs / zpool commands. I have found the zfs and zpool command sufficiently easy to allow limited access via appropriate sudoers entries. -- Grant. . . . unix || die

On Fri, May 31, 2019 at 2:50 AM Arthur Krewat <krewat(a)kilonet.net> wrote: At home I use ZFS (on Linux) exclusively for all data I care about (and also for data I don't care about). I have a bunch of pools ranging from 5TB to 45TB with RAIDZ2 (overall about 50 drives), in various hardware setups (SATA, SAS, some even via iSCSI). Performance is not what I'm used to on Solaris, but in this case, convenience wins over speed. I never lost any data, even though with that amount of disks, there's always a broken disk somewhere. The on-disk format is compatible with FreeBSD and Solaris (I have successfully moved disks between OSes), so you're not "locked in".