This week, we want to spotlight a very helpful thread created by Discourse user robgriff444 in order to help our new users familiarize themselves with ZFS and use it with TrueOS. The TrueOS devs want to thank robgriff444 for the outstanding contribution to our community!
Since installing TrueOS (after being in the Linux world for many years, always using testing/unstable repos), I have struggled with ZFS and Snapshots for the last few days – mostly due to conventions, syntax and assumptions, and I have found that this is because both ZFS and Snapshots are high-level enterprise technology with multiple amazing abilities for datacentres, developers etc., so they don’t work in the same way that a typical desktop user would reasonably expect.
From my perspective, as much as all this is interesting, to begin with, all I wanted was a partition on my drive, and the ability to rollback if something went wrong. Learning can come later. And i could only start with what I already knew, and rolling back was something I had only seen on Windows
Everywhere I looked, whatever I read, there were many assumptions made about what I, the reader, already knew. As if I had missed a training course that everyone else had been to. It seems to me that this is typical of enterprise technology – you get training when you use it and you probably already know most of the conventions because of the nature of your job.
I’m writing this to explain some basics in order that new desktop users can 1) grasp what ZFS/Snapshots really can do, 2) know how they go about using them for their basic needs and 3) know any tricky things that don’t get illuminated by trial and error.
ZFS, Snapshots, and TrueOS
ZFS goes beyond the premise of partitions and is designed for enterprise use – it uses something it calls pools and these can span multiple drives. It can do lots of cool things that do away with partition restriction and the problems of managing them e.g. resizing the mounted root partition. It is way more than this but you can find that out later. And it’s fine on a humble desktop too
So, TrueOS installs using ZFS and creates a pool called tank. Look in Disk Manager and you’ll see a tiny boot partition, a swap partition at the end, and the rest is a ZFS partition with a state called Pool:tank. In the ZFS Pools tab you’ll see that your pool is the parent to the drive partition – this is the premise of having a pool span multiple drives. In the ZFS Filesystems tab you’ll see (after a snapshot has been taken) a split of your filesystem in a way that is apt for ZFS in the enterprise world, not necessarily what you may expect.
Snapshots use a system of writing new data next to the old and keeping pointers to both states so if you have a 10MB file and increase it to 15MB, a snapshot would store a pointer to the 1st 10MB and a pointer to the whole 15MB so no extra space is taken up. If you delete data, the space won’t go down, and if you modify data, you will see an increase.
Snapshots are for the whole filesystem, not parts of it. They can be taken as a whole and rolled back as a whole and rolled back file by file by hand (e.g. in Insight – look at the slider below the menu bar – you can go back in time and copy files from the snapshot but not to it), or snapshots can be taken ‘recursively’ (bit of a misnomer here IMO – unlike the ls command, non-recursive doesn’t mean the snapshot only does / ). Here, recursively means that the filesystem snapshot will be split into multiple snapshots, the sum of which makes up the whole, that can be more handily rolled back in sensible lumps. TrueOS Control Panel (Sysadm) Lifepreserver uses recursive snapshots so you’ll see these multiple snapshots in it’s drop-down list.
Now (once you’ve created a snapshot) go back to Disk Manager and you’ll more easily see the layout (in the ZFS Filesystems tab). Here, you can see the 3 types of listings under your pool – e.g.
1. tank/ROOT, (disk/file icon), it says it has data but right-click shows can’t be rolled back as it’s a parent.
2. tank/ROOT@snapname, (camera icon) these have no data so if you rolled back you wouldn’t change anything.
3. tank/ROOT/”CURRENT” (or something similar), (camera icon), AND they have data – these can be rolled back.
Note that if you’ve performed an upgrade, you’ll have 2 directories under ROOT named something like Initial and Current. Initial is a cloned snapshot of the filesystem just before the upgrade. Cloning is making a complete filesystem from snapshot data so it’s a complete copy but using the snapshot method. These are known as Boot Environments and can be chosen in the BSD boot loader in the case of problems after an upgrade. BE’s are managed by typing beadm – this is a shell script that manages snapshots for this particular purpose.
Disk Manager is the easiest way to become familiarised but you can use Sysadm (which also works on remote systems) and the command line.
So , to recap, once a snapshot is taken with Sysadm Lifepreserver, you can look at Disk Manager and see what is a snapshot and what is a parent, and you can see multiple Boot Environments (if you’ve performed an upgrade). And now you can rollback with a right-click.
On the command-line, note that zfs rollback won’t tell you if it’s done anything, and -r (recursive) appended to zfs snapshot is NOT the same as -r appended to zfs rollback . Also note that ROOT/”current” contains most of the filesystem (e.g. /usr is listed separately but has no data – it is a parent for /usr/home so it doesn’t contain directories such as /usr/lib – these are in ROOT – this is by design and is logical to enterprise systems and even though it may not be logical to a desktop user, it is not obfuscative). If you want to see what directories are included in a snapshot, go to the relevant directory e.g. cd / and you can cd into .zfs/snapshot (hidden even to ls -a) and type ls -Rp | grep /. Every snapshot has a .zfs/snapshot directory. Note that /usr doesn’t have .zfs/snapshot.
So if you have a filesystem problem (bad install etc), rollback ROOT/”current”. The other datasets can be rolled back as you see fit e.g. /usr/home/username:
zfs rollback tank/[dataset name with data in and not a parent]@snapshotname
And once you are familiar with the layout of the snapshots as shown in disk manager you can more easily use sysadm life preserver (which doesn’t have icons) to rollback. And remember, sysadm has been designed for enterprise systems and remote control so it doesn’t have newbie nice touches to it (yet?).
Also, if you have recursive snapshots, in order to rollback the whole system if you ever need to, you have to individually rollback each snapshot, there is no way to use a top level snapshot to automatically trigger the rest, again this is by design and if you want to do this repeatedly, you can use shell scripts.
And finally, note that many snapshot actions are instant, there is no hard disk chugging (because it’s not reading/writing, it’s mounting/unmounting) which can be disconcerting because it’s not working the way you may expect so just because a rollback command doesn’t take 25s doesn’t mean it didn’t happen, this is normal. If you use sysadm and think it was too quick, go look at the data before assuming it did nothing!
So… How to:
- Create a Snapshot:
Start Menu > Control Panel > Life Preserver, click Create.
- Create Regular Snapshots:
Start Menu > Control Panel > Life Preserver, select Schedules tab, click the small camera icon on bottom left (hovering over it says “Add snapshot to schedule”). To change it, click the small cog icon to the right.
I initially set mine up to do 7 snapshots at 2100 every day.
- Recover root:
Start Menu > Disk Manager, select ZFS Filesystems tab, right-click on the /tank/ROOT/[your current boot env name] @[snapshotname] and choose Rollback to this snapshot.
In my case this is /tank/ROOT/12.0-CURRENT-up-20170316_125139@auto-2017-03-18-21-00-00
(NB. Disk manager lists the oldest first so go down to find the latest.)
- Recover /usr/home/yourhome:
As above but choose the latest /usr/home/yourhome.
In my case this is /tank/usr/home/robert@auto-2017-03-18-21-00-00
- Recover individual files:
Start Menu > Insight, go to the relevant directory, then click Current (just under the back, up and home buttons) and choose a snapshot, or use the slider next to it, or use the arrows on the far-right of the slider.
When you can see the file you want, open a second tab with the green + icon to the right of the directory field and make sure the new tab is in Current state (I find the left, right buttons under the green icon easiest).
Then switch back to the 1st tab with the older snapshot state and simply drag and drop the file to the right tab. IT WILL NOT TELL YOU IT’S DONE IT! Click on the right tab, then click left and right under the green + icon to refresh the screen, and the file will be there.
- Recover everything:
Disk Manager, right-click and rollback every snapshot form the latest snapshot. This is where Control Panel is simpler – in Life Preserver, in the Snapshots tab, select the latest snapshot (lowest is newest) and click the drop-down box under Revert to Selected Snapshot. You can easily choose one, then click Revert, then choose the next and click Revert and on and on.
Or write a shell script starting with zfs rollback [poolname]/ROOT/[Boot env name]@[Snapshot name] and create a new line with each different snapshot e.g. zfs rollback [poolname]/usr/home/[yourhome]@[Snapshot name].
- Change the Boot Environment (e.g. after a failed update):
In the BSD boot loader, choose Select Boot Environment. Or, on the command line, first type beadm list, then type beadm activate [BE name] (from the previous list), and reboot.
- Remove a Boot Environment:
Type beadm list, and to remove the older one, type beadm destroy [BE name] (from the previous list). BE VERY CAREFUL – IF YOU ARE DESTROYING THE CLONE, WHEN IT IS COMPLETE, IT WILL ASK IF YOU ALSO WANT TO DESTROY THE PARENT SO MAKE SURE YOU PRESS NO. For anything else with beadm, just type it on it’s own and it will list operations, and it has a helpful man page too.
Be part of the discussion! The TrueOS community is full of friendly and knowledgeable people willing to help troubleshoot issues or discuss the latest goings-on in the Open Source world.