Well I’ve joined the “accidentally trashing your system with rm -rf” club! Luckily I didn’t delete my home directory with all the things I care about, but I did delete /boot and /usr, and maybe /var (long story, boils down to me trying to delete non-system directories named those but reflexively adding the slash in front when I should not have). I have backups of those as well, so what are my prospects of recovering from this by just copying them back in using a live USB? Only issue is they’re stored in my server as belonging to the server user (I assume everything in those directories should belong to root and I can just use chown?) But I also don’t know if they retain the same permissions when backed up.
Has anyone had any luck recovering a system in this way? I’m hoping not to have to reinstall everything because I had gotten pretty cozy with the current installation.
UPDATE: I finally had the time to sit down and try it, and, I was at least hoping to document some glitchy or unstable behaviour but it just didn’t work at all. No matter what I tried I couldn’t even get the UEFI to recognize the old system as bootable, so I cut my losses and just reinstalled. Gonna make sure I have btrfs snapshotting enabled this time, which I’m realizing I probably should have done in the first place.
Is anything keeping you from just reinstalling the system and mounting your home into it again (maybe the majority of your customisations live in /home too)? I feel that is a lot less of a hassle than copying files around.
In principle you should be able to restore your system by just copying all of the relevant files from the backup to their correct partitions - it can’t really get any worse if it doesn’t work.
For the future: A backup is only any good if you know how to restore it and tested that that actually works.
Regarding the permissions: If you do a
cp fileA.txt fileB.txt
fileB.txt
will normally be owned by the creating user. So asudo cp ...
will create the files as root.I would personally use
rsync
with a few additional options, archive among them. This way the fs is restored exactly as it was. But that doesn’t make a whole lot of sense if the files weren’t copied that way too.Just reinstall :)
Copying back the files to the right partition/directory works, but if you didn’t backup the owner and permissions for each file it’s gonna be a pain to restore those.
After reinstalling, you can compare your new system with your backup to see what changes/configs you had made
Give it a try! System is broken anyway. Also fix your backup to include file metadata, maybe disk images?
Heck I would try using testdisk to undelete the files onto another filesystem then copy them back if the permissions look okay.
you dont say the o/s but if the pkg manager works, or you can add a statically compiled version, you could force reinstall all pkgs
I did a similar fucky-wucky before and honestly i just cut my losses and backed up the user data before reinstalling the OS from scratch. Took a few days of tinkering to get my system back to where it was but there’s no telling what kind of system you’ll be left with when you merge a known good image with a broken system.
Your chances are pretty good if you copy them back - ultimately, that’s what the restoration function of backup software does.
As for ownership of the directories and files, that’s a bit trickier and might involve some trial and error. root:root is a safe bet for most of it, but there is a lot of stuff in /var that is owned by system accounts.
What distro are you running? That’ll help figure it out.
I’m running Fedora 39 KDE. I think I’m going to see what the file metadata of my other Fedora systems look like and try to replicate that. Worst case I just reinstall. At this point I’m a little curious how the system will react.
That’s entirely valid. Good luck.
Thank you! Regardless of the outcome I will update the main post with my findings in hopes giving anyone else in the same position some more info.
Yes but it’s hard work.
I did it from the other side of the planet. I accidentally ran an
rm -rf ...
command on a running system. Luckily I had an identical system running that I could use to copy over the files, devices, etc.Learning about inodes and
/proc/xxx/fd
works, I was able to recover enough files to then copy over the rest from the other system.Doing it over SSH from the other side of the world was a tough 14 hours.
Depends on specific machine setup and how good the backup is.
Backup requirements for /usr there are sticky bits set on some binaries. That needs to be preserved. In all cases soft links likely need to be preserved for things to work correctly on future package installs. Hard links can be problematic, but if you have a large enough drive or not that many it wont matter. Running package verification can be help after restore to make sure everything looks right. If running a Linux system with SELinux in enforcing mode (RHEL on many derivatives), then the security context will also need to be preserved BUT running a relabel will probably work if the security context was not included in backups. Sometimes running the relabel process wont work if there are files that needs a specific security context but are not listed in the security context database. Can’t provide more details because most of my experience with that is on systems we just replace (LSPP custom labeling resulted in systems that if you booted into permissive would then be unbootable, so they were just reinstalled once any debugging was done).
For /boot things can get tricky depending on the distribution, what boot manager is used, and /boot was a separate partition or not. Basically the boot manager (probably grub) needs to know how to find the files in boot so it can load the kernel. In most cases if you restore /boot and rerun the tools to update the boot manger everything will be fine. BUT some distributions, hardware setups, or dual boot configurations are more complicated, so extra work might be needed.
You didn’t mention /dev, which is all special files. These don’t need to be restored, just make sure the right processes recreate them. There are tools to do this, hopefully the packages are installed. Or boot from a rescue disk and fix it. Look up instruction for your specific distro.
What system are we talking about here? The
--no-preserve-root
option is part of every modern release ofrm
.Edit: Never mind, I didn’t read the post properly.
Normally its better practice to have the server configuration stored in a declarative way like ansible or similar and only store the userdata in the backup.
So you can fast and easy reinstall your server including all of its config files and then clone the usage data like dbs or files into the new machine. This is more reliable and also faster than just do a full dump of the system.
I did this way back in the day on my Mandrake installation with a 1.44" floppy. Only tricky part was that I had to run cp from the floppy instead of from normal $PATH as I’d wiped out /bin.
That can be done, but as others mentioned, if you don’t have permissions/other attributes for the files it’s going to be a real PITA to get everything working. If I had to do that I’d just copy over the files, chown everything to root and then use package manager to reinstall everything, but even that will most likely need manual fixes and figuring out what to change and to what value will take quite a bit of time and complexity of it depends heavily on what you had running on the host, specially things under /var.
No way, reinstall.
If even file owner is not preserved (it is not always root, espetially in /var), you likely lost files’ extanded attributes an, maybe, also permissions. Without them your system won’t work normally.
Then, contents of these directories must be consistent with other ones. E. g. /var contains a package manager data about packages you installed. If you installed/removed anything after creating a backup, information about this will be lost.
If you created the backup while system was working, some files (espetially under /var, again) could be changed during that process, and this also makes such backup unusable. Every sysadmin knows that to create a database backup by copying files, dbms must be stopped.
In future, think about restoration before planning a backup and test if this possible immediately after it is done.
Only issue is they’re stored in my server as belonging to the server user (I assume everything in those directories should belong to root and I can just use chown?) But I also don’t know if they retain the same permissions when backed up.
Not everything will be owned by root, and some of the binaries will be setuid or setgid, some might even have extended attributes (e.g. ping will usually have a security.capability attribute).
/var
will also have a lot of different owners.Use dd next. Go for a trash your install speedrun! Hey, it’s the quickest way to learn 🤷♂️