Difference between revisions of "2024/05/03"

Revision as of 21:38, 5 May 2024

The Latest Tedious Tech Saga: Background and Updates

The Incident

About a month ago^[1], I had made enough progress on VbzCart that I was ready to post a partially-working demo. I wanted to use Statler (our home server) for it, but I've been writing in the latest version of PHP (which, until this week or so, was v8.2) and Statler was still running Ubuntu 22.04 LTS which only has v8.1 or so, and some of the stuff I had written wouldn't work -- so I thought okay, I'll force an upgrade to the latest Ubuntu (which, until last week or so, was 23.10).

That went fine until sometime late in the process when the upgrade encountered some kind of disk error, and the filesystem switched to read-only. The upgrade app said it had completed "with errors", but it couldn't actually be rebooted after that.

Time for another episode of "that's terrific! / that's terrible!":

(+) I did have regular backups set up, but...
(-) they did not include a proper SQL dump of the database.
(+) The database files themselves were (are) still readable -- no data lost...
(-) ...but I couldn't, of course, boot the disk to run the MySQL server to load those files.
(+) I have, on several prior occasions, succeeded in recovering MySQL databases just by copying the wanted files from the MySQL data folder on the dead server (or backup thereof) to the MySQL data folder on a working/live server.
(-) My initial attempts to do this did not work.

The Recovery

So I set up a new Hetzlet (VPS on Hetzner^[2]) -- like a "droplet" on DigitalOcean) specifically for this recovery, since I didn't immediately have another SSD drive handy with which to get Statler back up and running. ...and since Statler had "successfully" upgraded to 23.10, I thought I'd better upgrade the Hetzlet as well, for the best possible DB engine compatibility.

I forget exactly how long I futzed around with that, but I did finally manage to get the data to load so I could export it. I found out there's at least one alternative way of doing it, but the direct method turned out to be the least painless option. Next task on the list, then, was to make the data accessible.

There was really only one database to which I really needed access, and it's a MediaWiki instance. (There's also a Nextcloud instance upon which we used to rely heavily, but I had to disable it a few months ago because it was causing weird network saturation issues. We switched to using SyncThing for accomplishing the same purposes... and I have a lot to say about that, too... but later.)

The next item of business, then, was to set up that one database on the new Hetzlet and give it a domain so I could access the wiki it drives. This involves configuring Apache for a virtual domain -- for which I have been using VirtualMin for over a decade now because it reduces the brainpower and time needed for configuring many server-things (including Apache). It was already starting to be less useful, however: CertBot hasn't been working properly for some years now, PHP configuration sometimes wasn't right, and there were a lot of details it never handled very well. I'd found myself writing all kinds of config include-files^[3] to fix not-quite-rightness in how VirtualMin sets up PHP.

It turns out, however, that VirtualMin can't be installed on 23.10^[4], period. It just flat-out refuses. I also couldn't get my standard includes to work^[5]. In the spirit of making sure I never finish anything by always introducing a side-quest halfway through, I therefore decided to rewrite those include-files as a set of templates for generating normal flat .conf files from a PHP script.

About 3 weeks later, I actually got that working, and then proceeded to install the latest version of MediaWiki (which then required upgrading the database) and... actually got it back online again about a week ago.

The Cleanup

The next thing that needed doing was to make a complete backup of Statler's dying SSD drive. This, of course, ran into at least one unreadable file.

Now... maybe I should have been using rsync for this, but I don't trust rsync to be doing what I want it to do -- in this case, how will it handle files that it can't read? Will it log them somewhere, will it hang forever while trying to read them, will it just give up and stop copying? It also has its own awkward-ass^[6] syntax that it uses for file specs, which adds additional brain-overhead whenever I try to use it.

Consequently, I have (as one does) rolled my own Paranoid File-Copy/Move program in PHP which may not be perfect but at least it does what I want it to do (and I can fix it whenever it doesn't).

As half-expected, however, it also got hung up on the unreadable sectors of the drive... so I spent maybe another week reworking it (Obligatory Side-Quest: refactor the Futilities library to use one-class-per-file, structured with folders and namespaces, to make things easier to find) and then reorganizing options and, oh yeah, doing something about the fact that it wasn't logging anything... and as of last night, we are now to the point where ...[drumroll, please]... it still gets hung up on unreadable files, but now I can fix it so it logs them and moves on! (Thank you thank you, I'll be here all week. Again.)

Continued journaling: 2024/05/04

Footnotes

↑ Looks like it was April 13, from commit-timestamps on the repository origin. Really? Seems longer ago... but I guess it's good that time isn't racing past quite as fast as I thought it was.
↑ I've been planning to move from Digital Ocean to Hetzner because the latter came recommended and is also significantly cheaper than DigOc. Recently, there was some concern about Hetzner abruptly cancelling service for another fedi instance - woem.men - with some kind of moral panic excuse about... well, here's what I wrote about it in Discord on March 24:

The story as I understand it is that they're hosted on Hetz, and Hetz had warned them that their account would be suspended if [something] -- and at first they thought the [something] was a specific image, which they removed, but Hetz never acknowledged whether that was it, never acknowledged that they had taken corrective action, and is still threatening to suspend the account. (Much of this is between the lines and not stated explicitly -- which makes me discount it a great deal, because vagueness is often used as a tool for plausible deniability when trying to manipulate sentiment.)

So for now, we're still moving things (including TootCat) there -- but I'm putting a higher priority on automated migration processes, so we can get out again quickly if needed.
↑ Shirley there's a way to link to a specific commit in GitLab, but I can't find it. For now, though, until I remove from the "main" branch, they're here.
↑ It seems possible that if I had installed it on the Hetzlet as initially provisioned with 22.04 and then upgraded, it might have continued to function. ...or possibly not.
↑ Firefox has been telling me for several days that it needs an update, and that it would force a restart about 12 days from now. I was all, like, "Firefox, you know you're not going to be able to wait that long. Even when I restart you as soon as you ask, it seems like you need another update the next day... or even right after I've restarted." ...and then, literally in the exact middle of typing this word, Firefox stopped responding to keystrokes. I had to save my work (at least it was just the keyboard, and not the mouse), save my sessions, close it... and go to bed, because that was the last few minutes I had. I'd have moved everything to Vivaldi by now if I could figure out how to migrate my bookmarks and passwords.
↑ It's possible I've been watching too much A Black Lady Sketch Show. I've had to stop myself several times now from using this phraseology, and finally decided what the blippity.

[date1-1] Looks like it was April 13, from commit-timestamps on the repository origin. Really? Seems longer ago... but I guess it's good that time isn't racing past quite as fast as I thought it was.

[hetz-2] I've been planning to move from Digital Ocean to Hetzner because the latter came recommended and is also significantly cheaper than DigOc. Recently, there was some concern about Hetzner abruptly cancelling service for another fedi instance - woem.men - with some kind of moral panic excuse about... well, here's what I wrote about it in Discord on March 24:

The story as I understand it is that they're hosted on Hetz, and Hetz had warned them that their account would be suspended if [something] -- and at first they thought the [something] was a specific image, which they removed, but Hetz never acknowledged whether that was it, never acknowledged that they had taken corrective action, and is still threatening to suspend the account. (Much of this is between the lines and not stated explicitly -- which makes me discount it a great deal, because vagueness is often used as a tool for plausible deniability when trying to manipulate sentiment.)

So for now, we're still moving things (including TootCat) there -- but I'm putting a higher priority on automated migration processes, so we can get out again quickly if needed.

[conf-inc-3] Shirley there's a way to link to a specific commit in GitLab, but I can't find it. For now, though, until I remove from the "main" branch, they're here.

[vmin-4] It seems possible that if I had installed it on the Hetzlet as initially provisioned with 22.04 and then upgraded, it might have continued to function. ...or possibly not.

[ffox-5] Firefox has been telling me for several days that it needs an update, and that it would force a restart about 12 days from now. I was all, like, "Firefox, you know you're not going to be able to wait that long. Even when I restart you as soon as you ask, it seems like you need another update the next day... or even right after I've restarted." ...and then, literally in the exact middle of typing this word, Firefox stopped responding to keystrokes. I had to save my work (at least it was just the keyboard, and not the mouse), save my sessions, close it... and go to bed, because that was the last few minutes I had. I'd have moved everything to Vivaldi by now if I could figure out how to migrate my bookmarks and passwords.

[blss-6] It's possible I've been watching too much A Black Lady Sketch Show. I've had to stop myself several times now from using this phraseology, and finally decided what the blippity.

[1]

[2]

[3]

[4]

[5]

[6]