Engine room report feburary 2022

Dear friends of wikiloops,

thanks for stopping by and checking out this post, in which I'll attempt to cover what has been going on in the wikiloops engine room these days.

I guess most of you will have noticed that we had some server outage on Thursday 24th of February, lasting into the following Friday.

Maybe some of you saw the "currently unavailable" message during that period, or later the rather crude "we are in maintenance mode"-page which I quickly came up with, to at least signal out that someone was trying to fix the problems.

The armed conflict between Russia and Ukraine started the very same day, and I don't know about your feelings, but I couldn't help wondering if there might be some connection between the two events.
To tell you the truth - I'm not sure I have a straight answer to this question, but I'll share what I have, and what my current thoughts are.

When I learned that there was a problem on wikiloops (a classic Kaiser-family situation: my mom calls and tells my wife to tell me wikiloops is down), there were a couple of things I suspected as probable causes - hardware failure, some data-center-wide connection failure, a bot attack of some kind... the kind of things that we have seen in the past.

So, digging into the matter, I do some routine checks to see which parts are still working, and which ones are not.
Turns out:
Both server and database are operating quite smoothly,
but anything trying to connect via the wikiloops domain name had only a 3% chance of getting a proper response - anyone else sees the dreadful 503-Error screen.
What is this? Me wonders - clearly no hardware crash there, that rather looked like some sort of bot attack thing to me.

Now, without going into all the tech-details, one thing about bots is that all you'll ever know about them is their traces in the servers log-files, which basically reveal little more than "someone tried to access page X at this time, and received/didn't receive a response".
The potential responses are labeled by types, you'll know some of those, like the well known code "404" for "page not found".
The crucial problem on day one of the incident was: I couldn't access these log files straight away to see what was going on, but had to wait half a day to get the daily log extract supplied from the hosting company, so, there I sat with little insight what could be happening.

So I went troubleshooting, and since there was not much to break in that situation, I followed the very basic idea of "if I block some countries, and if there is an attack coming from a certain region, I might get to the point where wikiloops comes back up for the remaining unblocked regions".
And yes, I frankly admit that there are some countries that are a bit more notorious for rude bot attacks than others, so I blocked these first.
Long story shortened: I went on until anything but Germany was blocked, but to my surprise none of that made a difference. I even blocked Germany for a moment and saw the block coming up on my own screen, i played around, nothing worked.
So, back to nagging the support folks and waiting for the logs...

I'll try to cut this story, so you hopefully have a moment left for my follow-up thoughts:
After quite some mail exchange with our hosting company, it turns out what we witnessed was a mix of a really heavy bot attack, plus our hosting companies efforts to fire-fight the issue.
What the support guys did was basically throttle general access to wikiloops, as zillions of Chinese bots tried to call up wikiloops pages.

After I could make it clear to them that their defensive move was just killing our platform they slowly lifted their throttling on the main .com-domain, letting the bot requests be blocked by the regional block I had set up one level lower - that's how you guys got a working wikiloops back some time on Friday, if you weren't trying to access the German or french versions of wikiloops, and if you happened to live somewhere outside of Asia.

The actual attack wore off during the course of Saturday, and as the logs show, traffic piked at around 12 times of what we usually have on wikiloops.
And we do have a lot of bots crawling wikiloops on any day, I'd estimate that bots make up 90% of wikiloops traffic on a normal day, so to see that figure multiply that much is quite scary, and certainly beyond what our current server setup is built for.

I did announce an optimistic "we are back in full service" some time on Saturday,
and luckily wikiloops has been operating quite smoothly since.
So, is that a happy end?
Not quiet, now that I have had some time to think this thru (watch out, turning point of this news post approaching...)

I could have put this in the "shizzle happens" section, and we might as well say let's sit and wait if something like this happens again.

What struck me was this feeling of total dependency of those server support folks, who at will shut down wikiloops "in protection of their other customers".

I'm aware that the internet is a rough place, and digital attacks of all kinds happen on any day,
but the moment I sit there and block a whole continent + some select countries from accessing wikiloops,
I can't help but notice that I am just leaving the path of what wikiloops was intended to be:
An international meeting place for musicians, where one could cross borders and make friends just by making music.

As I see things today, our current hosting company is probably not willing to let us re-open for any country (or they will come back and ask me to raise fences again as soon as something goes wrong), so my choice comes down to:
Let go of the "global jam" ideal, or get a different server setup that can provide the "global access" plus the needed defense systems to stay online when under attack.

Personal note:
I've done this here for over ten years now, and it means something to me, you know?
Some time this weekend I received a birthday notification of one of wikiloops Russian members, and knowing Russia is on the block list since this weekend really made me feel bad.
Alex, if you ever read this - a late happy birthday to you!
I'll say the following as a flaming personal statement:
Folks, we are here to look a little beyond our local horizons,
and wikiloops is there to let us experience something different to the stereotypes we may have adopted. Let's never forget how many funky bassists live in any country before thinking some nation was "totally evil" - end of my personal peace-pledge for the day, sorry, had to say it.

Back to the engine room news:
I have spent the weekend "evacuating" our current server and moving wikiloops into the almighty amazon cloud, with the aim to upgrade our infrastructure to a point where wikiloops can continue its global mission,
independent of some German hosting companies ideas of how to limit traffic.

I'll be really honest and add that the location of our current data center on the border between Germany and Poland doesn't really feel like a safe place to be in the current political situation,
so when I called my server moves "evacuation" earlier, then that's the reason why.
One benefit of amazons cloud is regional independence, I might end up replicating servers to anywhere if need be.

Now, the moving process for a project the size of wikiloops is nothing one could do in a day,
but I am happy to say that I managed to set up a replicated instance of wikiloops on some Irish aws-server over the weekend,
so even if things should go down the very bad way some time in the future, I'm positive I'll be able to relaunch quite quickly.

You may imagine it was quite a relief off my shoulders when importing the data tables holding more than 1.000.000 comments and similar treasures to a safe environment...
and I guess I'll continue getting that server ready so we are set to hop over.

I'll make sure you are informed before that happens,
because that change-over might take a day or two.

What I would like to leave you with before I'll end are two impressions:

First, please note that someone is taking care of this project, and that the will to keep it in its best shape is strong, even if the technological hurdles to achieve that have not become less steep in the past years.
I feel wikiloops is worth the effort, and if I say wikiloops, then to me that is our international community of people willing to collaborate. This is worth saving, so that's what I'll try to do.
Second, I'd like you to know that I've had moments during the past weekend where I felt like "Richard, you need to get all the help you can possibly get to pull this off!", and we actually had a small family council first to make sure the time and cash investment needed to move on with wikiloops can be covered.
My decision to upgrade the infrastructure will come with a hefty price tag attached, so if anyone feels like supporting the "global jam"-missions future, I'd really appreciate if you'd send some support via the wikiloops donations page.

All that being said,
let me thank you for reading thru and for being part of this community

stay safe and healthy out there & keep up the positive spirit in these odd times we are witnessing

all the best from Germany, a thousand miles west of Kiev
Richard

wikiloops founder