On Sunday 18th October the eChook data server was slow or unreachable for four and half hours – from 9:50am to 2:20pm. All live data services were down as a result – I’m incredibly sorry for the failure, right in the middle of the international final!
The first issue
I realised it had failed after a couple of teams reached out to me (Thanks!), having completely missed the automated email that was sitting in my inbox…
First thing I noticed in the logs was an unusually high number of junk requests from spam bots. A few a minute is totally normal, but it was peaking around 100 per second. This shouldn’t be enough to disrupt a well configured server… but well… oops. Due to the order in which I was processing the incoming requests, the spam protection wasn’t dropping the request until a reasonable amount of CPU time had already been used on it. This was compounded by the server being a very resource limited machine as the actual eChook data handling isn’t heavy work, and I’m keeping costs down to provide the free service. The combination of more processing than ideal and a slow CPU resulted in timeout responses to every incoming request.
So the first fix was to move the spam protection to the first thing each request hits once through to the server – where it should have been originally. This fix was set live at about 1pm, but didn’t fix it quite as planned…
The second issue
First fix in place, the server was restarted – only for data.echook.uk to still timeout every request. Going back into the logs, the spam requests were still there, but being dealt with properly now and using a negligible amount of resources. The new issue appeared to be a flood of genuine login requests from the Omni Android app, upwards of 300 per minute, coming from about 25 active cars.
Each login request requires a hashed password check using bcrypt. This is a relatively heavy operation by design, and I’d never tested a scenario where 25 cars all try to connect in the same second. It was now the password hash checks that were overloading the server, with the result that the Omni app was timing out and sending a new request before the server had finished the first one, so each response was ignored, and the login requests just kept backing up.
The fix for this was twofold but simple – login rate limits in the server code, and a rate limit applied at the reverse proxy handing the requests. With both of those in place, the server finally became responsive again around 2:20pm, just in time for the second race of the day.
There is actually a fix to limit the login retry attempts in the the current development version of the Omni app that Keith put in place a month or so ago – but it’s yet to be pushed to the play store. Neither of us had seen it as an urgent issue.
Was any data lost?
No existing data on the server was affected, but the first race data was never received by the eChook server, and so never logged. Thankfully the Omni app logs it all locally so teams can retrieve it from the phone by following the instructions here: https://docs.echook.uk/using-the-app/logging-data
If teams would like to be able to retrospectively upload this data to the server let me know and I can add a feature to eChook Live to facilitate it.
Stopping it happening again
Both of the issues occurred due to areas of inefficient code. Those have now been fixed, and I will be looking for further inefficiencies that can be fixed off the back of this. The root cause behind both however, was the low power (and therefore cheap) server that this is running on.
Efficiency improvements will help for a while, but ultimately as the user base grows I suspect I’ll have to start spending a little more on hosting to ensure a better quality of service. There will always be a very useable free eChook Live service but I’m going to take this opportunity to float the idea of a paid tier in the future, likely in the region of £20 per year, giving access to more advanced features I’m working on at the moment that would cover some of the service running costs.
Again – I’m incredibly sorry for the outage on the most important race of the year, and wish you all a productive off-season of Greenpower tinkering!