Random overload!

What is happening to my server?!

I run a bunch of servers, most of them tiny and used for redundancy, but one that is central to my business. Lately, I’ve been noticing that the main server is consistently being bombarded by high CPU usage and multiple crashes, which, when it’s the primary email handler for my clients to access their incoming email, is a big problem.

Full disclosure with this: I’m not going to give away much in the way of details (log messages and such) as I don’t want to risk any privacy breaches with my clients data.

Where for art thou logs?

Seeing as how I’ve never run into that problem before on CentOS 7, or CentOS 6 in the years I’ve been using them both, I started investigating. Problem is, I couldn’t find any reason for WHY the system kept crashing. In my ignorance, I turned to the guys over at the Sysadministivia to get some advice. (Those guys are great by the way).

Brent got back to me really quickly and told me that CentOS 7 doesn’t store the journal persistently by default! How crazy is that?! After turning on persistent journaling (they even told me how to do that) and remote logging (rsyslog is still amazing), so I could at least go through the logs next time the problem happened.

An excerpt from Brent’s response to me:

Before we go into anything else, I should note that the default CentOS 7 behaviour for journald is "auto" storage, meaning: log to volatile memory (RAM) if the directory /var/log/journal does NOT exist (and it
doesn't, in default cases). If you want persistent logging (and it sounds like you do), you can either:

- uncomment "#Storage=auto" in /etc/systemd/journald.conf and change to
"Storage=persistent" (in which case it will force-create the directory if it doesn't exist), OR

- simply just mkdir -p /var/log/journal

The Problem is found!

When it did happen again, about a week later, I was able to examine all of my logs, and lo and behold, found so many references to PHP FCGI processes crashing from a lack of resources (DDoS), always from the same IP range (who knew Russia was so interested in my Small Business?) that I was able to just mass drop of all packets and requests coming from them. If I didn’t keep my systems patched religiously, I would be in much bigger trouble right now!

Lessons I learned on this issue:

  • Ask experienced people for advice when you need it. I don’t have any kind of formal training in Systemd and Journald, so was very confused why I couldn’t examine my logs using the provided journalctl tools.
  • Having the redundant servers backing my primary server is great, as it kept all of my services running without issues.
  • Keeping all of my servers updated and patched daily, in sequence so I never have an outage, is a great way to run small business servers.

Next is playing with a better, more automated way of updating and rebooting my servers in sequence.

Thanks again to the guys at Sysadministrivia for guiding me in being able to actually get the information I needed to fix the issue. If you want to hear their comments about it, check it out at their website, or follow the link to S2E3: Ass-Backwards Passwords, and check their show notes for their response to my email.

Taking a rest from RESTful

I have a lot of projects on the go. As I’m sure most developers do, I get curious and decide I need to try and make something that’s been done a hundred times before.

What I have I done that’s been done a hundred times before? Rolled some dice. It’s been done physically, and in almost every programming language I can think of. I did it in high school in QBasic. I did it in college in C, C++ (using OOP methodology), Fortran, and even COBOL. This time, I did it in PHP.

Why did I put myself through the bother of creating a dice roller in PHP that’s been done a hundred times before? The short answer is because it gave me something to do that has nothing to do with any paying project I have on the go right now.

The long answer is that I miss playing D&D. I used to play at least twice a month, over skype, with my brother in-law and a friend of his. It was a small crew, my brother in-law was both the DM and the fighter of the party. His friend was a warlock, who was completely obsessed with finding new books. Not spell books mind you, but books. Last time we played, he had to leave his pack behind while we went off to talk to a Dragon, as he was afraid his books would get burned. I was the unbalanced Moon Elf wizard, who suffered a traumatic brain injury as an apprentice. This basically caused my character to roll randomly on a chart for every decision that had to be made. This has resulted in some pretty funny and scary situations, but my party has adapted and sometimes casts silence on me to prevent me from saying something stupid.

Anyhow, back to the reason for the dice roller in PHP.

We haven’t been able to play for a long time. We are all quite busy in life, and keep missing the opportunity to pick up the game again. Part of the issue is the fact that we need to be rolling so much, so our 3 hour sessions sometimes turn into 6 hours while we figure out what should actually happen and all the dice results to come back.

By making a new dice system, I’ve started the building blocks of being able to make faster decisions, publicly available to anyone who logs into a game server. This may be a small part of a much larger project that I’ve started for myself, but it’s one I enjoy working on.

Originally, I was going to turn this all into a restful system that can be plugged into any system (including mobile apps) to have dice rolls done based on anything that is a valid D&D dice string. In other words, send the server the phrase “2d6+3” and you will get a value anywhere from 5 to 15.

I’ve decided that creating a complete RESTful interface to ask for a dice roll was a little much. Instead, I’ve created a library that can be used in any PHP application (even a RESTful one if you want) to roll the dice in any project. I’ve gone back to my roots of creating libraries that can be plugged into applications, instead of the current method that is popular of making everything a service. For this project at least, I’m taking a break from REST. I don’t have any data that needs to be updated. I don’t have any regular data that needs to be processed, stored, logged, or downloaded.

And it feels good. Go back to your roots when you can. Keep yourself grounded, and remind yourself why you started developing in the first place. It’s been surprisingly rejuvenating.

If you want, I’ve put the dice roller library on github. Take a look at it, and let me know what you think.