Server management. How I envision it.

In the last article, I set out my thoughts on "the cloud". So we now agree that running small and inexpensive VMs can have advantages? No? Then read the article again. When you're ready, feel free to return to this article. Now then: VMs. Those things that run longer than they should, constantly call for updates and are often "End Of Life" seconds after successful setup.

Well. That's how it used to be. And of course this cries out for solutions, as nobody wants to constantly set up new servers, let alone decommission them productively.

A lot has changed

The days when you had to set up your VMs manually are long gone. There are now services such as Laravel Forge, Ploi and many more that make setting up a virtual machine easy and convenient. The integration with services from the Laravel ecosystem has been particularly successful.

So all's well then, right?

Almost. Because Ploi and Laravel Forge, although they have overlaps, are very different tools and solve problems in their own way. Laravel naturally integrates the Laravel ecosystem very closely and is fantastic at implementing new technologies from the Laravel ecosystem. And yet, when using it, I have noticed time and again that I would like to solve some things differently.

However, this is not intended to be a reckoning with Ploi or Forge. I would like to point out that both services are really good. My wishes do not necessarily have to coincide with your wishes. But now. Here they are.

The management of users, teams and their deployment to the servers was imperfect at the time of my last use. A team member did not have their own user account and if a user account could be created (with Ploi this was possible), it was not possible to automate this task, so I had to create these accounts manually again and again.

This brings me to what has been on my mind for some time now: recurring tasks.

I want to be able to create servers using templates. This is not necessarily a technical problem and there are possibilities in Ploi and Forge to implement this at least partially according to my wishes.

From my point of view, the only problem has always been the way in which templates are created. I have always dreamed of a way to version the state of servers and use a specific revision as the basis for a new server. This has long been possible in a professional environment: Terraform, Ansible, Puppet and more allow infrastructures and server states to be pressed into text form. Versioning, for example in Git, is therefore no problem. However, not every user wants to familiarize themselves with Ansible. And there is certainly no good, easy-to-understand solution as a service that is suitable for more than just system administrators.

Server states as the result of events

And so a thought matured: Wouldn't it be wonderful to simply store the events that affect a server and understand the state of the server as a result of the stored events?

A simple example would be:

Of course, these examples are very simple. And yet there is already an interesting idea here. At 2024-08-19 08:08:55, there is a server that has a web and database server. Two domains and a database schema including users exist on this server.

At 2024-08-19 08:03:46, however, the world looked different. There was only one web server with a domain and one database server.

As it is possible to display the status of a server as text, it would also be possible to set a server to this status. If the underlying configuration management (Ansible, Puppet, Chef, Saltstack and many others) guarantees actual idempotency, "time travel" between the different server states would be possible.

It would also be possible to use the events as a basis for other servers. For example, I could create a service that allows me to build a new server based on the events of an existing server. I could also limit these, for example only taking events 1-5 into account.

I've heard all that before

Does that sound familiar? Of course: in accounting, for example, the principle is that every event is recorded and at the end of the chain of events there is a result.

And for all those of us who have been confronted with the term "event sourcing": Yes. That would be a good example of the possibilities of event sourcing. Eventsourcing means that the final state is the result of the things that cause this state. So you simply write down everything you have done with a thing and thus inevitably describe the state that is achieved in the end.

An example from "real life":

The result at the end of this chain of events is: I am awake and have had a coffee. This must necessarily be the case if we assume that these stored events are complete and that I have not omitted anything.

However, these events can also be used later. For example, I could ask myself how much coffee I drink every day. I could also find out in which periods I drink the most coffee. Or how often I usually set the alarm clock to "snooze". Or how long it usually takes from the first ring of the alarm clock to the first coffee.

All these things I can find out, although at the time I started writing down these events, I didn't know what exactly I wanted to do with this information later.

In relation to servers and their state, this means that the way in which I want to create the state of a server does not have to be planned from the outset. Nor does it have to be planned how this state is to be achieved. Whether Ansible, Puppet or Chef is to be used is also irrelevant. In the end, all events are put together and a state is obtained from them.

The problem with secrets

It's all quite simple so far, isn't it? Well, there's this thing about secrets. Unlike in a marriage, we don't usually want secrets to be permanently written down somewhere. And a system of this kind, which is based on event sourcing, means that all content describing the end state must be recorded. How else would it be possible to restore the end state?

Let's take a simple example:

[
    "users" => [
      [
        "username" => 'example',
        "password" => 'password',
        "publicKeys" => [
          "ssh-ed25519 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICkpqD8aGHWS9dt/vaXcWhSrtwIxJR3+1w8IouHtqioF example@localhost",
        ]
      ]
    ]
  ]

This payload could easily be transferred to a server using Ansible. And the password, which is available here in plain text, could also be reliably encrypted. There are various methods for storing and encrypting sensitive data in a database or another service. Rotating the private key is also no problem here.

But what about the stored events? Such an event could be stored in the database as follows

A bad idea. And of course, the individual value or just one field in the JSON object could also be encrypted. But that also comes with problems:

1. how can keys be rotated here?
2. would all secrets be encrypted with just one general key?
3. how would it be ensured that deleted data really disappears? Because event sourcing does not provide for the deletion of events.

There are solutions for this too. And I will go into this in another article.