Design philosophy #

HTTQ was designed with the following pillars in mind: simplicity, transparency and reliability.

Simplicity #

Many systems are hard to understand. Many systems were designed to be hard to understand.
This is most likely not because their designers set out to design them that way, but for one reason or another, they ended up being hard to understand.
They may be powerful, very full-featured, stable, scalable, flexible, almost flawless.

Yet, the lack of simplicity means one day, when the system is behaving somewhat oddly, an engineer will not be able to tell why the system is demonstrating its current behaviour.
To correct the situation, said engineer will have to interpret log messages, analyse garbage collection patterns, check network latency, memory usage, so on and so forth.
It is not rare for engineers to Google for the exact error messages they’re seeing appear on a system’s logs. If you’re a developer, you’ve probably done this before.

This effect so true, in fact, that even operating systems provide a rich set of detailed analysis tools, such as eBPF, dtrace and perf. These tools are wonders of modern engineering, but they may not always be required if systems are designed to be easily understandable, making these tools more of a last resort, and not the first course of actions.

Engineers do all of this in an effort to try and get a grasp of what it is the system is actually doing, because, to be honest, they’ve lost track of it. And many times, it’s not their fault, software is just hard, ironically enough.

We believe systems should help us keep track of them, by being simple by design.
Therefore, HTTQ was designed having simplicity as its main pillar.
If something’s not simple, if we are not able to communicate an implementation detail on a single piece of paper, in just a few minutes, we don’t do it.

Wouldn’t it be wonderful if every system was easy to reason about?
Bear with us, and ponder about this:

What if every system were built in a way so that the basic details of the system always remain relevant? Each layer of abstraction builds on top of the previous one, but never ruins its importance.

What if every decision that was made when deciding a system’s architecture could be reasonably explained in a few minutes, using nothing but a small piece of paper to communicate the architecture’s design?

What if every entity in software design were relatable and usable by other entities so as to be functional and useful, but never so tightly coupled as to be complected?

Well, there you have it. That’s how we define simplicity, and it’s our commitment when designing software never to cross limits that would somehow harm this notion of being simple by design.

Mission statement for simplicity #

When it comes to simplicity, HTTQ makes the following promises:

• Everything is done via HTTP: Even restarting the service may be done via HTTP. There is no special protocol.
• The HTTP interface must be very straightforward: There shouldn’t be very many endpoints, they should all be well documented, logically organised and adhere to HTTP’s own standards.
• The HTTP interface will never be versioned: There’s no api/v1 and api/v2. It’s a single, cohesive, well-designed API, not two half-broken APIs.
  HTTQ uses semantic versioning, thus the HTTP API version and HTTQ’s own version are the same, semantically.
  A changelog is also provided, it will let you know how the API evolves over time.
• The whole system is distributed as a single binary: It runs on every platform the Go compiler supports. This means it’s very easy to deploy. You just run it, and that’s it. Oddly simple, isn’t it? And it works great with Docker, systemd-nspawn, Kubernetes, everything, basically.
• HTTQ uses zero third-party libraries: We only use Go’s standard library.
  If we understand all the code, we can always determine its simplicity from a well-informed position.
  We ship: our code + Go’s standard library + Go’s runtime. That’s it.
• There are no dependencies: not even a database, only very few shared objects (such as glibc, on Linux).

Transparency #

Once HTTQ is up, running and configured, it should keep users informed of every important code path evaluation, every important event that happens during execution.

The only way for us to deliver such a great experience is to make the whole system extremely transparent. Users shouldn’t have to look for information, instead, the system should provide all possibly relevant information, constantly. Users may filter out and ignore what’s not interesting for them, but all details will be there if they’re ever useful or required.

This goes hand-in-hand with being simple. The simpler a system is, the easier it is to correlate its design decisions with the messages that are being written to its log, as the fewer details there are to be communicated by the system and understood by the user.

Mission statement for transparency #

Sure, HTTQ is proprietary software, but that doesn’t mean we don’t want you to make use of your microscope.
When it comes to transparency, HTTQ makes the following promises:

• Every possibly relevant event that happens during execution will be logged: This means that every state change is logged. Every message being submitted, or read. Every queue being created, manipulated or deleted. Every asynchronous action, when it started, when it finished, how many other asynchronous actions were running when it started and finished executing. How much time it took. Everything.

• We log the same content to both stdout and to the OS logger: Users want to see all log messages as they happen. But they also want to check them later, retroactively.
  /var/log and journalctl are your friends.

• Logs are also be available via HTTP (coming soon): Not connected to HTTQ’s server, and want to check something quickly? A special endpoint provides a tail-like response, keeping you posted on what’s going on inside HTTQ, in real time.

Reliability #

Reliability (noun): the quality of performing consistently well.

Just as it says on the tin.

Mission statement for reliability #

• Predictable resource usage: a message that is 4 megabytes in size should consume, roughly, either 4 megabytes of RAM, or 4 megabytes of disk space.

• Constant-time performance: a message that is 4 megabytes in size should always take the same amount of time to be both written and read, regardless of how many other messages are currently in the same queue. There could be 10 messages, or 10.000 messages, the performance should be constant for a fixed-size message.
  Different threads running concurrently could impact this, but the design itself supports constant-time operations, and this is carried throughout the system as a whole.
  O(1) is better than O(N).

• Atomic writes: Message writes are fully atomic. Concurrent writes are safe and messages are never overwritten/discarded by mistake. Feel free to write from multiple message producers, all at once! Once you get a 200 OK, it’s guaranteed that the message is now in its given queue.

• Atomic reads: HTTQ’s memfifo driver features fully atomic reads.
  Messages are read and returned exactly once (given no network failure), automatically unlinked and garbage-collected, even if there are multiple consumers pulling from a queue, they’ll never get the same message twice.

• Benchmarked: Every HTTQ release is submitted to our benchmarking suite, powered by Apache’s HTTP server benchmarking tool.
  We conduct benchmarks using 100 concurrent writers, and payloads having pseudo-random data that’s unsuitable for data compression, meaning there are no cheats!