Zaur's Thoughts

The "Handover" teams

2025-11-08T18:15:00+02:00

Do you know what my favourite corporate sin is when it comes to code bases? It's a handover from one team to another. "But wait, Zaur, there are legitimate reasons for handovers!" Of course there are. Organisations are evolving structures, and sooner or later quite a few parts attached to one of the limbs will be cut off and stitched to the other. The code bases follow.

We can't avoid handovers, and it's our job - as the source team, and more importantly, as the receiving team to make a code handover as smooth as possible. But the process is so often compressed into a half-day "handover session", after which the source team wishes all the luck in the world to the new owners and blasts off to their new adventures.

A short handover session between teams

Good luck to the Blue team!

Do you truly own the code at this point? You can own the code on paper, but you truly own the code only when you know it inside out, can picture data flows with your eyes closed, understand the architecture, and have lived the decisions made along the way. Your code is part of you - your experience, your knowledge, your skills, your vision. Simply put, you have to get intimate with the code to own it.

You see where I'm going with this - no amount of "handover" will make you the true owner of the code, at least not immediately. You'll get there eventually, with time and the hundreds and thousands of lines added and removed by you. To smoothen the transition period, you'll need help from the source team.

Working together

What if, instead of having a handover session, you had a handover period in which both teams form a single virtual team and work together? And by working together I mean mob/ensemble programming sessions, in which knowledge gets shared and slowly transferred from one team to another. You write new code, fix bugs, do maintenance and refactoring, release and monitor - all with the specialists from the source team. If the teams are not physically co-located, try to get together and work together for a few days - this will create a strong bond between you.

A short handover session between teams

Blue team truly owns the product after working together with the orange team.

I was there, Gandalf

I've been through different handovers, some lasted hours, others months. The first left me with a feeling of emptiness, dissatisfaction and anger: "How anyone in their right mind expect me to maintain and work on this code?". The latter were quite the opposite - after weeks of hands-on experience with the product and its code base, our team had a solid "know-how" foundation under our feet, and we were eager to take over and drive it further.

Why did I accept the first one? Because I was young and green. I accepted business needs as challenges, and threw myself into the fire with no hesitation (Am I not a professional?). I never considered the long-term consequences. Then they hit. Business demanded new features, but I couldn't deliver because I didn't truly own the code.

Lessons learned.Today I thoroughly explain to the business all the risks associated with rushed handovers. Instead, we design a migration plan - together with the business, the old team, and the new teams. Then we work together: fix bugs, deliver new features, make experiments and monitor them in production. When migration period is over, we are the true new owners of the code base.

Fixing low display brightness on LUKS password prompt

2025-08-19T22:34:00+03:00

Given:

Framework Laptop 13 AMD 7040 edition;
Arch Linux with busybox-based initrd;
Encrypted drive: LVM partitions on LUKS

The problem: The laptop display brightness fluctuates after booting when running on battery power.

At first, the brightness is rather high, and systemd-boot entries are clearly visible.
However, after selecting the kernel to boot, the brightness drops to minimum. You hardly see anything when the boot reaches LUKS password prompt.
The boot process continues after successfully entering the password. When systemd kicks in, the systemd-backlight service restores the brightness.

The solution: set a brightness on startup via a udev rule.

Add a new udev rule which sets brightness through ACPI:
1
SUBSYSTEM=="backlight", ACTION=="add", KERNEL=="amdgpu_bl1", ATTR{brightness}="20000"
save it to /etc/udev/rules.d/10-backlight.rules.
Include the rule path to mkinitcpio's configuration in /etc/mkinitcpio.conf:
1
FILES = ("/etc/udev/rules.d/10-backlight.rules")
Generate a new initrd by running mkinitcpio -P.
Take out the power cable and reboot the machine. Enjoy the visible decryption password prompt :).

The Power of Atomic Habits

2025-08-18T22:02:00+03:00

We shape our habits and our habits shape us. You go to sleep late every day — that's a habit. Do you wake up early? That's also a habit. You always eat dessert after dinner? You play violin at 5 o'clock in the evening? You kiss your kids good-night every day? You get a bottle of wine every time you go grocery shopping? So many things we do without thinking much about them! Things that have become our second nature, our identity. Behaviours we would like to stick with or get rid of. At the end of the day, most of them are habits.

On the surface, a habit is a really simple thing. It's a behavioural pattern we repeat again and again in similar situations. When you want a routine to stick, you need to repeat it often, until it becomes your second nature. And if it's a bad habit you want to break, you just need to stop doing it and avoid repeating it in the future. Ah, if only it were that easy!

The good news is that you have already taken the first step — you are aware of your current situation and you've set a goal. Where people usually fail is in the implementation. How can you convert the desire to change into something that becomes part of you? Let "Atomic Habits" by James Clear to guide you in this thrilling quest.

There is a reason the book topped the New York Times best-seller list for 164 weeks. It explains how our evolutionary ancestry affects human behaviour and habits and provides practical tools developed by the author to help readers cultivate the habits and becoming the type of person they want to be.

James describes a habit as a loop consisting of four stages: cue, craving, response, and reward.

A cue is a trigger that initiates the behaviour, which is expected to result in a reward. Just yesterday, I came home from work and found my daughter eating an ice-cream. I was hungry, and that cue signalled that I could quickly deal with the hunger by grabbing an ice-cream!

A craving is a desire that follows a cue. I want an ice-cream! I start imagining how it melts on my tongue, its sweetness and texture. The urge for sugar rush kicks in, drowning out the reasonable voice in my head saying that perhaps I should first have a proper supper.

A response is the action. I rush to the kitchen, and grab an ice-cream from the freezer. I unwrap it impatiently and sink my teeth into the frozen flesh.

A reward is the outcome that satisfies a craving and reinforces a behaviour. The latter part is crucial, because reinforcement is what makes the behaviour more likely to be repeated in future. In my case, the reward was the pleasant taste of the ice-cream and no longer feeling hungry. Though the "Why did I just do it, instead of dining properly?" question pops up right after :). Indeed, you're truly free only when you are in control of your cravings!

Building a new habit requires creating this loop, while breaking a habit means breaking the loop. This "technical" task is a mere facade. To make it work, you've got to make the habit part of who you are, part of your identity, and it is not easy at all. That's why James recommends starting with "atomic" habits - small, but regular steps which nudge you toward your new self.

Note how he avoids the "goals" and the "results". Losing 5 kilos is a goal and an impressive result, but becoming a person who has healthy eating habits and regularly exercises is the mission.

The problem with a goals-first mentality is that you're continuously putting happiness off until the next milestone.

...

The true question is: "Are you becoming the type of person you want to become?" The first step is not what or how, but who. You need to know who you want to be. Otherwise, your quest for change is like a boat without a rudder.

One small step at a time — eating sweets only once a day, then once every second day, then once every third day - as long as you stick to the rules, you've cracked it! Although, we've all been there trying hard and inevitably failing. That's when James introduces the framework called the Four Laws of Behavior Change.

In a nutshell, to build a new habit you need to:

The 1st law (Cue): Make it obvious.
The 2nd law (Craving): Make it attractive.
The 3rd law (Response): Make it easy.
The 4th law (Reward): Make it satisfying.

Contrary, to break a bad habit you need to invert the laws:

Inversion of the 1st law (Cue): Make it invisible.
Inversion of the 2nd law (Craving): Make it unattractive.
Inversion of the 3rd law (Response): Make it difficult.
Inversion of the 4th law (Reward): Make it unsatisfying.

James dives deep into these laws, shares examples, gives you tools — all so you don't have to rely on the scarce willpower, but rather build a system that actually helps you build and break habits. As you flip the last page of the book, you realise, that the only thing standing between you and that amazing future yourself - is just you.

One thing I'd add from my own mess-ups: breaking a habit isn't just about stopping — it's also about swapping. The cue and craving might be unavoidable, but you can tweak the response. (Disclosure: this is my experience, but not my original thoughts, they come from another book I love and recommend - "The Power of Habit" by Charles Duhigg.)

Habits, as James Clear says, are simply reliable solutions to recurring problems in our environment. But people who don't have their habits sorted? They're often the least free. Do you feel like your freedom was taken away? You can take it back — one atomic habit at a time!

xkbcommon and my custom Russian phonetic layout

2025-08-01T12:00:00+03:00

The Soviet Union officially dissolved in 1991, but Russian language remained the region's lingua franca for decades. Many children of my generation were simultaneous bilinguals; we spoke both Azeri and Russian since childhood. I learned to type in Russian, but I never ever learned the official Windows Russian keyboard layout, because I didn't have a keyboard with Russian alphabet printed on it. The alternative was a phonetic layout, and I still remember a program called "Alt-Win", which allowed to select Azeri Latin, or Russian phonetic keyboard layouts in Windows 9x. Over the years, I used other tools in Windows XP and Windows 7 to recreate this layout. And then I moved to Linux.

In Linux, the only tool you need to create your own keyboard layout is a text editor. Since you are already reading this, scroll down to feed your curiosity!

Note

In this article, I refer to Russian phonetic keyboard layouts as keyboard layouts where Russian letters are mostly located at their similarly sounding letters in English, on US QWERTY keyboard. For example, the Russian letters "а", "о", "м", "т" are typed using the English "a", "o", "m", "t" keys.

The trick is to map 33 Russian alphabet letters to a keyboard that has only 26 Latin letters (used in English) while retaining the necessary punctuation marks.

Custom layout - the naive way

The naive way to create a custom layout is by modifying an existing variant definition in one of the symbols files from /usr/share/X11/xkb/symbols. That's what I originally did — I modified the ru file and its phonetic section:

partial alphanumeric_keys
xkb_symbols "phonetic" {

    name[Group1]= "Russia - Phonetic";

    key <TLDE>   {[       Cyrillic_yu,       Cyrillic_YU ]};
    key <AE01>   {[                 1,            exclam ]};
    key <AE02>   {[                 2,                at ]};
    key <AE03>   {[                 3,        numbersign ]};
    ...
    ...
}

Here, the first column contains the symbolic key names [1], the second column contains the corresponding key symbol for a key pressed without modifiers, and the last column contains the key symbol for a key pressed with Shift.

This works, BUT your user needs write permissions for the file, and the file will be overwritten anyway every time XKB gets updated by the system.

Can you believe that for over 15 years I used the same customised ru symbols file, and replaced the upstream file over and over? I did this manually every time the package responsible for /usr/share/X11/xkb got updated. The file followed me as I moved from Ubuntu to Debian, and from Debian to Arch. And honestly speaking, I never bothered to dig deeper and learn the mechanisms behind the command to set up my desired layouts:

setxkbmap -layout 'us,fi,ru,az' -variant ',,phonetic,' -option grp:shifts_toggle

That allowed me to cycle through US, Finnish, Russian phonetic and Azeri layouts by pressing both shifts.

This changed when I installed Arch on my new laptop. I finally decided to find a "smart way" and learn the mechanisms behind layouts in Linux graphical environments.

Bits about XKB

Imagine a system with a graphical environment backed by Wayland compositor. Let's find out how pressing a key on the keyboard results in a symbol appearing in an application. I'll omit deep technicalities to keep the narrative short.

Diagram explaining how pressing a keyboard key ends up in application while passing through libxkbcommon.

A high-level diagram of how a key press event travels from a keyboard to a user-space application.

Wayland compositor notifies the application about the current keymap configuration. This happens when application starts or the user changes keyboard layout. The application loads the keymap via xkbcommon library.
The keyboard device sends a hardware signal - a scancode to the Linux Kernel.
The kernel's evdev subsystem maps the scancode to a keycode - a representation abstracted from hardware.
Wayland compositor uses libinput to receive keyboard and other input device events from the kernel. It notifies the application (a Wayland client) about a pressed/released key with a keycode, and modifiers.
The application (using GTK, Qt, or other Wayland client implementations) passes the loaded keymap context, the keycode and modifiers to xkbcommon.
xkbcommon applies keycodes/compat/geometry/symbols/types (KcCGST) configuration [2], to map the input to the final symbol.

In a nutshell, it's xkbcommon's job to translate Shift + L keystroke to Cyrillic Л symbol, when I have my Russian phonetic layout activated.

As Daniel Stone, the maintainer of xkbcommon nicely puts it:

It’s about two things: parsing and loading keymaps, and managing their ongoing state. State, in terms of keyboards, is the usual suspects — modifiers (e.g. Shift, Alt), multiple layouts (mostly for multiple languages), and LEDs. In general, you only want one person keeping a canonical copy of the state, and distributing it to its clients, as both the X server and Wayland do today. xkbcommon allows for this mode of operation, and is indeed how all current Wayland clients handle keyboard input [3].

Historical reference

X Keyboard Extension (XKB) protocol was developed in early 1990s and included in X Window System, Version 11 Release 6 (X11R6) in May 1994 [4]. XKB extends the core X protocol by offering support for

Multiple keyboard layouts
Sophisticated modifier handling
LED control
Keyboard geometry description,
Various keyboard behaviors beyond what the core X protocol provides [5].

Rather than reinventing the wheel, Wayland protocol currently relies on XKB format for keymaps. Unlike X11, where clients received a symbolic representation of the key, in Wayland it's up to the clients to interpret keyboard events and map the pressed keys to symbols. xkbcommon library is typically used for this purpose.

Custom layout - the smart way

The xkbcommon documentation comes with a tutorial on adding a custom keyboard layout [6]. I don't want to repeat that here, and rather encourage you to dive into it and get the information from the original source.

However, I'll briefly describe my Russian Alternative Phonetic setup.

Symbols file:

I placed my custom layout variant in a new symbols file $HOME/.config/xkb/symbols/ru_alternative. This file contains translation of symbolic key codes into the desired key symbols (link).
Rules file:

I added a rule to ~/.config/xkb/rules/evdev file (link). The purpose of the rules file is to map between user-friendly configuration, and the configuration used by xkbcomp keymap compiler [7].
I made the layout discoverable in GNOME by adding an entry to ~/.config/xkb/rules/evdev.xml (link).

That's it! A simple solution for a single-person machine, and no more half-baked solutions with broken layouts after system updates :)

References

[1]	xkbcommon - The XKB keymap text format, V1 / Keycode.

[2]	xkbcommon - RMLVO vs. KcCGST.

[3]	xkbcommon: what is it?

[4]	X Window System, Version 11 release 6 (X11R6).

[5]	DeepWiki - X Keyboard Extensions.

[6]	xkbcommon - User Configuration

[7]	xkbcommon - Rule file format

Framework Laptop is awesome!

2025-07-20T23:16:00+03:00

As I am writing this text, enjoying Finnish summer from an armchair in my backyard, a feeling of satisfaction fills me. After nine years with a home desktop machine, I'm finally back to using a personal laptop. I can once again work on my projects from anywhere - from a sofa in the living room or an airplane 10 km above the ground. "Good for you, Zaur, you've got a laptop, why are you so excited?" Let's go back a few years ago. It all began with a loud rant...

A rant about a broken screen

"...Dammit! I did cut that antenna cable! Well, there's still one left. Damn Lenovo and all modern electronics manufacturers!" - I cursed every second of the two hours it took me to disassemble and reassemble my better half's IdeaPad to swap the broken screen. She had accidentally closed her laptop lid with a smart card reader cable lying on the keyboard — resulting in the lower-right quarter of the screen dying.

"Replacing a laptop screen — how hard can it be?"

It can be quite hard. That's because nowadays we design electronics for fast production, not easy repairs. Even when the broken component is not soldered into the motherboard, you need a ton of luck to find a replacement part and fix things. You won't find any official manuals — perhaps only videos made by kind people on Youtube.

In my case, I had to disassemble the chassis from both the top and bottom, use a hairdryer to loosen the glued screen frame, unfasten dozens of screws, deal with fragile antenna cables (and yes, I ended up smashing and cutting one!). All this — by following a video guide bought from a random Australian laptops repair shop!

That's when I promised myself: next time I buy a laptop, it's gonna be the most repairable laptop out there. Little did I know, the Framework Laptop had already been on the market for two years!

Imagine a laptop

Imagine a modern, sustainable laptop. It should be designed to last, be easily customisable and repairable. That's the Framework Laptop in a nutshell. It all starts at the web shop. Would you like a pre-built laptop with Windows, or a DIY edition with components in separate boxes and no OS installed? - Your choice! Are your daily needs specific to two USB-A ports and an Ethernet port? Sure thing! Don't like the official memory and storage? Bring your own! Want a display with a higher refresh rate? You've got it! And that's just the tip of the iceberg.

Configuring your Framework Laptop.

The experience continues as you own it. Is your display broken? The official manual will guide you to replace it in just ten minutes. The only tool you'll need is a Torx/Philips screwdriver, which comes with the laptop. Your battery doesn't hold a charge anymore? You know what to do. Has it been a while, and you need the latest and fastest -generation processor? Get a new motherboard, and put the old one into a case designed to convert Framework mainboards into desktops!

Replaced an old but fully-functioning motherboard? Why not give it a second life as a mini PC?

Finally, imagine a laptop with full Linux support out of the box and a great community of enthusiasts around it! You clearly see, why Framework laptop is so appealing to me.

What I also appreciate about Framework Computers Inc. is that their sustainability page starts with a big, fat, and honest "We are not sustainable" heading. However, the page explains how the company aims to reduce environmental impact by creating products that last longer.

The chassis

The chassis quality is top-notch. The light aluminium body and matte glass touchpad look and feel beautiful! The bezel is made of plastic, but it's customisable - you can choose from a handful of colours and transparent options. The keyboard provides good feedback, and is quite comfortable for long typing sessions. Its backlight is soft and adjustable. Thin exhaust air vents at the back are a plus when you're using an external mouse - you won't get your palm fried.

The laptop is warm and rather silent for an x64 machine. I can comfortably keep it on my lap under light coding/browsing workload for hours, though it feels like burning compared to an M-series MacBook. Watching 4K videos boosts the fan and the bottom plate gets noticeably hotter.

The 13.5'' display has 3:2 aspect ratio — that's more vertical space compared to the widespread 16:9. Perhaps not ideal for watching movies, but amazing for vertical content: coding, writing documents, reading, and web surfing. And guess what? The display even comes in two options: the original 2.2K and the newer 2.8K with higher resolution and refresh rate.

The battery life is good for me, but there have been plenty of complaints from the users. I've restricted battery charging to 80% — this prolongs battery life, and is more than enough to last for 4-5 hours of light workload, till the charge drops to 30%. Which can be roughly approximated to 8-10 hours of full-to-zero battery life — a solid result for an x64 laptop.

Nothing beats hardware switches when it comes to privacy!

Oh, did I mention that there is a fingerprint reader and there are hardware switches for the microphone and the camera?

Using GNU/Linux

Getting into nerdy stuff — I ditched Windows from my computer back in 2007 and never looked back. So Linux support is paramount for me. It was also the reason I got the "latest-minus-one" generation laptop — because all the issues that may surface with the latest and greatest hardware have already been solved for the previous generation.

I am happy to say, that everything just works. It's been two weeks, and I had zero problems. Oh, don't get me wrong, I'm still tuning and tinkering with the setup (after all it's Arch Linux!). There are issues, like the uncalibrated display, or imbalanced speakers I've learned about from the fantastic Framework Laptop 13 wiki page. Luckily, the community has created necessary colour profiles to improve colour reproduction and equaliser presets to enhance the sound. There are some other minor annoyances, like display automatic backlight adjustment jumping in discrete steps, instead of changing gradually, but all that can be fixed with proper software updates.

Shall you join the club?

Before I say anything, I must make a confession: I like MacBook chassis, especially the battery-life-to-processing-power ratio. But I absolutely hate Apple's policies when it comes to the right to repair. As another example, compare this to this. So, let's assume you're also not an Apple person and you want a PC. Should you get Framework Laptop?

A pre-built Framework Laptop is more expensive than other manufacturers' laptops with similar specs. However, money-wise it's about long-term savings and voting with your wallet for a sustainable option. But if you are making this choice, why not go with the more affordable DIY edition from the start to get the full experience? You can still get all components and even Windows from Framework. It'll take you five minutes to assemble the chassis, and about an hour to install Windows.

And if you're Linux enthusiast or an IT professional — JUST GET IT :)

My setup specs and other numbers for nerds

Framework 13 DIY edition with 2.8k 2880x1920 screen, with AMD Ryzen 5 7640u CPU.
Translucent Black bezel.
Crucial 32GB RAM Kit (16GBx2) DDR5-5600.
Crucial P3 2TB PCIe M.2 2280 SSD.
Ports: USB-C, USB-A, HDMI, SD Card.
The CPU core temperature sensor (k10temp) showed the highest temperature among other sensors. The values were 40°C for "light workload", 50°C while playing YouTube FullHD videos, and 60°C while playing FullHD 60FPS HVEC-encoded video. The ACPI Thermal Zone sensors showed 35°C, 40°C and 50°C respectively.

Hexagonal architecture and Python - Part IV: Lightweight integration tests

2025-05-10T18:00:00+03:00

Intro

It is very easy to unit test the components of an application that follows Hexagonal architecture principles. The components belong to certain layers, and it's quite straightforward to mock the following dependency layer. However, the problem with such an approach is that we never test the application as a whole, and have to rely on expensive integration or end-to-end tests to do so.

We can test the application components in sociable manner, leaving our components' direct dependencies as-is, and pushing the mocks further to the application edges.

If we push the mocks too far, we end up with lightweight integration tests. Let's explore how we can utilise such tests in a context of Django application and what benefits we can reap.

The original article focused on such "lightweight integration" tests. I even dared to call them "sociable"! Jarkko "jmp" Piiroinen was very generous to thoroughly review the article. He pointed that sociable tests, rather implies testing a thing within the application core (or "Application" in Alistair Cockburn's definition) with its neighbours/collaborators. Testing the whole flow by mocking the rightmost calls is "sociable tests pushed too far", or as Jarkko put it "rather lightweight end-to-end tests".

The updated article is about exploring the ways to conduct such tests in a Django application setup, focusing on the benefits and caveats.

Table of Contents

Intro
Hexagonal architecture - as described in the articles series
Testing the application
Architecture revised
Lightweight integration tests -driven development
Use case: Vote for an article
Use case: Only existing user can vote for an article
The price to pay
Conclusion
Acknowledgements

Time flies (darn, I say that again)! I wrote the original "Hexagonal Architecture and Python" series back in 2022, which was only three years ago (honestly, it feels like yesterday!) Since then my understanding of RESTful API design and Domain-Driven Design has deepened. But the biggest shift, however, has occurred in where I start and how I test.

I used to believed that the implementation of the domain model should precede exposing any API. I still design the domain model first before deriving API structure from if. However, I implement barebones API first, initially hard-coding and short-circuiting domain logic, to enable early integrations and feedback. This approach helps steer the development in the right direction, keeps API consumers happy and avoids "big bang" release surprises.

This led to the second change. I used to TDD every layer and every component of the application independently. Today, I aim at unit testing the interconnected parts of the system.

Bear in mind, that the system users often don't care about what's happening under its hood. They are more concerned with the system's public contract - its behaviour when queried and commanded through the public interfaces. System developers, on the other hand, also need assurance that the system interacts with its downstream dependencies as expected. We can apply behavior testing similarly: by testing the system via its public interfaces and downstream interactions, we reduce the need to test the application layers in isolation.

Just so we're clear, I'm not calling anyone to put the whole thing together and test it on the launch pad. That's a recipe for disaster - remember what happened to the Soviet N1 Moon rocket?

Adverse characteristics of the large cluster of thirty engines and its complex fuel and oxidizer feeder systems were not revealed earlier in development because static test firings had not been conducted. (Wikipedia)

Instead, I'm looking for relatively lightweight ways to test the entire system, without going for a full end-to-end setup.

Hexagonal architecture - as described in the articles series

Let's recall the application and its architecture as described in the articles series:

A hexagonal architecture application diagram

The example application is built using hexagonal architecture principles. This is beyond what Alistair Cockburn describes in the original "Hexagonal Architecture" article. Cockburn only defines Ports, Adapters, Application and their interactions.

The Application has a layered structure: the Core, which includes Domain models and services; The driving/in/API (application programming interface) and driven/out/SPI (service provider interface) ports for incoming and outgoing interactions; and API and SPI adapters.

Python wise:

The Domain model consists of classes and methods which reflect the real world domain rules. They are "pure" in nature, because they don't have any outer world dependencies.
We can think of API and SPI ports as delegates - basically, function signatures. We can use Python Protocols or Abstract Base Classes to define them in OOP-compatible way.
The Application Services implement API ports and orchestrate the flow between domain models and SPI ports. They invoke SPI ports and can also invoke other API ports when necessary.
The SPI adapters implement SPI ports. SPI adapters are the bridges between the application and downstream dependencies (databases, cashes, message queues, file systems, other services etc.)
The API adapters connect the outer world to the application. They receive commands and queries through different channels (command line, http, RPC, message queue etc.) and invoke API ports. The same API port can be invoked by many adapters - think of making the same query via a command line, or via a RESTful HTTP interface.

Testing the application

Martin Fowler speaks of "Solitary" and "Sociable" tests in the renowned article "Unit Test". Solitary tests mock the direct dependencies of the component under test. This is how I originally implemented tests in the "Hexagonal Architecture with Django" project. Tests like these run blazingly fast, but they only test the isolated behaviour of a given component (like an HTTP controller, an application service, or a database entities mapper) with fake downstream layer interactions.

A diagram of solitary API test

A mockist-style HTTP adapter test.

Defining sociable tests is trickier:

When xunit testing began in the 90's we made no attempt to go solitary unless communicating with the collaborators was awkward (such as a remote credit card verification system).

I think that the term “unit testing” is appropriate because these tests are tests of the behavior of a single unit. We write the tests assuming everything other than that unit is working correctly.

—Martin Fowler, "Unit Test".

So, in my perspective a unit test is sociable when it verifies a unit of behaviour across multiple components boundaries. In the verified interaction, at least one direct dependency is not mocked, however the further dependencies can be mocked.

A diagram of sociable API test

A classical-style HTTP adapter test.

Finally, we utilise higher-level integration or end-to-end tests to verify the behaviour flow throughout the application and its dependencies. Nowadays, tools such as Testcontainers allow the isolation of these tests by spinning up the real databases, caches and other dependencies for a short test lifetime. However, this setup might be too resource-intensive, or other downstream dependencies, that cannot be dockerised may need to come into play. This leaves us with slow and fragile classical e2e tests in an integration environment.

What if we push sociable tests a bit further? We can mock the layer which stands between our adapters and external dependencies. This ensures that an API test flow passes forward and back through the whole application in a deterministic and isolated environment:

A diagram of a lightweight integration API test.

A lightweight integration test.

Architecture revised

Software architecture is important because it allows us to independently evolve different parts of the application and delay decisions. In a way, architecture constrains us, but helps keep the system organised.

My presentation of Hexagonal Architecture in this series of articles, especially the testing part, was a bit bloated. Alistair Cockburn said nothing about the application's (core) internals. It's up to us whether we introduce "Service", "Domain" or any other layer or concept. Just think: if an HTTP adapter handles a GET request to return raw data from a single table in the database, why even bother with "Service" and "Domain" layers which would only perform data transformations?

So, when it comes to lightweight integration testing, we leave the application core as a black box, and interact with adapters only.

A hexagonal architecture application diagram

We are free to implement the application core in any way when applying Hexagonal Architecture principles.

Lightweight integration tests -driven development

A classical end-to-end test is a poor development driver. It takes minutes, sometimes tens of minutes to run, often breaks, has a complicated setup, and ... (your favourite fallacies here). A lightweight integration test is somewhat better: it's fast and fully under our control. However, it rather drives adapters development, NOT the application core development. It also requires a complicates setup, especially when it comes to test doubles.

Nevertheless, let's explore how such a test drives implementation. What steps would we take?

Pick a behaviour unit.
Write a test for its API, asserting only the response.
Write a minimal implementation to pass the test.
If the behaviour is not pure and hits downstream dependencies, write a test similar to (2), but this time verify the dependency interaction. Mock the dependencies and verify interactions via these mocks.
Write a minimal implementation to pass the test.
If any other tests break, fix them. Here, you enter a recursion, starting at step 1 with a broken behaviour unit.
Refactor. Remove duplication in application and tests code. Apply Clean/Hexagonal architecture principles when necessary.

Let's walk through these steps by implementing the original "Vote for an article" use case from scratch. I will not include implementation code here, just the tests code. That's because the implementation stays the same, but the tests are quite different.

Use case: Vote for an article

Let's recap the requirements for this use case:

Every article has a rating.
Users can change an article's rating by either "upvoting" or "downvoting" it.
To vote on an article, a user's karma (their user rating) needs to be higher than 5
A user can only vote on each article once.

This translates into the following user story:

Given A registered user
      who can vote on articles.
When they vote on an article.
Then the article's rating changes to reflect their vote.

1. Pick a behaviour unit

We start with a happy case scenario. Our TODO list looks like the following:

### TODO

* [ ] Test that "When user successfully votes for an article,
      the system responds with HTTP CREATED status and the vote details".
* [ ] Implement the view which handles voting for articles.

2. Write a test for its API

def test_when_user__successfully_votes_for_existing_article__system_returns_http_created_with_vote_details():
    response = post_article_vote(
        article_id='3f577757-0000-0000-0000-000000000000',
        user_id='9af8961e-0000-0000-0000-000000000000',
        vote='DOWN'
    )

    assert response.status_code == HTTPStatus.CREATED
    assert response.data == {
        'article_id': '3f577757-0000-0000-0000-000000000000',
        'user_id': '9af8961e-0000-0000-0000-000000000000',
        'vote': 'DOWN'
    }

Here post_article_vote() is a helper function which invokes the view via APIRequestFactory. This keeps the tests fast and limits the context to the strictly necessary pieces.

We run the test and it predictably fails.

3. Write a minimal implementation to pass the test

The first round implementation which passes this test is amazingly simple: the view just echoes the posted data.

Echo HTTP request payload in response.

The minimal implementation which passes the test - echoing POST payload back.

Is that it? Does the API interaction fully describes the system behaviour? Not really, because this behaviour is not pure, i.e. it has side effects, and we need to store the vote somewhere.

4. Write a test to verify downstream interaction

Mocking Django model persistence

### TODO

* [ ] Test that "When user successfully votes for an article,
      the system persists the vote"
* [ ] Implement persisting the vote.
* [X] Test that "When user successfully votes for an article,
      the system responds with HTTP CREATED status and the vote details".
* [X] Implement the view which handles voting for articles

We're using a database for storage, and Django provides out-of-the box tools to run tests with a database backend. You could use an in-memory SQLite, spin up a heavier DB in a Docker container, or even use a shared database (oh the horror!). But in my book, all these methods are rather integration tests, than unit tests, because you are not really testing the application in isolation anymore. The same applies to other integrations like message queues, event busses, downstream services, file operations etc.

What we can do though, is mock and spy on the persistence mechanisms that are furthest to the right (from the application perspective) or at the very top (from Django's perspective). For instance, we can mock a model's save() method and spy on it. We can even create a pytest fixture like this:

@pytest.fixture
def mock_persisting_article_vote():
    with patch.object(ArticleVoteEntity, 'save', autospec=True) as mock:
        def _mock_persisting_article_vote():
            return mock
    yield _mock_persisting_article_vote

Now we can spy on the save() method calls and verify the instance details passed as self argument. But let's write a test first!

def test_when_user__successfully_votes_for_existing_article__system_persists_the_vote_in_the_database(
    mock_persisting_article_vote,
    post_article_vote
):
    spy = mock_persisting_article_vote()

    post_article_vote(
        article_id='3f577757-0000-0000-0000-000000000000',
        user_id='9af8961e-0000-0000-0000-000000000000',
        vote='down'
    )

    # use the captured ``self`` value from ArticleVoteEntity().save()
    entity = spy.call_args[0][0]
    assert entity.article_id == UUID('3f577757-0000-0000-0000-000000000000')
    assert entity.user_id == UUID('9af8961e-0000-0000-0000-000000000000')
    assert entity.vote == 'down'

Substitute Django database persistence mechanism with a mock.

Substituting Django database persistence mechanism with a mock in a unit test.

5. Write a minimal implementation to pass the test

The implementation for this can be also naive - we can create and save an ArticleVoteEntity from the HTTP POST data. A simple and well-known Django Rest Framework flow.

7. Refactor

Mocks are for humans, not machines!

I personally find spy.call_args[0][0] very ugly and non-intuitive. The fact that there is a comment explaining what the line does is already a code smell. How about writing a custom spy object which stores the saved entity, so that we can get the captured entity via self-explanatory

def test_when_user__successfully_votes_for_existing_article__system_persists_the_vote_in_the_database(...):
    ...
    entity = spy.saved_article_voted_entity
    assert entity.vote == 'down'
    ...


class SaveArticleVoteEntitySpy:
    saved_article_voted_entity: Optional[ArticleVoteEntity] = None

    def save_article_vote_entity_mock(self, entity, *_args, **_kwargs):
        self.saved_article_voted_entity = entity


@pytest.fixture
def mock_persisting_article_vote():
    spy = SaveArticleVoteEntitySpy()

    with patch.object(ArticleVoteEntity, 'save', autospec=True) as save_mock:
        def _mock_persisting_article_vote() -> SaveArticleVoteEntitySpy:
            save_mock.side_effect = spy.save_article_vote_entity_mock
            return spy
        yield _mock_persisting_article_vote

Remove duplication in tests

You have probably noticed that the API response and Dependencies interaction tests have the similar Given and When or Arrange and Act parts. As a matter of fact, they should have identical parts because we're testing the same behaviour, just from different ends!

Let's not repeat ourselves. One way is to group the behaviour tests in a single class and extract the Arrange and Act parts into their own fixtures and methods. Now, this might be an overkill for a small setup, but here is what I got in the end (spoiler alert!):

class TestWhenUserSuccessfullyVotesForExistingArticle:
    @pytest.fixture(autouse=True)
    def arrange(
        self,
        given_a_user_who_can_vote,
        given_no_existing_article_votes,
        mock_persisting_article_vote,
        post_article_vote
    ):
        given_a_user_who_can_vote(
            UUID('9af8961e-0000-0000-0000-000000000000')
        )
        given_no_existing_article_votes()
        self.persisting_article_vote_spy = mock_persisting_article_vote()
        self.post_article_vote = post_article_vote

    def act(self) -> Response:
        return self.post_article_vote(
            article_id='3f577757-0000-0000-0000-000000000000',
            user_id='9af8961e-0000-0000-0000-000000000000',
            vote='down'
        )

    def test_system_returns_http_created_with_vote_details(self):
        response = self.act()
        assert response.status_code == HTTPStatus.CREATED
        assert response.data == {
            'article_id': '3f577757-0000-0000-0000-000000000000',
            'user_id': '9af8961e-0000-0000-0000-000000000000',
            'vote': 'DOWN'
        }

    def test_system_persists_the_vote_in_the_database(self):
        self.act()
        entity = self.persisting_article_vote_spy.saved_article_voted_entity
        assert entity.article_id == UUID('3f577757-0000-0000-0000-000000000000')
        assert entity.user_id == UUID('9af8961e-0000-0000-0000-000000000000')
        assert entity.vote == 'down'

So far, so good, nothing is failing! Our next step, however, is going to shake things up a little bit.

Use case: Only existing user can vote for an article

1. Pick a behaviour unit

### TODO

* [ ] Test that only existing user can vote for an article.

First, let's define an existing user as a record in the database. Second, our current tests already cast a vote by the given user_id. So, no matter how we tweak the "Assert" and "Arrange" (or "Then" and "When") parts of the test, it will pass as long as we keep those parts aligned. We need a new test that will break, and guide implementation further:

### TODO

* [ ] **Test that a non-existing user can't vote for an article.**
* [ ] Test that only an existing user can vote for an article.

2. Write a test for its API

Our system responds with HTTP NOT FOUND in this case:

def test_when_voting__as_non_existing_user__system_returns_http_not_found_with_error_details(
    given_no_existing_users,
    post_article_vote
):
    given_no_existing_users()

    response = post_article_vote(
        user_id='a3853333-0000-0000-0000-000000000000'
    )

    assert response.status_code == HTTPStatus.NOT_FOUND
    assert response.data == {
        'detail': "User 'a3853333-0000-0000-0000-000000000000' not found",
        'status': 404,
        'title': 'Error'
    }

This looks simple - we setup a system that doesn't have any users and then try to vote. The system should respond with an error. given_no_existing_user() is a stub which returns no users when you try to search for them. In Django terms, that's a temporary ObjectManager, with the necessary methods' stubs:

class VotingUserEntityEmptyObjectManagerMock:
    def get(self, *_args, **_kwargs) -> VotingUserEntity:
        raise VotingUserEntity.DoesNotExist()

The fixture substitutes the object manager for a single test run:

@pytest.fixture
def given_no_existing_users():
    original_voting_user_entity_manager = VotingUserEntity.objects

    def _given_no_existing_user():
        VotingUserEntity.objects = VotingUserEntityObjectManagerMock()
    yield _given_no_existing_user

    VotingUserEntity.objects = original_voting_user_entity_manager

If we run the test, it fails because the logic implementation is missing. To turn the test green, we need to query the database, check whether a user exists, and return HTTP 404 otherwise. Once that is ready, the last test passes, however, the first two tests fail when trying to access the database since there is no way to set up existing users yet:

Unexpected error occurred: Database access not allowed,
use the "django_db" mark, or the "db" or "transactional_db" fixtures
to enable it.

Important

Think about it for a sec! How would you mock an existing user?

6. Fix failing tests

Mocking existing users

We can mock existing users the same way we mocked "no existing users" - by mocking the objects manager. First, we'll enable the manager to hold аnd return a VotingUserEntity stub:

class VotingUserEntityObjectManagerMock:
    stub: VotingUserEntity | None = None

    def __init__(self, stub: VotingUserEntity):
        super().__init__()
        self.stub = stub

    def get(self, *_args, **_kwargs) -> VotingUserEntity:
        if self.stub is None:
            raise VotingUserEntity.DoesNotExist()
        return self.stub

You can probably see where this is going, right? All we need to add is a fixture builder:

@pytest.fixture
def given_voting_user():
    original_voting_user_entity_manager = VotingUserEntity.objects

    def _given_voting_user(
        user_id: UUID = uuid4(),
        karma: int = 10
    ):
        VotingUserEntity.objects = VotingUserEntityObjectManagerMock(
            VotingUserEntity(
                user_id=user_id,
                karma=karma
            )
        )
    yield _given_voting_user

    VotingUserEntity.objects = original_voting_user_entity_manager

7. Refactor BIG

Did you notice that we have already implemented a couple of behaviour units inside a Django View, i.e. an API adapter? At this point we can start refactoring the code by adding the necessary abstractions, service, and domain layers, and moving Django model interaction code to SPI adapters.

Note

This is where things get a bit controversial. Remember that tests are meant to guide development. Hence, we must drive the development of the application core via proper solitary and sociable tests!

We are able do these refactorings because our tests cut through all the application layers and verify the behavior on its edges.

The price to pay

The lightweight integration tests let us quickly test application as a whole, using mocks only at the system edges. However, these mocks can become quite sophisticated because a single use case triggered at the API often results in many downstream interactions. For example, imagine a scenario where the application returns HTTP 409 - Conflict, when someone votes for an article a second time. We'll need to mock User.objects (a Django DB model ObjectManager) to provide an existing user, as well as provide an existing article via a mocked Article.objects manager.

At some point, the mocks can become too sophisticated and create a mess of their own. That's when real integration testing alternatives like in-memory SQLite, or even Testcontainers start making more sense. We may loose the rapid feedback of unit tests, but save ourselves a lot of time spent on maintenance of messy test doubles.

Conclusion

Solitary unit tests are excellent for developing individual components in isolation. Sociable unit tests are a classical approach to testing pieces together, mocking the complicated dependencies, for example, those which cross the I/O boundary. Both solitary and sociable unit tests are my tools of choice for test-driven development.

Lightweight integration tests ensures that all components work together correctly as a complete application, while preserving the speed and flexibility of unit tests. However, we need to invest significantly in edge mocks to emulate rightmost interactions. Moreover, these tests primarily drive the implementation of application edges - the adapters.

Lightweight integration tests are a good alternative to either missing integration tests or integration tests which require an equally complicated setup. They can be quite useful when refactoring a legacy system, that doesn't have any tests. We can capture the interaction on the edges and ensure that it doesn't change during development.

Acknowledgements

Once again, Jarkko "jmp" Piiroinen nudged me to dive deeper and challenge my own understanding of the subject. I'm very grateful for your input, Jarkko!

Peopleware

2024-12-29T22:20:00+02:00

A new, shiny office. Polished desktops. Comfortable chairs. Floor-to-ceiling windows. A spacious kitchen. You are moving here to make history! A new project is about to start, and your organisation is going to expand. You have butterflies in your stomach. Is everything going to be all right? It feels like a good time to recap Peopleware - Productive Projects and Teams by Tom DeMarco and Timothy Lister.

Our industry, technological in nature, has far more sociological problems and challenges. The tar pit swallows yet another project, not because we couldn't figure out the technical parts, but because the people-oriented aspects had the lowest priority. Peopleware is about teams and individuals, about the people of the office - the "white collars". It's about bosses and employees, leaders and managers. It's about how, why, where, and what are our daily working activities about, how every bit of surrounding environment affects them.

Here is a hard truth about reviewing the book: I couldn't make it! There are 39 chapters in "Peopleware". And although they are grouped into six parts, and every chapter is rather short, the breadth of topics is enormous! The chapters are basically essays dedicated to a certain topic, and almost every chapter could be a book of its own!

If you think, that you know and have seen it all, did you give much though to the office environment? Did you know that people who program while listening to music easily miss obvious solutions? Or, how about hiring and letting people go? Is the cost of turnover as low as you see it? Do you remember to manage the obvious non-performance risks? What it takes to grow a jelled team, which can read each other's minds when working together? And many more questions for which Peopleware has answers backed up with studies and the enormous experience of the authors.

Every chapter is full of discoveries. I highly recommend this book for leaders, managers and employees alike, because Peopleware is about betters ways that concern us all.

If there was only a single advice from the book I could take, here it is:

Hire the right people.
Put them together into a fertile environment.
Give room for learning, for trial and error.
Help them grow towards company targets and their personal goals.
And stay out of the way :)

Testified Documentation

2024-09-29T14:47:00+03:00

The manual is outdated unless you are continuously building it alongside the application.

—Arho Huttunen

It's hard to sell a software product - whether it's a library, a program, or generally speaking, a system - if it doesn't come with a top-notch manual. In my opinion, a good manual should describe every aspect of the system, provide plenty of different high-level examples and lower-level API specifications. It includes a changelog with emphasised important and breaking changes, explains how to install, upgrades and maintain the system.

Keeping such documentation up-to-date is hard. Even when engineers who work on the system take care of the manual, they easily miss some tricky aspects or don't bother to document certain "obvious" behaviour. It's even harder for technical writers who work in silos and don't know much about the internals of the system, or don't fully understand the business case covered by the product feature they are documenting.

Specification by Example and Cucumber

Gojko Adzic proposed the concept of Living Documentation in his renowned book "Specification by Example". But Living Documentation doesn't come for free. For a JIRA ticket a new feature description to become an executable specification, one must first lay down a bridge between the human-friendly narrative and the machine.

For example: A fictional Flixnet movie streaming platform wants to notify the customers that their subscriptions are about to expire. After user mapping sessions, the stakeholders wrote down the following specification expressed in Gherkin/Cucumber syntax:

Feature: Remind a customer about an expiring subscription

  The customer might forget that their Flixnet subscription is expiring.
  We want to retain the customer by reminding them about the expiring subscription
  and providing an easy way to remain subscribed.
  We will notify the customer only once, 3 days before the subscription expires,
  so that the customer is not annoyed.

  Scenario: An active customer whose subscription expires in 3 days gets a notification
    Given an active customer
    And their subscription expires in 3 days
    And we have not sent them a reminder yet
    Then send the customer a reminder

How do we organise the document that incorporates this specification? Should it have a link to the specification? Or take some of its parts, like the feature name and description? Remember that Gherkin pays no attention to those, they are for the reader's convenience only. With some effort, it is possible to program docutils or MyST directive to include the specification into a Markdown file:

### User notifications and reminders

We send our users notifications on certain occasions. They are listed below:

```{specification}
:feature: Remind a customer about expiring subscription
:include-description:
```

...

Coupled with the ability to check specification execution (aka tests) results, this could provide a living documentation! It's quite close, to what Gojko Adzic described in his 10-years-later aftermath article written in 2020. He noticed, that with then-available breed of Given-When-Then tools, the most effective solution was converting the specification files into something nicer and easier to read and then publishing a read-only version somewhere outside the version control system.

The sad part is that specifications written in Gherkin and executed via Cucumber are largely abused. Aslak Hellesøy wrote in 2014 that Cucumber was largely misunderstood. Cucumber is first a formost a collaboration tool that aims to bring a common understanding to software teams - across roles. It is not a testing tool for QA to write end-to-end tests on an existing implementation. I witnessed horrible Cucumber misuse: specifications written weeks after implementation got integrated; Specifications which have only technical terms and describe implementation details instead of business rules; Specifications meaningless and repetitive, such as:

Given a user
When user logs in
Then user is logged in

In other words, no one benefits from including bad Gherkin specification details into a manual. And the tooling does not exist yet, anyway.

Testified Documentation

Back to square one - how else to ensure that the behaviour described in the release manual aligns with the released program behaviour?

A simple idea struck me in September 2024. What if we use something that is widely adopted by software engineers? That is - the output of automated tests: unit, behaviour, end-to-end - all of them that run on every build of the system.

Test automation tooling is widely available and is quite stable for popular programming platforms. Moreover, test runners usually support a common JUnit XML output format. For example, here is the formatted report of two executed smoke tests:

<?xml version="1.0" encoding="utf-8"?>
<testsuites>
  <testsuite name="pytest" errors="0" failures="0" skipped="0" tests="2" time="0.041" timestamp="2024-09-29T14:19:49.292476+03:00" hostname="z">
    <testcase classname="test.test_smoke" name="test_smoke" time="0.001" />
    <testcase classname="test.test_smoke.TestBigSmoke" name="test_big_smoke" time="0.000" />
  </testsuite>
</testsuites>

Now imagine that for every paragraph in documentation file, we include a directive that checks whether certain test cases passed successfully. The results are invisible for the end user, it's intended for developers and copywriters: if any of the required tests fails, documentation won't build.

It's nothing new though: some products already utilize testifying artifacts produced by test automation in documentation. For example, visual tests make screenshots, and the documentation embeds them as example images.

I quickly assembled a small PoC for Sphinx - a tool widely used to write manuals for Python libraries and programs. Here, I included a test name as a role (yes, it should have been a directive ;-):

This is a documentation for the "Spectest" library.

Spectest starts with a simple smoke test. :spectest:`test.test_smoke.test_smoke`.

The build reads test results from the XML above, passes successfully and outputs a Sphinx-based HTML manual. However, if we simulate a test failure by changing the verified name to :spectest:`test_ABCD`, the build breaks along:

1	/home/zaur/.../index.rst:13: ERROR: TEST test_ABCD not found [docutils]

That's the idea in a nutshell. To me it sounds technically simple enough to get implemented for any extensible documentation generator.

The devil is however, in the details. Testified documentation won't work well unless you have thorough behavior tests. That doesn't imply E2E tests at all! (I profess keeping those to a bare minimum.) Rather, have a suite of fast sociable tests that verify system behaviour and utilize stubs and mocks for external dependencies.

Feedback

Rainbows and unicorns :) I would love to hear your thoughts on this topic and include them here. I am a lucky guy who can share the wildest ideas with my colleagues and friends and get honest feedback.

Jarkko 'jmp' Piiroinen had a very valid and cool-minded point: there is zero incentive to make this kind of documentation-to-test-results bindings if the target audience doesn't care about full-fledged product documentation in the first place.

Seems to be the case, when a system is small enough to fit into product team members heads, and especially when the external interfaces are already documented in a form of specification, like OpenAPI spec.

Arho Huttunen reminded that Cucumber is not the only tool for Specification by Example and Living Documentation. Uncle Bob's FitNesse was there already in 2002! There are also Concordion, Serentiy, and most likely, others. We also discussed that for code examples in documentation, tools like Doctest which allow executing and testing their output during manual generation phase are even more useful.

The dark patterns of seat selection in Turkish Airlines

2024-08-17T20:44:00+03:00

Turkish Airlines is our first choice when it comes to flying to Azerbaijan from Finland. It’s not the cheapest, but the service is great. You get good food even in economy class, and there is an entertainment system with a broad selection of movies to keep you and the kids busy. The baggage allowance is huge - 30kg for check-in and the standard 8+kg for carry-on, although I’ve never seen them put the carry-on baggage on scales.

So, what’s all the fuss about?

Ticket prices gone wrong

It started in mid-spring when we were checking ticket prices for Helsinki-Baku flights. The figures were crazy, to say the least. Since we were also planning a summer trip to Estonia, we decided to check the price from Tallinn. Bingo! Twice as cheap. There are roughly 100 km between Tallinn and Helsinki (Vantaa) airports. The distance doesn’t justify it. Helsinki airport tariffs are certainly higher than Tallinn’s, but not that much! So, the good old supply and demand: if Turkish Airlines can sell the tickets for that price, why go lower? It’s summer season after all, and Finns are rushing to warm countries.

This ticket price talk is rather a rant. After all, the route is not monopolized; one can fly, for example, Air Baltic when on a strict budget. However, I wanted to emphasize that the tickets are not cheap, and considering all the services included, the dark patterns are not something you would expect.

Seat selection gets weird

For years, we had a default seat arrangement where either all of us sat together, or (after our youngest daughter was born) 3 + 1, making it possible for both kids to sit with either mom or dad.

But on the flight from Tallinn to Istanbul (our first leg), the seats were split as 2 + 2. This was on an Airbus 321 (or Boeing 737, I can’t remember exactly) where economy class has a 3 + 3 seat configuration.

Well, the kids have grown up enough to sit alone with either parent, so it’s truly not a big deal, right? But that was the first subconscious bell - have Turkish Airlines updated their algorithms?

Seat selection gets worse

Two days before going back to Finland, we got an email:

The all-caps subject is certainly emphasizing. What's inside?

Dear ZAUR NASIBOV

Your seat selection on the 10 AUGUST TK0339 Baku (GYD) -Istanbul,Istanbul (IST) has been changed to 11F due to operational reasons.

For details and seat transactions to your *** reservation, you may visit our MOBILE app / WEBSITE Air.tk/manage-booking.

Thank you for your undestanding.

Who said that I am undestanding? Because I was not understanding, neither I did want to. Upon checking the seats I saw this:

What the... Has someone reserved both D and E seats in row 10 or 11? Perhaps all the seat pairs and triplets are taken, and there is no way to accommodate us closer to each other? Let's check the rows at the tail:

So many unreserved seats! Yet, instead of keeping us together ten rows further to the tail, the algorithm decides to split us in the most annoying way.

Support can't do anything

I had some hope that support might help with the problem. Although caring, the operator could do nothing, since the seat reservations were not paid for - they were "randomly assigned". According to the operator, I could either pay for the desired seats or... request better seats while checking in at the airport.

Frequent fliers know too well that nowadays the chances of ending up at the very tail of a plane are increasingly high when checking in at the airport. Usually, passengers check in via airline apps hours before the flight to secure their seats. And just few years ago, selecting seats while checking in was free of charge, at least with Turkish airlines. Not anymore.

Five stages of grief

Twenty four hours before the flight, I am awake early to check in via the app. Sure, you can pick the seat! Only 14 Euros per passenger!

Denial, anger, bargaining, depression, and acceptance - all happening within a blink of an eye. It is not about the money; 14 Euros is what lunch costs in Finland. It was really a question of principle. First, they spread people from a group (read: family members) across the rows, and now they make us pay to get back together. And you know what? That’s ingenious! Because I paid that money so that we don’t end up scattered even more or at the last rows in the tail. Bravo, Turkish Airlines, you’ve discovered a dark well of gold!

Bad luck?

Perhaps I’m wrong. Perhaps it’s just bad luck. On the second leg of the journey, our seats were on the same row: B, C, D, E. Imagine my surprise when the neighboring passengers from seats A and F asked to change places because they were a mother and daughter flying together! (Note: This is a crafted screenshot; I didn’t take a screenshot of these seats back then).

Aftermath

Will I stop flying Turkish Airlines? Let’s see what next year brings :) As a passenger, I despise dark patterns like this and would rather pay extra from the start to have a pleasant experience, instead of getting “YOUR SEAT HAS BEEN CHANGED!11!!!11” emails.

Building Microservices - more than just another "microservices book"

2024-07-09T18:32:00+03:00

Finally! Hundreds of pages dedicated to microservices have been transferred to my brain. Even if only a tenth of what I read stays with me, then Sam Newman has definitely succeeded. "Building Microservices" is enormous - not just in size (there are certainly bigger books!) but in the vast range of topics it covers. What some people think of as "Microservices" might only be the summit of an enormous "MICROSERVICES" mountain.

This book is about the topography of that mountain. It doesn't delve too deeply into every aspect, but it provides an overview, gives directions, helps with navigation, highlights pitfalls, and suggests proven routes - all to ensure that you can reach the summit safely. Like any mountain climb, adopting microservices comes with decisions to make and trade-offs to take.

Newman starts the book with the fundamentals: What are microservices? How do we model microservices in relation to the problem we are solving (i.e., the product we are building)? How do we split an existing monolithic solution into microservices?

He then moves on to the technical aspects: communication, security, building and testing, monitoring, and even front-end development.

The last part is about us, the people who build microservices. Once again, we learn how important Conway's Law is, what the optimal organizational structure to support microservices development looks like, and the role of architects in all of this.

I was very excited to read the book. I loved that the author provided enough information to understand the topics and left breadcrumbs, referencing books, articles, and videos so that one can delve deeper and learn more. "Building Microservices" will find its audience among architects, novice and experienced backend developers alike. I highly recommend it.

Teams Topologies - Book Review

2024-03-23T21:42:00+02:00

Are you familiar with the exiting feeling of making a discovery? Butterflies in your stomach, intermittent breathing, galvanized palms? Every chapter draws a "WOW!", every page turns into a vivid picture in your imagination. That is "Team Topologies" - a book about systems architecture. This may come as a surprise, since the title clearly states otherwise. It's also a book about teams interactions and organizational design.

Part I - Teams As the Means of Delivery. Matthew Skelton and Manuel Pais dedicate the early chapters of the book to show how organizational structure affects the system design. This effect is nowadays known as "Conway's law", which originates from Mel Conway's 1968 paper "How Do Committees Invent":

Organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.

It is a scary discovery. It means that organizational structure prevails over system architecture, unless the structure is made to support the desired architecture. If the former and the latter are in conflict, one of them has to change. Which one? If the desired architecture is the target, there isn't much to change about it. This leaves us with the organizational structure.

But before diving into the structure, the organization needs to have team-first mindset. In the book, a team is a stable group of five to nine people who work toward a shared goal as a unit. The team is the smallest entity of delivery within the organization. An organization should never assign work to individuals; only to teams.

How can an organization improve team performance? The team should be long-lived and stable, of relatively small size (as stated above). The team should own the software while being able to limit its cognitive load. The organization should reward the team, not individuals. Last but not the least: the team members should develop a team-first mindset, prioritizing team goals and all required activities that help achieve these goals.

Organizations must design teams intentionally by asking these questions: Given our skills, constraints, cultural and engineering maturity, desired software architecture, and business goals, which team topology will help us deliver results faster and safer? How can we reduce or avoid handovers between teams in the main flow of change? Where should the boundaries be in the software system in order to preserve system viability and encourage rapid flow? How can our teams align to that?

The answers to these questions are found in Part II - Team Topologies that Work for Flow.

The authors start by examining team anti-patterns, such as frequently shuffling team members and ad-hoc or reactive team design. A team that has grown too large may be split without clear boundaries. A dedicated DBA team might be created after a software crash in production due to poor database handling. These situation certainly require action, but "the most natural solution" might slow down delivery and weaken the autonomy of teams.

In order to be as effective as possible, we need to consciously design our teams rather than merely allow them to form accidentally or haphazardly. We call these consciously designed team structures team topologies, a term that acknowledges that teams should be deliberately “placed” into organizations while also referring to the boundary of responsibility of each team.

A high-performing team takes responsibility and the ownership of the features they develop - from design all the way to production. Here, production means monitoring feature usage and performance, rather than handing it over to an Ops team. Such a team is cross-functional and equipped with all necessary skills to design, develop, test, deploy and monitor application usage and performance. To maintain a rapid flow, the team remains autonomous, ideally utilizing non-blocking external dependencies in the form of self-services developed and maintained by other teams.

Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.

—Antoine de Saint-Exupéry, Airman's Odyssey

Skelton and Pais reduce the number of team variations to four fundamental team topologies:

Stream-aligned team
Enabling team
Complicated-subsystem team
Platform team

The authors argue that these are the only topologies necessary to build and run modern software systems. Limiting to these four team types acts as a powerful template for effective organization design, especially when combined with well-defined software boundaries and efficient team interactions.

The stream-aligned team is the primary team type in an organization, and the purpose of the other fundamental team topologies. A stream represents the continuous flow of work aligned to a business domain or organizational capability. The team is empowered to build and deliver customer or user value as quickly, safely, and independently as possible.

The enabling team assists stream-aligned teams in acquiring necessary skills and knowledge, setting up pipelines, adopting tools and solving engineering problems in unfamiliar environments. Enabling teams have a strongly collaborative nature; they thrive to understand the problems and shortcomings of stream-aligned teams in order to provide effective guidance.

A complicated-subsystem team is responsible for building and maintaining a part of the system that relies heavily on specialized knowledge. This is to such an extent that most team members must be specialists in that particular area to understand and maintain the subsystem effectively. Examples of complicated subsystems might include a mathematical model, a mainframe module, a voice recognition algorithm, and so on.

The purpose of the platform team is to enable stream-aligned team to deliver their work autonomously. It provides internal services to reduce the cognitive load that would otherwise be required from stream-aligned teams to develop these underlying services. Examples of these services include: continuous integration (CI), code registries, repository hosting, databases, deployment, monitoring, logging aggregation and so on.

Based on reports from successful organizations, the ratio of stream-aligned teams to other types of teams should be between approximately 6:1 and 9:1.

How should stream-aligned teams divide responsibilities? The authors advice breaking down the system, so that the resulting parts can evolve as independently as possible. Consequently, teams assigned to these parts will experience greater autonomy and ownership over them.

It is also essential to consider the teams dynamics when splitting a monolith. Splitting a monolithic application into autonomous services that need to be released and deployed together is merely creating a "distributed monolith" with tightly coupled pieces. Microservices sharing a database, monolithic world models with similar domain language and representation across different formats, or even a single office layout for all the teams, can all contribute to a distributed monolith.

Now that the reader is familiar with the four fundamental team topologies, Skelton and Pais discuss how teams interact and when to change the teams and their interactions. Well-defined interactions are key to effective teams, and the authors outline three essential ways teams can and should interact:

Collaboration: working closely together with another team. Imagine two or more teams coming together to solve a problem that requires competences from all of them. Their collaboration is rapid; they quickly make discoveries and share knowledge, experiment and prototype solutions.
X-as-a-Service: consuming or providing something with minimal collaboration. This interaction is ideal when a team needs to focus on its tasks and consume the necessary services - from CI/CD infrastructure to upstream production API services - without being involved in their development and maintenance.
Facilitating: helping (or being helped by) another team to clear impediments. When a team lacks certain competences, a facilitating team can save the day by coaching and providing help. This is the main interaction mode for Enabling teams.

The last chapter of the book - Evolve Team Structures with Organizational Sensing.

As Frederic Brooks mentioned in "Mythical Man-Months":

Conway goes on to point out that the organization chart will initially reflect the first system design, which is almost surely not the right one. If the system design is to be free to change, the organization must be prepared to change.

The software design must align with the way in which teams communicate; otherwise, it is doomed to fail. The methods of communication evolve as the system and the organization matures. What begins as collaboration eventually becomes part of platform services with strict boundaries. Stream-aligned teams seek help from Enabling teams when necessary or provide assistance to other teams upon request.

I tried very hard to distill the essential ideas of the book into this small review to get you interested in reading it. As you can imagine, there is so much more! If you are ever involved in designing systems architecture that requires the interaction of many teams, this book is a must-read.

The Mythical Man-Months - thoughts

2024-03-19T21:42:00+02:00

As the year 2023 was coming to an end, I finally had some time to recap "The Mythical Man-Month: Essays on Software Engineering" by Fred Brooks. The book talks about software development fallacies from software engineering and project management perspectives. However this is not "yet another book". It was first published in 1975, nearly fifty years ago! So much changed since 1975, and yet, so little! It's unbelievable that today we are still struggling with the problems that haunted us in the age of mainframes.

You smile a lot as you read the book. Programming with punch cards? A big fat machine with only 2 MB of RAM? Software documented on paper? Sure, things were different back then. But there are so many similarities! Instead of reviewing the book, I'd prefer to study these similarities, draw parallels, and find out whether things have changed for the better.

1 - The Tar Pit

Programming is joyful. Every time our program behaves as we intended, it's like a small masterpiece coming to life. However, programming is also full of woes. How many programs - big and small, beautiful and ugly, useful and useless - have been sucked into a tar pit? Initially, it may not be apparent; after all, a creature can pull out one of its paws. But what if it's all four paws are stuck? Therefore, we should learn to avoid such tar pits.

How painfully true! Even in my relatively short career (14 years and counting), I've witnessed software systems - both small and large - struggling under the weight of their features, unable to escape the tar pit and eventually fading away. That's precisely why we begin with Proof of Concepts (PoCs) and Minimal Viable Products (MVPs) and develop systems iteratively. This is why we continuously evolve software architecture - to ensure it remains resilient and elastic - and prevents getting into the tar pit.

2 - The Mythical Man-Months

The second chapter explains why measuring a job in "man-months" is inappropriate for software development.

Men and months are interchangeable commodities only when a task can be partitioned among many workers with no communication among them. This is true of reaping wheat or picking cotton; it is not even approximately true of systems programming.

And yet, we continue measuring software development in "story points" - a measure of task complexity. We even play "agile poker" to determine a reasonable amount of story points. Our teams digest fewer story points when someone is on vacation or sick leave, and more story points when everyone is present. When story points seem too abstract, we convert them into days and hours using arbitrary logarithmic scale. For instance, 1 story point represents couple of hours of development, while 8 story points correspond to about a week of work for a seasoned developer... But wait a second, aren't we back to man-months again?!

Are you playing agile poker? Consider using these accurate estimation cards for the next round:

3 - The Surgical Team

In the third chapter Brooks discusses team efficiency based on size and structure.

For efficiency and conceptual integrity, one prefers a few good minds doing design and construction. Yet for large systems one wants a way to bring considerable manpower to bear, so that the product can make a timely appearance.

How wild does a 200-man team sound? Brooks talks of ever bigger teams - 1000 people working on OS/360. And it's not just programmers but writers, machine operators, clerks, managers and more. Even a brilliant, but small team is not able to tackle work that requires the collective effort of hundreds of minds.

Brooks refers to a proposal by Harlan Mills, where the chunks of work are split between surgical teams. A surgical team includes people in different roles, much like a real surgical team that performs medical operations. There roles include: the surgeon, the copilot, the tester and the toolsmith, the language expert and the editor (i.e. a technical copywriter), the administrator and the secretaries.

Interestingly, you can easily map all these roles to the roles in modern software development teams: Product Owner (PO), team lead, engineer, tester. While some of the roles have been automated and become unnecessary in our age, the essence of the idea - that collaboration and specialization are crucial - remains relevant:

10-man team can be effective no matter how it is organized, if the whole job is within its purview.

4 - Aristocracy, Democracy and System Design

The chapter begins with a photo of the Reims Cathedral, which unlike some of European cathedrals, exhibits sound design and architectural integrity.

The guidebook tells, this integrity was achieved by the self-abnegation of eight generations of builders, each of whom sacrificed some of his ideas so that the whole might be of pure design.

Conceptual integrity is equally important for software systems as for buildings. Think of Unix with its small programs that each excel at a specific task. Can you imagine a "jack of all trades and the master of none" among them? Probably not, because it wouldn't align with the Unix philosophy and wouldn't be accepted by the Unix community. Or imagine a bunch of connected microservices, each written in a different language, deployed across various clouds, collaborating via non-compatible protocols. Such an architecture is a maintenance hell destined to fail.

Conceptual integrity implies that design comes from a limited set of minds.

Brooks argues that to achieve conceptual integrity, the design of the architecture (i.e. the interfaces of the system) must be separated from its implementation. This doesn't mean that the architects are "aristocracy" who make all the decisions. Rather, the architects decide "what should be done", while implementers decide the "how".

5 - The second-system effect

The second is the most dangerous system a man ever designs. When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.

The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one.

I believe that overengineering can happen at any stage of one's journey. The fundamental concepts of our field are stable, compared to all the hyped frameworks, tools, programming languages and processes, that claim to solve the issues of their predecessors.

In my experience, the most skilled engineers create the solutions that are simple, elegant and efficient.

6 - Passing the word

How can a group of 10 architects maintain the conceptual integrity of a system which 1000 men are building? This chapter is a perfect example of eternal problems on one hand and technical progress transforming (but not completely eradicating!) the others.

How did people tackle this problem on the verge of 60s and 70s? They used

Written specifications - the manuals.
Formal definitions - API specifications.

I think we will see future specifications to consist of both a formal definition and a prose definition.
Direct incorporation - #include of function definitions in C, or import in Java to ensure correct function signature usage in compile-time.
Conferences and courts - the communities of practice.
Multiple implementations - The definition will be clearer, if there are at least two implementations in the beginning (web standards, compilers etc.)

In most computer project there comes a day when it is discovered that the machine and the manual don't agree. When the confrontation follows, the manual usually loses, for it can be changed far more quickly and cheaply than the machine.
Telephone log - Brooks advises architects to keep recordings of every question and answer, combine and distribute those on a weekly basis - in paper format! We have better tools (no, Confluence is not a better tool!) for this nowadays, and a well structured and up-to-date Q&A log is quite handy!
Product test - we tend to shift left in testing strategies. Software built with good test-first practices and automated integration testing has a solid foundation, yet manual exploratory testing can reveal so much! Independent testing is also necessary to have get the feedback and perspective of the people who were not involved in the development.

The project manager's best friend is his daily adversary, the independent product-testing organization.

Every development organization needs such an independent technical auditing group to keep it honest. In the last analysis the customer is the independent auditor. In the merciless light of real use, every flaw will show. The product-testing group then is the surrogate customer, specialized for finding flaws.

7 - Why did the tower of Babel fail?

This chapter discusses communication, or rather synchronisation problems. Brooks emphasizes the need of an up-to-date workbook - a structure imposed on the documents that the project will be producing.

All the documents of the project need to be part of this structure. This includes objectives, external specifications, interface specifications, technical standards, internal specifications, and administrative memoranda.

Sounds like our issue trackers, rich version control systems (think Github, Gitlab etc.), project wikis (Lord help to keep them in harmony!)

How hard was it? Brooks talks of keeping a workbook on paper - which became 5 feet (~ 1.5m!) thick and daily change distributions consisting of 150 pages! Microfische was a huge step forward which according to Brooks saved millions of dollars on wasting paper.

With today's system technology available, I think the technique of choice is to keep the workbook on the direct-access file, marked with change bars and revision dates. Each user would consult it from a display terminal. A change summary, prepared daily, would be stored in LIFO fashion at a fixed access point. The programmer would probably read that daily, but if he missed a day he would need only read longer the next day.

On a smaller scale - a good CHANGELOG is a crucial part of any system's API.

8 - Calling the shot

How long will a system programming job take? How much effort will be required? How does one estimate? Half a century later, we are still bad at answering these questions.

In the past, a common measure of system complexity was the number of instructions it contained. The chapter is mostly dedicated to that topic, which is quite outdated by today's standards. However, we can still draw parallels, for example,the effort required to create and maintain a system grows exponentially with its size:

compare to the "Cost per line of code" chart from "Clean Architecture":

9 - Ten Pounds in a Five-Pound sack

Talks of the costs we tend to often forget.

Consider the IBM APL interactive software system. It rents for $400 per month and, when used, takes at least 160K bytes of memory. On a Model 165, memory rents for about $12 per kilobyte per month. If the program is available full-time, one pays $400 software rent and $1920 memory rent for using the program.

Projecting this to the present day, a t3.2xlarge EC2 AWS instance with 32GiB of memory costs $0.333 per hour. This is roughly $0.00000714 per KiB per month - almost 1.7 Million times cheaper than Model 165 memory usage costs. This insane leap took half a century, and we are not slowing down. However, our systems have evolved as well. They grew in complexity, now serving millions of customers concurrently, streaming high-quality video, and processing real-time AI tasks. Consequently, despite more affordable hardware and broader network bandwidth, we still have to pay the price.

It’s a timeless truth: we've got to keep an eye on our budgets and watch how we're using our resources. Whether it’s CPU, memory, network bandwidth, storage, file sizes, startup time - you name it - we need to track it all.

Finally the following advice caught me off-guard, but then I recalled the world of embedded programming, where it makes total sense:

Every project needs a notebook full of good subroutines or macros for queuing, searching, hashing, and sorting. For each such function the notebook should have at least two programs, the quick and the squeezed.

10 - The Documentary Hypothesis

The documentary hypothesis states that

Amid a wash of paper, a small number of documents become the critical pivots around which every project 's management revolves. These are the manager's chief personal tools.

Brooks presents examples of documents from various projects and discusses the necessity of formal documentation. He states that the least questions these documents should address are: what, when, how much, where, and who. Take a software project, for instance:

What: objectives and product specifications
When: schedule
Where: space allocation
How much: budget
Who: organization chart - more on this one just in few paragraphs

Why have formal documents? Brooks emphasizes three matters:

Writing the decisions down is essential. It's only when we write things down, that the gaps and inconsistencies become evident.
The documents communicate the decisions to others.
Manager's documents provide them with a database and a checklist. Periodic reviews allow managers to assess their current status and determine necessary changes in focus or direction.

Brooks mentions Conway's law when talking of organization structure that will work on the software project.

Conway goes on to point out that the organization chart will initially reflect the first system design, which is almost surely not the right one. If the system design is to be free to change, the organization must be prepared to change.

Indeed, Team Topologies teaches us that we need to arrange teams according to the desired architecture and design, not vice versa. As the design and architecture evolve, so should the team topologies involved in product development.

To some extent, everything discussed in this chapter remains relevant to project management today.

11 - Plan To Throw One Away

So often we conflate the purpose and meaning of “Proof of Concept” with that of a “Minimal Viable Product”. A team working on a PoC might be forced to carry on with it to production, rather than starting from scratch and properly building the system from the ground up, applying the insights acquired during the PoC phase. There is nothing new under the Sun:

The management question, therefore, is not whether to build system and throw it away. You will do that. The only question is whether to plan in advance to build a throwaway, or to promise to deliver the throwaway to customers. Seen this way, the answer is much clearer. Delivering that throwaway to customers buys time, but it does so only at the cost of agony for the user, distraction for the builders while they do the redesign, and a bad reputation for the product that the best redesign will find hard to live down.

There is so much wisdom in these words. A PoC is a tool to test a business hypothesis and should be considered a throwaway. Delivering the PoC as a finished solution to the customer always backfires: developing on a weak foundation eventually leads to a collapse.

The next topic is - The Only Constancy Is Change Itself.

Both the actual need and the user’s perception of that need will evolve as programs are developed, tested, and utilized. The "throw-one-away" concept is essentially an acknowledgment that learning leads to changes in design. That’s precisely why we must plan the system for change.

The ways of designing a system for each change are well known and widely discussed in the literature perhaps more widely discussed than practiced.

Isn't it painful? We understand the need to design and implement and the system, keeping the need for change in mind. Yet so many systems eventually collapse under their own weight! Making changes to these systems becomes more expensive than not touching the systems at all.

Should we plan for change only in software systems? Brooks answers this by referencing J. Cosgrove's paper "Needed: a new planning framework", where Cosgrove advocates for treating all plans, milestones, and schedules as provisional to facilitate change. Although the section is only four pages long, it contains ideas that have been extensively discussed in "Accelerate", "The DevOps Handbook", "Team Topologies" and other studies. These include continuous learning and experimentation across various domains, team-first thinking, effective communication, facilitating (surgical) teams, the importance of leadership, and more.

Finally, Brooks discusses system maintenance, which he defines as the period following a program’s installation for a customer:

The total cost of maintaining a widely used program is typically 40 percent or more of the cost of developing it. Surprisingly, this cost is strongly affected by the number of users. More users find more bugs.

Software-as-a-Service has transformed the landscape such that software may never enter a traditional maintenance phase in its entire lifetime. However, without a solid architecture and a team committed to continuous improvement, an increasing amount of time will be devoted to bug fixing, to the point where a “complete rewrite” becomes a more viable option.

Theoretically, after each fix one must run the entire bank of test cases previously run against the system, to ensure that it has not been damaged in an obscure way. In practice such regression testing must indeed approximate this theoretical ideal, and it is very costly.

Today, regression testing is cheap. We have fantastic tools to fulfill the test pyramid. Test-first development of software has become the standard. Each modern programming language and environment comes with a handful of unit testing frameworks and libraries. Tools like Docker and Testcontainers enable us to set up a real database with a actual message brokers for integration testing. Even end-to-end (E2E) tests can be automated.

12 - Sharp tools

From laptops, IDEs, CI services, and Source Code Management systems to phones, circuit boards for debugging, and even the comfort of office chairs - we rely heavily on our tools. Nowadays, tools are generally much more affordable compared to personnel costs. Any savings made by not acquiring the best tools often result in losses due to slower market entry. However, tools weren’t always so affordable, particularly in the early days of computing.

Brooks discusses the machines required by development teams, which needed at least 1 MB of main storage and 100 MB of additional online disk storage, along with terminals. There were other constraints as well. Remember that machine time was a costly commodity? This led to the scheduling of machine access among developers, often limiting them to a few 20-minute sessions during a workday.

Another common challenge was that the target computers for the final product were not the same ones developers used to create the systems. Either the target machines were too expensive, or they weren’t available yet. Brooks also touches on machine simulators, performance simulators, documentation systems, and, finally, high-level programming languages and interactive programming!

In my view, the jewel of this chapter is the discussion on the Segregation of Duties (SoD) in the software release process. If you ever experienced a heavy, multi-gate release processes keep in mind that it was not invented recently!

When a man had his component ready for integration into a larger piece, he passed a copy over to the manager of that larger system, who put this copy into a system integration sublibrary. Now the original programmer could not change it, except by permission of the integration manager. As the system came together, the latter would proceed with all sorts of system tests, identifying bugs and getting fixes. From time to time a system version would be ready for wider use.

Then it would be promoted to the current version sublibrary. This copy was sacrosanct, touched only to fix crippling bugs. It was available for use in integration and testing of all new module versions.

13 - The Whole and The Parts

How does one build a program to work? How does one test a program? And how does one integrate a tested set of component programs into a tested and dependable system? These topics were briefly touched in the book, in this chapter Brooks considers them more systematically.

Brooks begins by emphasizing the significance of Conceptual Integrity in system architecture. He then advocates for testing the Specification before any implementation starts. Consider clients testing a platform API solely based on specification, prior to the platform's implementation being ready. This approach can reveal numerous issues, such as unclear or missing specification pieces, API inconsistencies, lack of documentation etc., all of which may arise during the implementation of client-to-platform communication.

Top-Down design - a method formalized by Niklaus Wirth in 1971 paper - in Brook's opinion, was the most significant new programming formalization of the decade:

Briefly, Wirth's procedure is to identify design as a sequence of refinement steps. One sketches a rough task definition and a rough solution method that achieves the principal result. Then one examines the definition more closely to see how the result differs from what is wanted, and one takes the large steps of the solution and breaks them down into smaller steps.

From this process one identifies modules of solution or of data whose further refinement can proceed independently of other work. The degree of this modularity determines the adaptability and changeability of the program.

Wirth advocates using as high-level a notation as is possible at each step, exposing the concepts and concealing the details until further refinement becomes necessary.

This is a simple and yet a very effective procedure. We can prevent implementation details from impacting architecture, and concentrate on the interaction between its components - starting from the grand scale all the way down to the details. Sounds familiar? That's right, it's Clean Architecture for you.

As I write these lines, another though strikes me: design comes second. It follows the gathering the user stories, the mapping of events, and identification of the needs of all involved stakeholders. Unless we get that right, the system won't address the problems of its end-users, no matter how impeccable the design is.

We've focused a lot on design, but what else does Brooks mention in this chapter?

Structured programming: on the verge of the 60s and 70s, programmers commonly controlled the flow with GOTO statements. Brooks recommends the use of do/while loops and if-then-else statements instead. While we have made significant progress in programming languages development, we still employ these simple program building blocks. Moreover, loops and conditional statements form the basis of syntactic sugar provided by modern programming languages.

The next topics are about debugging, which wasn't straightforward at all:

The programmer carefully designed his debugging procedure planning where to stop, what memory locations to examine, what do find there, and what to do if he didn't. This meticulous programming of himself as a debugging machine might well take half as long as writing the computer program to be debugged.

Just reading about these challenges makes you appreciate the simplicity of setting a breakpoint and using a rich interactive debugger.

At the end of the debugging section, Brooks discusses what we now refer to as test doubles:

One form of scaffolding is the dummy component, which consists only of interfaces and perhaps some faked data or some small test cases. For example, a system may include a sort program which isn't finished yet. Its neighbors can be tested by using a dummy program that merely reads and tests the format of input data, and spews out a set of well-formatted meaningless but ordered data.

As previously mentioned, modern programming provides fantastic tools to design and execute tests and test doubles. Test-first development is not a luxury, but a necessity that ensures code maintainability as it grows.

Brooks concludes by addressing the control of changes, also known as Segregation of Duties (SoD), advocating for the addition of one component at a time and the quantization of updates. This involves releasing in small batches and conducting thorough regression testing.

What tools assist us in these tasks nowadays? Version Control Systems, test-first development with comprehensive unit and integration testing, continuous integration and continuous delivery, and the strategic launch of new features through dark launches and canary releases.

14 - Hatching a catastrophe

This chapter could be perfectly illustrated by the following meme:

Well, developers are not always the guilty party, are they?

When one hears of disastrous schedule slippage in a project, he imagines that a series of major calamities must have befallen it. Usually, however, the disaster is due to termites, not tornadoes; and the schedule has slipped imperceptibly but inexorably.

Yesterday a key man was sick, and a meeting couldn't be held. Today the machines are all down, because lightning struck the building's power transformer. Tomorrow the disk routines won't start testing, because the first disk is a week late from the factory. Snow, jury duty, family problems, emergency meetings with customers, executive audits the list goes on and on. Each one only postpones some activity by a half-day or a day. And the schedule slips, one day at a time.

Brooks asserts that the optimal way to manage a large project is to establish a schedule with defined milestones. Each milestone must be concrete, specific, and measurable.

It is more important that milestones be sharp-edged and unambiguous than that they be easily verifiable by the boss. Rarely will a man lie about milestone progress, if the milestone is so sharp that he can't deceive himself.

This advice is universally applicable, whether in software development or room cleaning. For instance, a ticket stating “Make the server respond faster” is inadequate. What does “faster” mean in this context? Which responses are deemed slow? A more precise ticket would be “Reduce the response time for Resource API endpoints from 500ms to a maximum of 250ms.”

Brooks praises PERT charts for visualizing the dependencies between activities necessary to complete the project and identifying the critical path - the sequence of interdependent activities that take the longest to complete. It also indicates how much an activity can be delayed before it impacts the critical path.

The preparation of a PERT chart is the most valuable part of its use. Laying out the network, identifying the dependencies, and estimating the legs all force a great deal of very specific planning very early in a project. The first chart is always terrible, and one invents and invents in making the second one.

I have no personal experience with PERT charts, so I cannot comment. However, I’m interested in your thoughts on the matter!

Finally, Brooks discusses human behavior in scenarios where a schedule begins to fall behind. Please indulge me not making commenting on this too, Instead, I suggest reading "Tribal Leadership" and "Turn the Ship Around!".

15 - The other face

What is the other face of a program? According to Brooks, it’s the documentation, whether it’s for the user or a technical manual.

Pause for a moment and consider the year 1970. There was no StackOverflow, no Internet, and not even a terminal for every programmer. Without comprehensive documentation accompanying the program, it would be virtually useless.

But even back then, Brooks advocates shipping a program with tests!

Every copy of a program shipped should include some small test cases that can be routinely used to reassure the user that he has a faithful copy, accurately loaded into the machine.

Then one needs more thorough test cases, which are normally run only after a program is modified. These fall into three parts of the input data domain:

Mainline cases that test the program's chief functions for commonly encountered data.

Barely legitimate cases that probe the edge of the input data domain, ensuring that largest possible values, smallest possible values, and all kinds of valid exceptions work.

Barely illegitimate cases that probe the domain boundary from the other side, ensuring that invalid inputs raise proper diagnostic messages.

Nowadays, we seldom need to verify that a faithful copy has been accurately loaded into the machine, but a regression test suite remains absolutely essential - both during development and upon the delivery of a program.

A well-designed test suite also acts as documentation of a program’s capabilities. A specification by example, with automated tests included, precisely documents what a program does.

Brooks recommends including a technical overview of the program to facilitate easy modification. This should encompass a flow chart, descriptions of the algorithms used, explanations of file layouts, and even the pass structure - the sequence in which programs are transferred from tape or disk. Finally

A discussion of modifications contemplated in the original design, the nature and location of hooks and exits, and discursive discussion of the ideas of the original author about what modifications might be desirable and how one might proceed. His observations on hidden pitfalls are also useful.

All this reminds of ARCHITECTURE.md.

The next topic Brooks emphasizes is Self-Documenting Programs. As stated in the book, the adoption of high-level languages was not mainstream at all. Programs were written in machine codes, which is (sarcastically speaking) intended for machines, not for human comprehension. However, even with machine code, it was possible to use human-friendly symbolic names, labels, declaration statements, etc. Comments were just as important. Just have a look at this listing, which carries the unmistakable aroma of the "old school" days:

Brooks uses this listing as an example and explains the techniques used to document it.

Today, we still struggle with writing self-documenting code. “Clean Code,” “Code Complete,” “The Art of Readable Code,” “A Philosophy of Software Design” wouldn't exist if high-level languages had solved all the problems. Engineers still name variables a, b, c. Software is still written without any tests. Specifications still come in the form of poorly-written JIRA tickets, which look nothing like actual specifications. Oh, I’m ranting again. :)

17 - No Silver Bullet

"No Silver Bullet - Essence and Accident in Software Engineering" (1986) is perhaps the most renowned essay by Brooks. It was included in the anniversary edition of the Mythical Man-Months, republished in 1995. The paper's opening statement immediately takes you to the point:

There is no single development, in either technology or management technique, which by itself promises even one order-of-magnitude improvement within a decade in productivity, in reliability, in simplicity.

Brooks differentiates between essential and accidental complexities in the software development process.

Essential complexity comes from the abstract nature of software's building blocks: algorithms, data sets and their relationships. Yet, they must be highly precise and richly detailed. There are countless ways to write a "Hello world" program. What is to be said of anything more complicated?

Brooks suggests attacking essential complexity by reducing productivity constraints. He identifies several promising strategies: Buy versus build, Requirements refinement and rapid prototyping, Incremental development - growing, not building, software, and Employing great designers. All these ideas have stood the test of time and are parts of modern software development.

Accidental complexity comes from limitations of tools, processes, hardware and the environment. For instance, high-level languages alleviate the accidental complexity of assembly language. Iterative software development prevents us from committing to extensive upfront designs, only to discover later that what we've released doesn't meet the customer needs. Having more computing power in our pockets than we had in our desktops just a few decades ago enables the use of more sophisticated algorithms that now run in a reasonable time frame.

Brooks evaluates technical advancements that have been praised as potential silver bullets: Ada and other high-level language advances, Object-oriented programming, Expert systems, Graphical programming, Program verification, Environments and tools, Workstations, finally AI and Automatic programming.

Looking at the latter through the lens of 2024 - Generative Pre-Trained Transformers (GPTs) and Large Language Models (LLMs) are flourishing. We can interact with these tools in natural language and receive code with explanations in return. The interaction doesn't stop there; these tools can maintain context, allowing us to refine queries for more targeted and sophisticated results. Is this sort of automatic programming a silver bullet? In my opinion - no, at least not yet.

Now, don't get me wrong, we can throw a colossal amount of computing power to build a vast, boundless model, capable of holding an entire Linux kernel with all its development history in the context. But the output is still intended for human interpretation. How much can we trust the code generated by such LLMs? How do we verify its correctness? Would we use another LLM to generate tests, or would we craft tests in the traditional way? Let's get back to this discussion in five years - it's going to be interesting!

16 - EPILOGUE

The tar pit of software engineering will continue to be sticky for a long time to come. One can expect the human race to continue attempting systems just within or just beyond our reach; and software systems are perhaps the most intricate and complex of man's handiworks. The management of this complex craft will demand our best use of new languages and systems, our best adaptation of proven engineering management methods, liberal doses of common sense, and a God-given humility to recognize our fallibility and limitations.

The DevOps Handbook - review

2023-08-11T17:40:00+03:00

I have started writing this review by describing the contents of each chapter in-depth, just to realize that it's hardly possible to do that without repeating the whole book. I usually put some conclusions at the end of a book review, but this book is different.

This is a short and emotional review of the DevOps Handbook: How to create world-class agility, reliability, & security in technology organizations.

For developers, programmers, craftsmen, and engineers, testers and architects, operations and security overseers, product managers, leaders, and many, many more - to all the exceptional IT professionals out there.

This is the book — the book that serves as a reminder of many things.

It reminds us that programmers are not mere machines that convert assignments into code. It underscores that QA professionals are not solely gatekeepers whose task is to extract bugs from the code programmers produce. It highlights that Operations personnel are more than overseers of programs functioning in production. It emphasizes that excelling as Project Managers is not about inundating products with requested features from customers. The book unites us in our shared objectives and reminds that ultimately value is created only when our customers can effectively utilize our product.

This is the book - the book that imparts to us the principles of flow, feedback, and continual learning and experimentation within the technology value stream.

The First Way: The Principles of Flow encompass the guidelines which help us to accelerate the progress of code, from the moment it was committed to the version control system, all the way until the customer is able to interact with it in a form of a software product.

The Second Way: The Principles of Feedback - are the guidelines that assist us in staying on the correct path. Are we truly addressing customer issues? Are we doing so efficiently? Is security ensured? Can we manage substantial customer loads? Are we addressing issues as they arise? Measure. Act. Analyze. Take action based on findings. Iterate. Apply this approach to every process, every hypothesis, and every segment of delivered software functionality.

The Third Way: The Principles of Continual Learning and Experimentation comprise the guidelines that aid us in maintaining a cutting-edge position and foster the creation of knowledge, both at an individual and organizational level.

The book begins by building a solid foundation out of these principles. It continues in Part II by discussing DevOps transformation, its starting points and ways to learn and experiment without jeopardizing the rest of the organization.

In Part III the author discuss the technical practices and architecture required to enable and sustain the fast flow of work from Development into Operations without causing chaos and disruption to the production environment or customers. These practices are commonly knows as Continuous Delivery.

In Part IV the authors discuss how to implement the technical practices of the Second Way, which are required to create fast and continuous feedback from Operations to Development.

In the last part - Part V, the book presents the practices that create opportunities for learning, as quickly, frequently, cheaply, and as soon as possible. This includes creating learnings from accidents and failures, as well as organizing and designing our systems of work so that we can constantly experiment and learn, continually making the systems safer.

Many of us have experienced some of these practices. I was fortunate in my career to grasp and adhere to the principles of flow and the principles of continual learning. However, I never had the opportunity to properly experiment or establish feedback mechanisms for such experiments, and I am still eagerly anticipating that chance.

This is the book. Get it. Read it. Embed everything you read in your daily work. Your colleagues, company and above of all, your customers will be grateful.

The curious case of denied repository access in Github Actions workflow

2023-06-01T22:59:00+03:00

GitHub Pages is a fantastic static site hosting service that takes HTML, CSS, and JavaScript files straight from a repository on GitHub. It is available for free on public repositories and allows configuring a custom domain for the site. The site updates just in few moments after git push. Therefore, when it comes to choosing a platform for hosting a static blog, opting for GitHub is an obvious choice.

There are three types of GitHub Pages sites: project, user and organisation. To publish a user site to https://<username>.github.io, you have to create a repository with the same name. By default, the site is served from the master branch and the root of the repository.

My blog is written in reStructuredText (reST) format and compiled into a static site via Pelican. I keep the source of the blog in a separate private repository called (surprise!) "blog." The built blog site is deployed to basicwolf.github.io via force-pushing, so there is only a single branch with a single commit in the repository. The whole setup allows me to hide intermediate work and expose only the end result to the public.

For years I've been manually publishing the blog from my computer - a paradoxical situation considering my longstanding advocacy for continuous integration and delivery. Even my other hobby projects incorporate a CI/CD pipeline in some form or another. Finally, the time has come to set up a pipeline for the blog, and fortunately, in the year 2023, GitHub Actions have shown to be highly effective and efficient in getting the job done.

I set up the pipeline as follows: when a push occurs on the master branch of the blog repository, an action is triggered. This action builds the website, commits the resulting changes to the gh-pages branch, and pushes the branch to the basicwolf.github.io:master repository.

Publishing from blog repository to basicwolf.github.io repository

At this point, things became a little complicated. The workflow runs within the blog repository context but pushes the results (i.e. the site) to the basicwolf.github.io repository. To accomplish this I had to utilize a dedicated personal access token (PAT) for the second repository, granting it Read and Write access to code permissions. The token was subsequently utilized in the following manner:

git push -f https://basicwolf:${MY_TOKEN}@github.com/BasicWolf/basicwolf.github.io.git gh-pages:master

I generated a PAT and tested it by pushing the site from my computer. Everything went smoothly, and I was looking forward to quickly complete the workflow setup. Little did I know that I would be banging my head against the keyboard for two days, trying to comprehend git push permissions denials.

I pushed the workflow file and began monitoring its maiden voyage. Everything was running smoothly until the final step - pushing the site to basicwolf.github.io:

git push -f ***github.com/BasicWolf/basicwolf.github.io.git gh-pages:master
remote: Permission to BasicWolf/basicwolf.github.io.git denied to github-actions[bot].
fatal: unable to access 'https://github.com/BasicWolf/basicwolf.github.io.git/': The requested URL returned error: 403

I won't burden you with the details of verifying that the token remained intact during the git push process. Whether it involved checking its length, echoing token fragments, or generating new tokens, the token itself was sure present in the workflow. However, despite its presence, access was denied when attempting to push from the workflow run, while it was granted when pushing from my computer.

I began suspecting that an internal authorization mechanism was to blame here. However, the question remained: which one? Fortunately, the checkout action includes comprehensive logging and provides a clue in the "Setting up auth" section:

Setting up auth
  /usr/bin/git config --local --name-only --get-regexp core\.sshCommand
  ...
  /usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***

Aha! The checkout action configures authorization by setting HTTP Authorization header in git's http.extraheader:

http.extraHeader

Pass an additional HTTP header when communicating with a server. If more than one such entry exists, all of them are added as extra headers. To allow overriding the settings inherited from the system config, an empty value will reset the extra headers to the empty list. [source]

What if I read the documentation for the checkout action?

The auth token is persisted in the local git config. This enables your scripts to run authenticated git commands. The token is removed during post-job cleanup .

The action includes a token parameter that has a default value of ${{ github.token }}! This token is equivalent to GITHUB_TOKEN - a unique token generated by Github at the start of each workflow run. However, it is important to note that this token is restricted to the repository where the workflow is executed.

Finally, all the pieces of the puzzle fell into place. What I required was a PAT with read access to the blog source repository and write access to basicwolf.github.io. Once generated, I passed this PAT to the checkout action as follows:

steps:
  - uses: actions/checkout@v3
    with:
      token: ${{secrets.ACCESS_TOKEN_BASICWOLF_GITHUB_IO}}

There is no need to explicitly provide the token for git push anymore, as the git configuration remains consistent across the steps.

This marked the end of an epic quest. The workflow pipeline successfully passed, and you have just completed reading the first article that was automatically deployed to this blog :).

Clean Architecture - book review

2023-05-18T22:39:00+03:00

Oh, I loved the book! A warm picture emerges in my mind: late evening, comfortable armchairs, a hot burning fire in a fireplace, and an engaging discussion with a dear friend who happens to be one of the greatest teachers in software engineering...

"Clean Architecture" is the "Art of War" of the software industry. It was written by a professional who has been through many tough battles, both wins, and losses. Uncle Bob talks about timeless topics of software development that are just as applicable today as they were fifty years ago.

Software architecture determines the necessary pieces and their integration to create the desired system. The pieces can vary from individual variables and functions to large-scale services like databases.

The goal of software architecture is to minimize the human resources required to build and maintain the required system. The measure of design quality is simply the measure of the effort required to meet the needs of the customer.

It is that simple! Or is it? How many times have you struggled trying to fit a new feature into an existing code base? How many times have you started from scratch, only to find later that the redesign became the same mess as before?

Here is a scary diagram from the book. It shows how the cost of each line of code increases with every major release. This trend is not sustainable; no business model can make a profit while letting costs grow exponentially.

Why does this happen? What goes wrong? Too often, we developers buy into the familiar lie: "We can clean it up later; we just have to get to market first!"

Of course, things never do get cleaned up later, because market pressures never abate. Getting to market first simply means that you’ve now got a horde of competitors on your tail, and you have to stay ahead of them by running as fast as you can.

And so the developers never switch modes. They can’t go back and clean things up because they’ve got to get the next feature done, and the next, and the next, and the next. And so the mess builds, and productivity continues its asymptotic approach toward zero.

Making messes is always slower than staying clean and the only way to go fast, is to go well.

We often forget that, unlike firmware, software was invented to be "soft." It was intended to be a way to easily change the behavior of machines. Therefore, software architectures should be as shape-agnostic as possible. Uncle Bob argues that for a software system to be easy to change, it is more important than for it to work.

If you give me a program that does not work but is easy to change, then I can make it work, and keep it working as requirements change. Therefore the program will remain continually useful.

One of our responsibilities is to find the balance and assert the importance of architecture over the urgency of features. Sometimes we have to fight for it with the business side that is pushing to implement new features. If architecture comes last, then the system will become even more costly to develop, and eventually, change will become practically impossible for a part or all of the system.

That is the end of Part I - Introduction. The rest of the tale is for you to discover, just like in 'One Thousand and One Nights'.

Inside the book

In Part II - Starting with the Bricks: Programming Paradigms, Uncle Bob discusses structured, object-oriented, and functional programming paradigms, and how to apply them from an architectural perspective.

Each paradigm is about discipline, telling us what not to do, and each paradigm takes something away from us. Each restricts some aspect of the way we write code.

In Part III - Design Principles we learn to put the well-made software bricks together.

Good software systems begin with clean code. On the one hand, if the bricks aren’t well made, the architecture of the building doesn’t matter much. On the other hand, you can make a substantial mess with well-made bricks. This is where the SOLID principles come in.

The goal of the principles is the creation of mid-level software structures that:

Tolerate change,

Are easy to understand, and

Are the basis of components that can be used in many software systems.

Part IV - Component Principles is about arranging larger software blocks to make a sound system.

Components are the units of deployment. They are the smallest entities that can be deployed as part of a system. In Java, they are jar files. In Ruby, they are gem files. In .Net, they are DLLs. In compiled languages, they are aggregations of binary files. In interpreted languages, they are aggregations of source files.

Regardless of how they are eventually deployed, well-designed components always retain the ability to be independently deployable and, therefore, independently developable.

We learn the principles that help in putting components together. We learn how to group and separate the components and how to manage dependencies between them.

Finally Part V - Architecture and Part VI - Details. So far the book has been preparing us to grok these parts.

The architecture of a software system is the shape given to that system by those who build it. The form of that shape is in the division of that system into components, the arrangement of those components, and the ways in which those components communicate with each other.

Uncle Bob talks about the importance of deferring decisions about details. Should we use a database or a plain text file? Shall we rely on TCP or UDP? Do we use React, Vue, or Angular? He argues that the longer we keep our options open, the more information we can gather and the more experiments we can conduct to make an informed decision.

The architecture that allows us to delay these decisions establishes clear boundaries between its components.

Software architecture is the art of drawing lines that I call boundaries.

This statement leads the way to Clean Architectures, which are

Independent of frameworks. The architecture does not depend on the existence of some library of feature-laden software. This allows to use such frameworks as tools, rather than forcing to cram system into their limited constraints.
Testable. The business rules can be tested without the UI, database, web server, or any other external element.
Independent of the UI. The UI can change easily, without changing the rest of the system.
Independent of the database. We can swap out Oracle or SQL Server for Mongo, BigTable, CouchDB, or something else.
Independent of any external agency. The business rules don’t know anything at all about the interfaces to the outside world.

The hidden gem

Book appendices are often skipped by the readers. Don't you dare to skip this one, because it is the hidden gem of the book. Architecture Archaeology is Uncle Bob's life-long software engineering journey. It is a story of creation and discoveries, of fights and struggle of everything that lead to writing this book.

Thoughts

This is one of the "must-read" for all software engineers. The only question is, "When?". This book discusses many abstract topics. Unless you are intimately familiar with software development, it may have little impact on you. However, when you start interacting with large systems, a time comes when you sense that something is wrong, and you can't simply formalize it. That's when you know the time has come. The time to do architecture right. The time to embrace Clean Architecture.

Good practices: Software deployment and feature releases

2023-04-03T21:52:00+03:00

Software release is the process of making a new version of software available to users. Think of an application or a library uploaded to a registry, where it can be downloaded by end-users.

Once software is released, it can be deployed to the target device or platform. Installing a mobile app from the App Store is a deployment. Publishing a front-end application package on a CDN is also a deployment.

A feature release is the process of making new functionality of software available to users.

The problem

Companies are notorious for delivering new features through "Big Bang" releases, which often means users are not involved in the development process and do not see the new functionality until it is fully completed. This approach can result in half-baked or even unnecessary features being released.

The process of Big Bang feature development is also problematic. A developer spends weeks or months working on a feature, writes the code in a separate version control branch, and does not merge it back into the mainline until it is finished. When the developer finally attempts to integrate the feature into the mainline, they often face numerous issues. Since no one else on the team is familiar with the feature's internals, the developer must be responsible for fixing all the related bugs.

A good practice

Develop the feature in short-lived version control system branches and continuously merge them into the mainline. By doing so, the feature code becomes a part of every software release. To prevent issues, guard the feature code using a feature flag and keep it in a "dormant" state.

Once the feature is viable enough, it can be enabled for a small set of users who are eager to test the latest and greatest of your software, even if some functionalities are still under development. This way, you can receive feedback and iron out any issues before releasing the feature to a wider audience. Once the feature is ready, it can be delivered to all users simply by flipping the feature flag.

Avoiding "Big Bangs" with gradual feature release.

This approach helps to avoid the risks associated with a Big Bang release. By testing the code on a smaller audience beforehand, you can release with more confidence and face fewer surprises, bugs, and defects when the feature goes live.

Specification by example - book review

2023-02-21T22:17:00+02:00

Merriam-Webster dictionary defines specification as

The act of specifying - naming or stating explicitly or in detail.

A detailed, precise presentation of something or a plan or proposal for something.

I think of specification as a definition of what the system is expected to do. Ah, if only systems always behaved as they were supposed to!

You darn well know, how that happens. First comes the business, with brilliant or not-so ideas. "We are going to make a new product! It will increase our revenue! ... [an hour later] and the button text should be in red!". The Business Analysts mince that into a Business case™. The POs scrutinize the case into user stories. The QA write acceptance criteria. Finally, the developers start implementing. But the business has already given the teams a detailed vision of how things should be, so there is not much thinking on the way, just turning JIRA tickets into code. There is even no need to roll it out in small chunks to the end customer - the business already knows what the customer wants... This half a century old illustration has not aged by a single second:

Gojko Adzic analyzed the software industry fallacies that lead to such a result. He found out that how companies deal with these fallacies is very much alike. The methods and processes are shared by the most successful teams that he interviewed while researching for the book. He talks of these methods as a set of process patterns that facilitate change in software products to ensure the right product is delivered effectively.

The foundation of every key pattern is collaboration. Gojko noticed that the "handover" journey of specifications through business people, product managers, developers, and QA is full of misunderstanding, ambiguity, information losses, errors and unnecessary work. "Specification by example" teaches how to build a shared understanding between everyone involved in the software creation process. It then demonstrates how to translate the shared understanding into specification documents, leverage them for continual system testing, and produce up-to-date documentation that evolves with the system.

Key process patterns in a tiny nutshell

A successful team begins by deriving scope from a customer's business goals. They collaborate to define the scope that will achieve the goal. The business representatives explain the intent and the value of the feature, so that everyone understands what's needed. Then, the team works with the business to determine the solution.

The teams collaborate and specify the solution together with the business people. Specifying collaboratively allows to utilize the expertise of stakeholders, gather different ideas about the solution and make everyone more engaged.

The stakeholders tightly work together to illustrate specifications using examples. They identify the key examples which unambiguously define what the software needs to do. They’re both the target for development and serve an evaluation criterion to see whether the development is done.

Key examples are a great starting point for creating specification, but they are not specification yet. Gojko emphasizes, that the common reason for teams to fail with Specification by Example - is not taking the time to process the raw key examples. Successful teams don’t use raw examples; they refine the specification from them. The teams carve out the essence of a key example, discard the unnecessary details and turn it into a clear definition of the desired system behaviour.

Note

Ideally, a specification with examples should unambiguously define the required functionality from a business perspective but not how the system is supposed to implement it.

A good specification, with examples, is effectively an acceptance test for the described functionality.

Automating the validation process is a proven way to avoid the bottlenecks of manual testing. What's important is that the teams validate without changing specifications. Otherwise, all the value of refining specifications is lost. Such automated, comprehensible and accessible to all the team members specification with examples becomes an executable specification.

When the validation process happens frequently, the stakeholders are sure that the software works according to the specification. However, how and in which form can they access the verification results? Hence the team continues sharpening the executable specifications and organises the verification output to provide all stakeholders a consistent, easy-to-understand and up-to-date living documentation.

Thoughts

Specification by Example is Behaviour-Driven Development. Gojko Adzic avoided the term since it was too ambiguous, and the meaning was changing all the time. It's been almost fifteen years since the book was written, and BDD is still ambiguous and misunderstood by large and small companies alike.

In one company, the team was participating in meetings with the clients, and together they derived scope from the business goals. They even documented the key examples that the customer provided. But it never went further - the specifications were extracted to JIRA tickets hierarchy and remained there never to be seen again, once the tickets got closed.

In another company, the user stories were extracted into Cucumber-based behaviour tests. But these tests got written AFTER the implementation in code! They have reflected how developers implemented the software, not how it was supposed to behave.

The third company used Cucumber for contract testing of HTTP API. That nicely fits into "101 ways of abusing Given, When, Then".

Least to say, that none of these companies had living documentation.

"Specification by Example" tells the story of teams and companies that succeeded in BDD. Gojko first talks about the obstacles that the teams should go through to change the mindset about BDD. He continues with a detailed explanation of every key pattern - every pattern has a designated chapter. The last six chapters of the book are the case studies, which narrate the companies journey from waterfall and "agile-ish" development processes to specification by example.

To my shame, I have yet to build a system that incorporates all the key patterns described in the book. And I am eagerly looking forward to it!

Hexagonal architecture and Python - Part III: Persistence, Transactions, Exceptions and The Final Assembly

2022-12-31T17:19:00+02:00

In Part II we discussed the Hexagonal architecture of ports and adapters, the role of the Domain layer and its influence on API and database layers. We exposed the application to the outer world via the HTTP API port.

In Part III we are going to put all the application pieces together. But before that, we have to connect the application to the database, research Django transactions in unit tests and set up exception handling on the HTTP API level.

The repository

We love Django for many things, but most of them all - is the fantastic ORM. Django ORM abstracts the underlying database so gracefully that swapping one database engine to another is just a matter of updating the application configuration. But what if there is another data storage not supported by Django? What if the data storage is a web service?

This problem is commonly solved via the Repository design pattern. A repository separates the logic of communicating with the database from the rest of the application. In terms of Hexagonal architecture, a repository is an SPI adapter that handles commands and queries from one or more related ports.

Code-wise, the VotingUserRepository [source] is a class that implements the FindVotingUserPort [source] and the SaveVotingUserPort [source] ports:

# /src/myapp/application/adapter/spi/persistence/repository/voting_user_repository.py

class VotingUserRepository(
    FindVotingUserPort,
    SaveVotingUserPort
):
    def find_voting_user(self, article_id: ArticleId, user_id: UserId) -> VotingUser:
        ...
        return VotingUser(...)

    def save_voting_user(self, voting_user: VotingUser) -> VotingUser:
        ...
        return VotingUser(...)

Strictly speaking, there is no need to use the Repository pattern here. We have already decoupled the application from the data stores via SPI ports. However, a repository is a convenient way of putting related adapter implementations in one place. The VotingUserRepository provides database-backed adapters for both FindVotingUserPort and SaveVotingUserPort SPI ports. It communicates with the database through Django ORM:

A diagram which shows the data flow between VotingUserRepository, Django ORM and a Database.

From domain models to database entities

The VotingUserRepository has to map the VotingUser domain model to data model(s) when saving to - and vice versa, when reading from the database. The VotingUser model holds three pieces of information, which eventually influence the database design [source]:

# /src/myapp/application/domain/model/voting_user.py

class VotingUser:
    id: UserId
    karma: Karma
    votes_for_articles: list[ArticleVote]

A modern relational database with native JSON support can store such an object in a single table row. Django natively supports JSON fields and provides unbelievable JSON querying capabilities.

A more traditional way is to store ArticleVotes and VotingUsers in separate tables with an explicit relationship between them:

Note

Although I drew article_vote.user_id and article_vote.article_id as foreign keys, this is not reflected in the example project code. There is some hidden complexity here, like - if a user gets deleted, should the article votes stay? Or should they remain intact if an article gets deleted? If yes - how should we preserve the data? I hope you would indulge me this simplification and leave the proper foreign key relationships and domain-specific deletion handling for another time.

I bet you already picture how to express these tables as Django models. Here is a tricky question: where should we put them? It's not hard to answer if you remember the structure of HTTP API adapters. Let's do something similar for our persistence SPI adapter:

myapp/application/adapter/spi/
    persistence/
        entity/
            article_vote_entity.py     # Django model here
            voting_user_entity.py      # and here
        repository/
            voting_user_repository.py
        exceptions/
            ...

Here is another unorthodox move: I called Django models "Entities". That is - to avoid confusion between domain models and Django models. There is no other hidden meaning of "Entity" in this case.

Here is the ArticleVoteEntity [source]:

# /src/myapp/application/adapter/spi/persistence/entity/article_vote_entity.py

class ArticleVoteEntity(models.Model):
    VOTE_UP = 'up'
    VOTE_DOWN = 'down'

    VOTES_CHOICES = [
        (VOTE_UP, 'UP'),
        (VOTE_DOWN, 'DOWN')
    ]

    id = models.UUIDField(primary_key=True, default=uuid4, editable=False)
    user_id = models.UUIDField()
    article_id = models.UUIDField()
    vote = models.CharField(max_length=4, choices=VOTES_CHOICES)

    class Meta:
        unique_together = [['user_id', 'article_id']]
        db_table = 'article_vote'

The VotingUserEntity is simpler [source] :

# /src/myapp/application/adapter/spi/persistence/entity/voting_user_entity.py

class VotingUserEntity(models.Model):
    user_id = models.UUIDField(primary_key=True, default=uuid4, editable=False)
    karma = models.IntegerField()

    class Meta:
        db_table = 'voting_user'

The entity definitions are lean and have no methods in them. All that is required - is to map data from Python to database and back. And Django flawlessly does that.

Inside the repository

Once you've seen the entities (i.e. Django models), things get very predictable. The repository implements two SPI ports - one to get a VotingUser from the database, another to persist a VotingUser to the database. The repository takes care of transforming the domain model to Django entities and calling the required Django ORM methods. The repository then transforms the result back to the domain model:

A diagram which shows how VotingUser domain model is persisted to a database through Django ORM mechanism.

The implementation fits just in few lines after hiding Django ORM boilerplate in private methods [source]:

# /src/myapp/application/adapter/spi/persistence/repository/voting_user_repository.py

def find_voting_user(self, article_id: ArticleId, user_id: UserId) -> VotingUser:
    voting_user_entity = self._get_voting_user_entity(user_id)
    votes_for_articles = self._get_votes_for_articles(article_id, user_id)

    return VotingUser(
        user_id,
        Karma(voting_user_entity.karma),
        votes_for_articles
    )

Remember that I put no explicit ForeignKey relationships here, so we have to load the votes manually. A user is saved in the same fashion [source]:

# /src/myapp/application/adapter/spi/persistence/repository/voting_user_repository.py

def save_voting_user(self, voting_user: VotingUser) -> VotingUser:
    saved_voting_user_entity = self._save_voting_user(voting_user)
    saved_article_vote_entities = self._save_votes_for_articles(
        voting_user.votes_for_articles
    )

    return self._voting_user_entity_to_domain_model(
        saved_voting_user_entity,
        saved_article_vote_entities
    )

Perhaps you are wondering: why map a VotingUserEntity back to a VotingUser domain model if we already pass it as an argument? The data passed in an INSERT INTO statement may run through numerous transformations before it reaches the data storage. The well-known case is an "auto-incremental" integer primary key generated via BEFORE INSERT triggers. So, it is paramount to return a domain model that resembles the updated database state, not what we pass to the routine.

Transactions management

A common way to handle database transactions in Django - is to wrap each HTTP request in a transaction. This works for simple cases. But what does it mean in terms of Hexagonal architecture? In Part II we created an HTTP API adapter, which invokes the VoteForArticleUseCase [source].

But there could be many other adapters - a command line adapter, a message bus adapter, an RPC adapter etc. We can't trust all those adapters to wrap a use case in a transaction. So, the burden of defining transaction boundaries is shifted to a use case implementation - an application service. An application service places the database write operations into a single transaction. The operations then roll back if one of them fails.

Transactions in Django

We know that an application service vows to be agnostic of SPI adapters' implementation details. But wrapping SPI calls in transaction breaks this vow. Correction: wrapping SPI calls in a transaction using Django routines breaks this vow. Django's transaction.atomic is the unambiguous way to wrap a code block in a transaction. It is used either as a decorator or a context manager. But the problem is that Python decorators and context managers are runtime constructs. Internally transaction.atomic operates on database connection object. While we certainly need a database connection at a run time, we don't need one for unit testing. This means that the following perfectly valid production code

from django.db import transaction

class ArticleRatingService(VoteForArticleUseCase):
    @transaction.atomic
    def vote_for_article(self, command):
        ...

fails in a unit test (pytest) with RuntimeError("Database access not allowed...") exception, unless a database connection is established. The exception suggest marking the test with @pytest.mark.django_db - but this turns a unit test into an integration test!

With all the "batteries included", Django doesn't include one for this case. My version of such a battery is @transactional decorator, which wraps a callable in a transaction for production, but skips it when running tests [source].

Warning

Code smell detected. A routine in a production code made solely for the sake of testing is a sign of a bad design. However, such hack is necessary to push a Django application out of the framework box.

SPI adapters exceptions

Traditionally handling exceptions in Django views boils down to wrapping a block of code in try..except and returning an error-indicating result:

def get_user_name(request, user_id):
    try:
        user = User.objects.get(user_id=user_id)
        return HttpResponse(user.name)
    except User.DoesNotExist:
        return HttpResponseNotFound("User not found :(")

But how does an API adapter handles an exception raised by an SPI adapter? The dependencies directions principle in Hexagonal Architecture reminds us that API adapters should not be aware of SPI adapters' innards.

Hexagonal Architecture layers, from top to bottom: Adapters, Ports, Application Services, Domain

Instead, they are aware of interfaces exposed by SPI ports. An SPI port is not necessarily a bare interface. It includes type definitions, exceptions and other constructs that make an SPI port a whole. Our SPI port includes such an exception [source]:

# /src/myapp/application/adapter/spi/persistence/exceptions/voting_user_not_found.py

class VotingUserNotFound(RuntimeError):
    def __init__(self, user_id: UserId):
        super().__init__(f"User '{user_id}' not found")

The SPI adapter catches the downstream User.DoesNotExist exception and re-throws it upstream in form of VotingUserNotfound [source]:

# /src/myapp/application/adapter/spi/persistence/repository/voting_user_repository.py

def _get_voting_user_entity(self, user_id: UserId) -> VotingUserEntity:
    try:
        return VotingUserEntity.objects.get(user_id=user_id)
    except VotingUserEntity.DoesNotExist as e:
        raise VotingUserNotFound(user_id) from e

Exception handler

Now that we know which exception to catch in the HTTP API adapter, let's discuss ways to handle it. The most obvious way is to wrap the body of ArticleVoteView.post() in try..except statement. That works fine for a small number of exceptions, but in a larger application with lots of exceptions of a similar origin, a common mechanism might be preferable. For example, an "Entity Not Found" exception can be expressed as the HTTP 404 response for any kind of entity. Luckily for us, Django Rest Framework provides a common exception-handling mechanism. You can define a custom exception handler function and set REST_FRAMEWORK.EXCEPTION_HANDLER settings value.

If a simple form, the handler catches SPI exceptions and converts them to an HTTP response [source]:

# /src/myapp/application/adapter/api/http/exceptions_handler.py

def exceptions_handler(exc, context):
    ...
    if isinstance(exc, VotingUserNotFound):
        return Response({
            'title': "Error",
            'detail': str(exc),
            'status': HTTPStatus.BAD_REQUEST
        })

The Final Assembly

All the components of the application - the domain model, application services, API and SPI adapters are ready. All that is left - is to put them together. We are going to use a small dependencies or Inversion of Control container that creates all the components instances and wires them into a complete application:

A diagram which shows how Django passes control flow to the IoC Container.

The container commences by instantiating VotingUserRepository. It is the sole component with no dependencies. Next, the container instantiates the components that depend on the already available components instances, i.e. ArticleRatingservice that solely depends upon VotingUserRepository. It continues recursively assembling components until the dependency tree is fully populated [source]:

# /src/myapp/dependencies_container.py

def build_production_dependencies_container() -> Dict[str, Any]:
    voting_user_repository = VotingUserRepository()

    article_rating_service = ArticleRatingService(
        find_voting_user_port=voting_user_repository,
        save_voting_user_port=voting_user_repository
    )

    article_vote_django_view = ArticleVoteView.as_view(
        vote_for_article_use_case=article_rating_service
    )

    return {
        'article_vote_django_view': article_vote_django_view
    }

Notice that container only exposes a single entry point: the HTTP view, i.e. the API adapter.

Configuration

Where and when should we build the container? Consider this: the container creates and wires the components together. We can use different components for different environments - e.g. have a VotingUserExcelRepository. So, the container configures the application. And Django applications have a well-known place for configurations - the AppConfig class and the AppConfig.ready() method:

# /src/myapp/apps.py

class MyAppConfig(AppConfig):
    name = 'myapp'
    container: Dict[str, Any]

    def ready(self) -> None:
        from myapp.dependencies_container import build_production_dependencies_container
        self.container = build_production_dependencies_container()

URLs

The application configuration registry is populated before Django sets up the URLs. It allows to get the application configuration from the registry and bind the view to a URL [source]:

# /src/myapp/urls.py

app_config = django_apps.get_containing_app_config('myapp')
article_vote_django_view = app_config.container['article_vote_django_view']

urlpatterns = [
    path('article_vote', article_vote_django_view)
]

Django models

Django expects to find models in application's models.py file. Since we declared the entities deep down in SPI adapters, they have to be imported manually here [source]:

# /src/myapp/models.py

from myapp.application.adapter.spi.persistence.entity.article_vote_entity import (
    ArticleVoteEntity
)
from myapp.application.adapter.spi.persistence.entity.voting_user_entity import (
    VotingUserEntity
)

Testing

The Final Assembly is a slightly misleading term. There is no need to wait for all the components to be available to assemble a production-level application container. On the surface, a container exposes an API entry point, backed either by a production-grade component or a dummy.

So, the dependencies container should not be the last but one of the first items on the list when building an application following Hexagonal architecture principles. You can add the components to the container as they get ready and test every time how they fit together [source]:

# tests/test_myapp/test_dependencies_container.py

def test_build_production_ioc_container():
    build_production_dependencies_container()

At the same time, a simple smoke test can make a vote by a non-existing user and expect a response with HTTP 404 - Not Found status back:

curl http://localhost:8000/api/article_vote \
--header "Content-Type: application/json" \
--request POST \
--data '{
   "user_id": "efed6f83-49ee-4cbc-bdbd-2b92bf428f2b",
   "article_id": "60ccea0c-0bf2-4726-8ac7-324fa03a74cd",
   "vote": "UP"
 }'

>
{
    "title":"Error",
    "detail":"User 'efed6f83-49ee-4cbc-bdbd-2b92bf428f2b' not found",
    "status":404
}

Bear in mind that the job of a smoke test is to verify whether the application works. An ideal smoke test touches all the application layers without affecting its state. Voting by a non-existing user cuts through all the application layers, down to the database and returns the expected error result.

This test can also run against an application-in-development with a dummy HTTP or SPI (database) adapters. Replacing a dummy adapter with a real one should not require any modifications to the test.

Final thoughts

It took me almost two years to complete this article series. The example code lived through two major rewrites and has been continuously evolving. I am satisfied with the output and the outcome of this work. It helped me to understand Hexagonal Architecture and dive deeper into DDD. I hope it makes an inspiring impact on fellow Pythonistas!

Acknowledgments

Traditionally, Jere "Urokhtor" Teittinen gave priceless feedback on the draft, for which I am very grateful!

Refactoring - book review

2022-10-15T12:00:00+03:00

Refactoring is a book by Martin Fowler (with contributions by Kent Beck) about what a fellow developer might describe as "a process of improving the code quality."

Martin Fowler defines its meaning more accurately:

Refactoring (noun) - A change made to the internal structure of software to make it easier to understand and cheaper to modify without changing its observable behavior.

Refactoring (verb) - To restructure software by applying a series of refactorings without changing its observable behavior.

So often developers refactor code and apply dozens of refactorings unconsciously! We won't name the small steps, and don't categorise our actions - we merely crave clean code. Often we forget that the code should roughly do the same things it did before, i.e. the observable behaviour should not change.

In a nutshell, "Refactoring" teaches detecting a piece of code that needs refactoring, determining which refactorings are necessary, applying them, and most importantly - preserving the code behavior while doing all these.

Inside the book

Chapter 1 pulls the reader straight into refactoring example. This chapter is available for free download. Before diving into the code, the author discusses the reasons for refactoring. After all, a compiler doesn't care about the quality of our code. It has no aesthetic judgement. We like clean code because it is easier to comprehend and change.

A poorly designed system is hard to change—because it is difficult to figure out what to change and how these changes will interact with the existing code to get the behaviour I want. And if it is hard to figure out what to change, there is a good chance that I will make mistakes and introduce bugs.

Fowler teaches how to improve the code without breaking it - in tiny, explicit, consequential steps. He emphasises the tests as the cornerstone of successful refactoring. If a piece of code has no tests, it's hard to tell whether its observed behaviour changed after refactoring.

When you have to add a feature to a program but the code is not structured in a convenient way, first refactor the program to make it easy to add the feature, then add the feature.

Chapter 2 - "Principles in Refactoring", discusses why and when we should refactor, the problems and benefits of refactoring, its relationship to the software development process in general and dozens of other aspects that one would not even imagine. This chapter also aids in explaining refactoring to а wide circle of people, from junior developers to senior managers.

Chapter 3 - "Bad Smells in Code", is written in cooperation with Kent Beck. While Chapter 2 talks of the right time to refactor, the third chapter describes the looks and smells of code that is crying for refactoring. Best of all - the chapter is in plain English - not a single line of code in it! "Global data", "Repeated switches", "Temporary field", and many more - I bet you've met them before.

Chapter 4 - "Building Tests", explains the value of self-testing code and why effective refactoring is impossible without a reliable test suite.

Refactoring is a valuable tool, but it can’t come alone. To do refactoring properly, I need a solid suite of tests to spot my inevitable mistakes. Even with automated refactoring tools, many of my refactorings will still need checking via a test suite.

The Catalogue

The rest of the book is a monumental catalogue of refactorings. There are six chapters with more than 60 refactorings thoroughly broken down. Fowler starts with a mix of refactorings that he considers the most useful and common, such as "Extract function", "Rename Variable", "Introduce Parameter Object". Every following chapter reveals a group of related refactorings.

Each refactoring is presented in a standard format:

Name. The name is important to building a vocabulary of refactorings. Fowler refers to a refactoring by its name elsewhere in the book.
Sketch. The sketch shows a small code example of the transformation of the refactoring. It does not explain what a refactoring is but helps to understand and recognise a refactoring quickly.
Motivation. The motivation describes why the refactoring should be done and the circumstances in which it should not. While it's possible to deal with a code smell via different refactorings, this section describes the "flavour" of a code smell that requires the given refactoring.
Mechanics. The mechanics are a concise, step-by-step description of how to carry out the refactoring.
Example. The examples show how to apply the refactoring in code.

It's worth mentioning that the online catalogue of refactorings with names and sketches is available for free at https://www.refactoring.com/catalog/!

Reading the book

This book primarily appeals to seasoned developers. One has to write thousands and read tens of thousands of lines of code before she is able to detect code smells effortlessly. The book won't make much sense for a junior developer making his first steps in programming.

Most of the refactorings would look familiar for the developers that have been around the block. Challenge yourself by trying to describe the exact mechanics of a refactoring before reading them. I failed for almost all the refactorings, missing (not even realising) some steps every time! The examples help to map everything you've just learned about a refactoring to real code. It might be tedious to read every example section thoroughly - after all, those are paper-printed code blocks. But the book won't be complete without them.

Thoughts

After I finished reading the book I felt that all my prior practical knowledge and experience has been "rebased" on a solid theoretical foundation. The book vocabulary is a ubiquitous language of the refactoring domain and makes it so much easier to explain your intentions to colleagues - especially when they also use the same vocabulary. I highly recommend this book to software professionals who want to improve and formalise their refactoring skills.

Hexagonal architecture and Python - Part II: Domain, Application Services, Ports and Adapters

2022-09-18T12:00:00+03:00

Update

February 2025

Originally, I said that the domain model is the first thing to put in code. Today, I always start with public interfaces, and in the case of RESTful services, the API. Though what I consider "the right way" has changed, I left this article almost intact, only commenting in a few places why API should be **coded* first.*

Intro

Now that you are familiar with the basic principles of Hexagonal architecture (see part I) let's try implementing a Django-based application following these principles. I've chosen Django for this exercise to demonstrate that even an opinionated framework is not an obstacle for Hexagonal architecture. What about the other web frameworks, like FastAPI, Flask, AIOHTTP with SQLAchemy or a NoSQL data store?

Hexagonal architecture painlessly decouples the business logic from the technical details of HTTP communication, file system and database access, messaging and so on. You can swap Django with FastAPI, get rid of Django ORM and go with SQLAlchemy, and the business logic implementation remains the same!

The source code of the example is available at BasicWolf/hexagonal-architecture-django Github repository.

Tip

Clone the repository before reading further. The layered hexagonal architecture means deeply nested python packages. It's much easier to follow the article when the example code is available locally.

Project structure

A Django project is easily recognized by the top-level structure:

src/
  hexarch_project/             # Django application essentials: wsgi.py, urls.py, settings.py
  myapp/
    migrations/                # Django migrations
    apps.py                    # Django app configuration (creates dependencies container instance)
    dependencies_container.py  # Dependencies container
    models.py                  # Django DB models (imports models from SPI adapters)
    urls.py                    # Django urls mappings
    ⋮
    application/               # application code; the structure follows the principles of hexagonal architecture

Under the surface, the application directory structure is much deeper. From Python's perspective, all directories under application/ are namespace-packages i.e. there is no __init__.py in them. The namespace packages allow the tests packages structure to reflect the application packages structure.

The example application is structured as follows:

domain/            # Business domain models, services
  model/
  service/
port/              # API and SPI ports (interfaces)
  api/
  spi/
adapter/           # API and SPI ports implementation
  api/
    http/          # Django views and serializers
    messaging/
    ⋮
  spi/
    persistence/
      entity/      # Django models
      exceptions/  # Generic (django-independent) persistence exceptions
      repository/  # High-level persistence abstraction
    messaging/
    ⋮
service/           # Application services

Notice that Django is used only in adapters. Django Views are HTTP API adapters and Django Models are Persistence SPI adapters. The rest of the project is independent from the framework.

Use case: Upvote an Article

Now, let's take a look at an example use case. Imagine a web blogging platform in development. We want to add articles rating and let the users influence them.

We gathered with the end-users, platform experts, QA, and other stakeholders and drew some ideas:

Every article has a rating.
A user can change an article rating.
To change the article rating, a user either "upvotes" or "downvotes".
Users can vote for the article only if their "karma" (i.e. user rating) value is high enough, greater than 5.
A user can vote once per article.

A user story is born:

As a user of a blogging platform
I want to give my vote,
So that the article's rating changes.

Where do we start?

That's a simple question, isn't it? Let's consider the options:

Database. It is important to integrate early with the downstream dependencies. The invisible bottlenecks could bring unpleasant surprises if integration is postponed till the last moment. Yet, a database is just storage and is usually located at the bottom layer of the application infrastructure. Do we really want to implement the database first and let it influence the application implementation?
Public API. Public API is the contract with the outer world. Its best design emerges when API consumers and producers collaborate. After the API specification is ready, the consumers and the producer (our service) can implement their part of the contract independently, and start integration as early as possible, even when no domain/business logic code exists yet!

We should always start by studying the domain, building shared understanding and deriving API from a detailed domain model. But when it comes to putting things in code, nowadays I tend to release API integration components as early as possible. In the early stages, I hard-code the returned values and focus on making sure that consumers can integrate. Only then do I start implementing the actual business logic.
Domain. Starting with the domain model gives an advantage: we can test our understanding of the domain model, by expressing it in the code. Behaviour-driven development is essential at this stage. BDD brings techniques and tools to test the domain model code against the previously defined user stories. The problem is that we don't have the means to interact with the model yet. Which means that we have to go API first.

However, the article continues as originally put - the Domain model is implemented before API View. The flow of implementing the API first is quite different from domain-first.

The domain

A discussion between the developers, users and domain experts gives a birth to a domain-specific language, where each term has a particular meaning. We call this language - Ubiquitous language (UL).

In Hexagonal architecture, the domain layer encapsulates the business logic and business processes. On the code side, the UL terms are used in the names of classes, methods, and other code units. Thus, by looking at the code, you can always tell how it is related to the problem domain.

For example, a vote can be represented via the following enumeration [source]:

# /src/myapp/application/domain/model/vote.py

class Vote(Enum):
    UP = 'up'
    DOWN = 'down'

Karma is an explicit type alias [source]:

# /src/myapp/application/domain/model/karma.py

Karma = NewType('Karma', int)

The most complex class of the domain is VotingUser, which represents a user that votes for an article. The karma value determines whether the user can vote. We also need to know whether the user has already voted to prevent repeat voting. Voting for an article produces a result:

@dataclass
class VotingUser:
    id: UserId
    karma: Karma
    votes_for_articles: list[ArticleVote] = field(default_factory=list)

    def vote_for_article(
        self,
        article_id: ArticleId,
        vote: Vote
    ) -> VoteForArticleResult:
        if self._user_voted_for_article(article_id):
            return AlreadyVotedResult(article_id, self.id)

        if not self._karma_enough_for_voting():
            return InsufficientKarmaResult(user_id=self.id)

        ## IMPORTANT! The model state changes! ##
        self.votes_for_articles.append(
            ArticleVote(article_id, self.id, vote)
        )

        return SuccessfullyVotedResult(article_id, self.id, vote)

    ...

Notice that even the private methods _user_voted_for_article() and _karma_enough_for_voting follow the domain language. A fellow developer can easily map the code to the domain model and business rules.

You may wonder what VoteForArticleResult and SuccessfullyVotedResult are. Recall the basics of Hexagonal architecture from Part I: ..

Dependencies are directed from the outer layers to the inner centre.

VoteForArticleResult [source] is a domain data transfer object model. It carries the voting result from the innermost application layer - the Domain - to the outermost API adapter layer.

Writing tests for a domain model is straightforward since VotingUser.vote_for_article(...) return value is determined only by the VotingUser instance state and the method input values. With meaningful fixtures names, tests turn into simple scenarios, for example [source]:

# tests/test_myapp/application/domain/model/test_voting_user.py

def test_vote_for_article_twice_returns_already_voted_result(
    voting_user_who_has_voted: VotingUser,
    article_id_for_which_user_has_voted: ArticleId,
    a_vote: Vote,
    expected_already_voted_result: AlreadyVotedResult
):
    voting_result = voting_user_who_has_voted.vote_for_article(
        article_id_for_which_user_has_voted,
        a_vote
    )
    assert voting_result == expected_already_voted_result

Once the domain model is fully implemented, we switch our focus to its primary users - application services.

Application service: a skeleton

Application services are the conductors that orchestrate processes and data flow in the application. An application service implements one or more related use cases and invokes all the necessary dependencies required to perform these use cases. The service implementation can start with a single return statement only:

# /src/myapp/application/service/article_rating_service.py

class ArticleRatingService(
    VoteForArticleUseCase
):
    def vote_for_article(self, command: VoteForArticleCommand) -> VoteForArticleResult:
        return SuccessfullyVotedResult(
            command.article_id,
            command.user_id,
            command.vote
        )

This dummy implementation is good enough to echo the commands back, but much more needed to execute the use case. For example, where does the application service get a VotingUser?

SPI Ports

In Hexagonal Architecture, an application service communicates with the outer world via Service Provider Interface (SPI) ports. The application service fetches the users by user_id and article_id. That can be expressed in a FindVotingUserPort interface as follows [source]:

# /src/myapp/application/port/spi/find_voting_user_port.py

class FindVotingUserPort(Protocol):
    def find_voting_user(self, article_id: ArticleId, user_id: UserId) -> VotingUser:
        pass

We add FindVotingUserPort as a dependency to the application service. In practice, we add a respective field and a way to initialize it via constructor:

# /src/myapp/application/service/article_rating_service.py

class ArticleRatingService(
    VoteForArticleUseCase
):
    _find_voting_user_port: FindVotingUserPort
    ...

    def __init__(
        self,
        find_voting_user_port: FindVotingUserPort,
        ...
    )
        self._find_voting_user_port = find_voting_user_port

Can you tell, what actual implementation is behind that interface? Is VotingUser found from a file? A database? Perhaps another HTTP endpoint? Or a hard-coded value? The application service does not care. It just makes a call:

# /src/myapp/application/service/article_rating_service.py

class ArticleRatingService(...):
    def vote_for_article(self, command: VoteForArticleCommand) -> VoteForArticleResult:
        voting_user = self._find_voting_user_port.find_voting_user(
            command.article_id,
            command.user_id
        )
        ...

Another responsibility of the service is to persist the voting results.

It terms of Domain-Driven Design, VotingUser is an Aggregate root. To update an article rating we have to persist a VotingUser as a whole. SaveVotingUserPort takes care of that [source]:

# /src/myapp/application/port/spi/save_voting_user_port.py

class SaveVotingUserPort(Protocol):
    def save_voting_user(self, voting_user: VotingUser) -> VotingUser:
        raise pass

Putting the service pieces together

Finally, ArticleRatingService has all the bits and pieces required to execute the use case [source]:

# /src/myapp/application/service/article_rating_service.py

class ArticleRatingService(
    VoteForArticleUseCase
):
    _find_voting_user_port: FindVotingUserPort
    _save_voting_user_port: SaveVotingUserPort

    ...

    def vote_for_article(self, command: VoteForArticleCommand) -> VoteForArticleResult:
        voting_user = self._find_voting_user_port.find_voting_user(
            command.article_id,
            command.user_id
        )

        voting_result = voting_user.vote_for_article(
            command.article_id,
            command.vote
        )

        match voting_result:
            case SuccessfullyVotedResult():
                self._save_voting_user_port.save_voting_user(voting_user)

        return voting_result

First, the service gets the required VotingUser. Next, the user votes for the article. Last, the service checks whether the user has successfully voted and persists the user state.

Note

Do you remember that an application service is supposed to orchestrate the flow without any knowledge of its content? You may have noticed, that our application service does not fulfil this promise. The service controls the flow in match voting_result: block. I had to cheat here to make the code easier to follow and comprehend. One of the purer alternatives is the Domain events mechanism. It is a very interesting, but a huge topic, and unfortunately it falls out of the scope of this article.

Test-driven application services development

Testing application service differs from testing a domain model. Unlike a domain model, an application service has SPI dependencies, which should be replaced with test doubles in each test. That fact should not complicate the tests, though. It is possible to construct and explicitly pass a dependency test double and rely on default values for the rest of them.

For example, we test that the service persists the voting user [source]. The only dependency explicitly declared and passed to the service builder is SaveVotingUserPortMock test double. All other dependencies are provided by the build_article_rating_service() builder function:

# /tests/test_myapp/application/service/test_article_rating_service.py

def test_voting_user_saved(
    self,
    vote_for_article_command: VoteForArticleCommand,
    saved_voting_user: VotingUser
):
    save_voting_user_port_mock = SaveVotingUserPortMock()
    article_rating_service = build_article_rating_service(
        save_voting_user_port=save_voting_user_port_mock
    )

    article_rating_service.vote_for_article(vote_for_article_command)

    assert save_voting_user_port_mock.saved_voting_user == saved_voting_user

The application service which implements VoteForArticleUseCase is ready. Next, we implement the adapter that invokes the use case.

HTTP API

Let's start with the specification skeleton. Notice how the request and responses are derived from the domain model:

paths:
  /article_vote:
    post:
      summary: Vote for an article.

      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Vote'

      responses:
        '201':
          summary: Voted successfully.
        '400':
          summary: Bad request. There was a domain constraint violation.
        '409':
          summary: Conflict. User has already voted.

Note

I deliberately omit the complete specification since the topic is out of this article's scope.

Django Rest Framework is the obvious choice to facilitate a RESTful endpoint implementation with Django. Beside the request and response processing, the logic fits into few lines of code [source]:

# /src/myapp/application/adapter/api/http/article_vote_view.py

class ArticleVoteView(APIView):

    def __init__(self, vote_for_article_use_case: VoteForArticleUseCase):
        self.vote_for_article_use_case = vote_for_article_use_case
        super().__init__()

    def post(self, request: Request) -> Response:
        vote_for_article_command = self._read_command(request)
        result = self.vote_for_article_use_case.vote_for_article(
            vote_for_article_command
        )
        return self._build_response(result)

    ...

The intentions are:

Accept the HTTP request, deserialize and validate the request data.
Invoke the use case.
Serialize the result and render the response.

The view has one dependency: an object which implements VoteForArticleUseCase protocol [source]:

# /src/myapp/application/port/api/vote_for_article_use_case.py

class VoteForArticleUseCase(Protocol):
   def vote_for_article(self, command: VoteForArticleCommand) -> VoteForArticleResult:
       pass

As we already know, the ArticleRatingService has implemented this protocol as is ready to be wired with the controller.

Testing a HTTP controller is no different from testing an application service. Every test injects a tuned double of the VoteForArticleUseCase dependency and asserts the expected state or behavior. This makes the view testing times more lightweight compared to the traditional, spin-it-all-up Django app testing.

For example, how to test a scenario, where a user tries to vote twice in a row? [source]

# /tests/test_myapp/application/adapter/api/http/test_article_vote_view.py

def test_user_votes_for_the_same_article_returns_conflict(
    arf: APIRequestFactory
):
    ## This is *the* view we are testing
    article_vote_view = ArticleVoteView.as_view(
        ## we are injecting a stub which always
        ## returns AlreadyVotedResult (see below)
        vote_for_article_use_case=VoteForArticleUseCaseAlreadyVotedStub()
    )

    ## A valid article vote is POSTed
    response: Response = article_vote_view(
        arf.post(
            '/article_vote',
            {
                'user_id': UserId(UUID('a3854820-0000-0000-0000-000000000000')),
                'article_id': ArticleId(UUID('dd494bd6-0000-0000-0000-000000000000')),
                'vote': Vote.UP.value
            },
            format='json'
        )
    )

    ## But the result is HTTP 409, as defined in the specification
    assert response.status_code == HTTPStatus.CONFLICT
    assert response.data == {
        'status': 409,
        'detail': "User \"a3854820-0000-0000-0000-000000000000\" has already voted"
                  " for article \"dd494bd6-0000-0000-0000-000000000000\"",
        'title': "Cannot vote for an article"
    }

And here is the VoteForArticleUseCaseAlreadyVotedStub [source]:

# /tests/test_myapp/application/adapter/api/http/test_article_vote_view.py

class VoteForArticleUseCaseAlreadyVotedStub(VoteForArticleUseCase):
    def vote_for_article(self, command: VoteForArticleCommand) -> VoteForArticleResult:
        return AlreadyVotedResult(
            user_id=command.user_id,
            article_id=command.article_id
        )

Please pause for a moment. Does it take much effort to grok the test? Did you notice that the test does not interact with rest of the application? Did you also notice that it takes only five lines of code (three, if you put the return on a single line!) to mock "the rest of the application"? The responsibilities are clearly decoupled, and there is no need to set up a database or any other service to test an HTTP endpoint.

What's next

This concludes Part II of the article series about Hexagonal Architecture and Python and Django. Part III will discuss how to use Django Models in SPIs, manage database transactions and put all the application pieces together. Stay tuned!

Acknowledgments

I would like to express my very great appreciation to Jere "Urokhtor" Teittinen and Jarkko "jmp" Piiroinen for reviewing the article and helping to improve it!

Good practices: Avoiding the use of test inputs in assertions

2021-11-10T12:00:00+02:00

Imagine that we have to implement a adding a title to a document. The relationship between Title and Document is straightforward and we begin by writing a test:

def test_add_title_to_document():
    document = Document()
    title = Title("Lorem Ipsum")

    document.add_title(title)

    assert document.title == title

Take a minute to spot the weak points of the code above and let's discuss them.

Test-First Loop

People who practice test-driven development sometimes focus on the details of the process too much. We start by writing a test. Sometimes we write it to the end, sometimes we add small portions. The test fails and we write implementation to fix the failing part. The cycles repeats until the test is fully written and the implementation fulfills the test. The satisfied developer moves to the next task, everyone is happy. Is there something missing though?

The Ultimate Test

If someone asks you, what was the ultimate purpose of any test (we're speaking of software development of course) what would you answer?

I think that the goal of any test is to confirm the relationship between certain inputs and outputs of a routine. The routine size doesn't matter - it could be as small as a function in a unit test or as big as a complex behavior of multiple services in an end-to-end test.

This brings us back to the code above. The input is Title("lorem ipsum"). What about the output?

Outputs ≠ Inputs

The output of the test above is also the input. That is the biggest evil here. There is absolutely no way to fail this test by modifying the input:

def test_add_title_to_document():
    document = Document()
    title = Title("dolor sit amet")  # changes don't affect test behaviour

    document.add_title(title)

    assert document.title == title

However, the test fails if input and output are separated:

def test_add_title_to_document():
    document = Document()
    title = Title("dolor sit amet")
    document.add_title(title)

    assert document.title == Title("Lorem Ipsum")  # hooray, this fails!

Test-First Loop: Extended edition

My colleague Jere Teittinen taught me an amazingly simple and useful trick about writing tests. When you have completed writing a test assertion and the routine behind it, change the inputs or outputs to verify that the assertion fails. This is an important step which protects the test from tailored routine implementation. Such routine is able to fulfill the test only with the original inputs and outputs. You shake them a bit and everything falls apart like a house of cards.

Hexagonal architecture and Python - Part I: Dependency Injection and componential architecture

2021-10-30T22:30:00+03:00

Time flies awfully fast! Two and a half years ago I left the world of Django and found myself in the world of Spring Boot, Kotlin and Java. It was a genuine cultural shock. An extensive amount of new knowledge bombarded my brains. Sometimes it was so furious, that I wanted to run back to the beloved and bytewise familiar Python ecosystem. Inversion of Control (IoC) was the hardest topic to digest. Automated Dependency Injection (DI) felt like black magic, compared to Django's direct approach. Spring Boot behemoth framework consumed me in nightmares. But all the effort was worth it.

We designed and implemented the application following Hexagonal architecture rules. And the final challenge was getting rid of the old "implement a backlog of features" habit in place of Domain-Driven Design (DDD).

Our product is rapidly growing in size and complexity. Still, it is easy to maintain, support and develop - thanks to the solid foundation. The code is expressive and comprehensible. The components are easily interchangeable. By all means this product is better than anything previosly written by the team members.

I look behind and see all the gaps in my experience which did not allow solving business problem as elegantly. Dear fellow Pythonista, I hope this short articles series about Hexagonal architecture would help you to achieve the same without going through my struggle.

Dependency Injection

You know what Dependency Injection (DI) is, don't you? Sure you do, even if you can't recall the explicit definition. Let's see what are the pros and cons of this approach (a pattern, if you prefer).

Imagine that we need a function which sends ALARM messages to a message bus. The first iteration is:

from my_cool_messaging_library import get_message_bus()

def send_alert(message: str):
    message_bus = get_message_bus()
    message_bus.send(topic='alert', message=message)

Is there anything wrong with send_alert() function? It depends upon message_bus object, but this dependency is hidden from the caller. What if you'd like to use another message bus? How about the level of magic required to test this function? Did I just hear mock.patch(...)? Ladies and gentlemen, this is going south, let's try a different way:

from my_cool_messaging_library import MessageBus

def send_alert(message_bus: MessageBus, message: str):
    message_bus.send(topic='alert', message=message)

This small change in function signature is a big change of paradigm. The caller sees that send_alert() function depends upon MessageBus object (viva type annotations!). All implicit mocking bells and whistles are gone in favour of explicit and clean code. Sounds too good to be true? Have a look:

def test_send_alert_sends_message_to_alert_topic()
    message_bus_mock = MessageBusMock()
    send_alert(message_bus_mock, "Hacking attempt detected!")

    assert message_bus_mock.sent_to_topic == 'alert'
    assert message_bus_mock.sent_message == "Hacking attempt detected!"

class MessageBusMock(MessageBus):
    def send(self, topic, message):
        self.sent_to_topic = topic
        self.sent_message = message

But doesn't this mean that we have to pass an instance of MessageBus to send_alert() function on each call? Isn't that cumbersome?

send_alert(get_message_bus(), "Stackoverflow is down")

Let's try solving this problem by means of OOP:

class AlertDispatcher:
    _message_bus: MessageBus

    def __init__(self, message_bus: MessageBus):
        self._message_bus = message_bus

    def send(message: str):
        self._message_bus.send(topic='alert', message=message)

alert_dispatcher = AlertDispatcher(get_message_bus())
alert_dispatcher.send("Oh no, yet another dependency!")

Now AlertDispatcher class depends on an object of type MessageBus. We inject this dependency when creating an AlertDispatcher object by passing the dependency into constructor. We have wired (not coupled!) the object and its dependency.

At this point the focus switches from message_bus to alert_dispatcher. This component may be required in different parts of the application. Which means that there should be a global context which holds and provides the object. Let's first discuss the nature of components and components wiring.

Componential Architecture

We didn't emphasize dependencies types while speaking of dependency injection. But you might have guessed that MessageBus is just an abstraction, an interface or a protocol [1]. Somewhere the application defines:

class MessageBus(typing.Protocol):
    def send(topic: str, message: str):
        pass

There is also a simple implementation of MessageBus in the project. It stores the incoming messages in a list:

class MemoryMessageBus(MessageBus):
    sent_messages = []

    def send(topic: str, messagge: str):
        self.sent_messages.append(topic, message)

In the same manner, an abstract use case scenario is decoupled from a business-driven implementation:

# An abstract use case
class DispatchAlertUseCase(typing.Protocol):
    def dispatch_alert(message: str):
        pass

# A concrete implementation in a service.
# Note that a service may implement multiple related use cases at a time.
class AlertDispatcherService(DispatchAlertUseCase):
    _message_bus: MessageBus

    def __init__(self, message_bus: MessageBus):
        self._message_bus = message_bus

    def dispatch_alert(message: str):
        self._message_bus.send(topic='alert', message=message)

Next, let's add a controller which accepts HTTP requests and invokes DispatchAlertUseCase:

class ChatOpsController:
    ...
    def __init__(self, dispatch_alert_use_case: DispatchAlertUseCase):
        self._dispatch_alert_use_case = dispatch_alert_use_case

    @post('/alert)
    def alert(self, message: Message):
        self._dispatch_alert_use_case.dispatch_alert(message)
        return HTTP_ACCEPTED

Finally, let's connect all the pieces together:

from my_favourite_http_framework import http_server

def main():
    message_bus = MemoryMessageBus()
    alert_dispatcher_service = AlertDispatcherService(message_bus)
    chat_opts_controller = ChatOpsController(alert_dispatcher_service)

    http_server.start()

How would a rational and clear-minded developer react to this? She would call it overengineered and overcomplicated, no less! Which is indeed true. On the first glance, everything above fits into a short HTTP handler:

@post('/alert)
def alert(message: Message):
    bus = MemoryMessageBus()
    bus.send(topic='alert', message=message)
    return HTTP_ACCEPTED

Is it short and simple? Absolutely! Is it maintainable? Hardly. But why? Because the components are strongly coupled in the code. By blending everything in a single function we tightly couple domain workflow and message bus implementation. And that's half the trouble. The worst part is that we melted and buried business logic in technical details. Don't get me wrong, such code has the right to exist. Yet its existence in a rapidly growing application will soon end up in a maintenance hell.

Back to the componential architecture. What are the advantages?

Components are isolated and are not directly dependent. Instead they are wired via abstractions.
Every component works in certain boundaries and has a single responsibility.
This means that components are immensely testable: either in full isolation or in a combination using test doubles. There is no need to explain that testing isolated parts of a program is easier compared to testing it as a whole. Your TDD approach improves from inaudible "well, we do tests..." to sonorous "tests-driven and test-first development".
It is easy to substitute components, thanks to abstract dependencies. In the example above MemoryMessageBus could be replaced with DbMessageBus, FileMessageBus or anything else. The caller of message_bus.send(...) should not care.

Suddenly it dawns upon you: "That sounds like... SOLID?" Hell yeah! It is almost what Uncle Bob would call a Clean Architecture. I encourage you to read his article, before moving towards our end goal - Hexagonal architecture.

Architecture is about intent

One of my favourite Uncle Bob quotes on software architecture is "Architecture is about intent".

What do you see on this screenshot?

No wonder if you recognized a "typical Django application". Brilliant! Can you also tell what does this application do? If you can, my sincere congratulations - you are level 80 telepathist. Personally I have no clue whatsoever - that is a screenshot of a random Django application from Github.

Robert Martin develops the idea further. Take a look at a floor architecture plan and guess what the building is intended for:

Answer

That is a floor plan of Oodi Library in Helsinki.

I hope this tiny puzzle was easy to solve and you got the main idea: architecture should meet us at the gate, literally after git clone. Isn't it great when the source code is organized in such way that the purpose and the meaning of each file, class, function and any other object lies on the surface?

Hexagonal architecture of Ports and Adapters

"We use the Hexagonal architecture of ports and adapters" - how we start describing the architecture application to the new team members. It follows by showing a weird Cthulhu-like picture:

Alistair Cockurn, the inventor of "Hexagonal architecture" term explains that "hexagon" is not strictly necessary:

The hexagon is not a hexagon because the number six is important, but rather to allow the people doing the drawing to have room to insert ports and adapters as they need, not being constrained by a one-dimensional layered drawing.

The term ‘’hexagonal architecture’’ comes from this visual effect.

—Alistair Cockburn

You may also recall the terms like "Onion architecture" or "Ports and Adapters" mentioned in Uncle Bob's "Clean archcitecture" article. All these terms describe a way to organize application architectecture in layers, with dependecies directed to the center. (We noticed though, that hexagon and ports and adapters are much simpler to imagine and explain compared to more abstract terms like "clean" or "onion").

The outer layer of the application - adapters - interacts with the outer world. The inner layer - domain and domain services - contains the business logic. The connecting layer in between is application services. The components in each layer are designed to have low coupling and high cohesion [2].

Domain layer - is the heart of the application. The business rules are defined here. The names of classes, methods, functions, constants and other objects resemble those of the problem domain.

Application Programming Iterface (API) adapters are the components which convey commands and queries from the outer world to the application through API ports. Hence, API ports are interfaces through which the application is controlled and queried. In the Componential architecture described above, the DispatchAlertUseCase is an API port and ChatOpsController is an API adapter.

Service Provider Interface (SPI) adapters are the components which convey the application commands and queries to the outer world. Hence, SPI ports are interfaces through which commands and queries are passed to SPI adapters. In the Componential architecture, the MessageBus is an SPI port and its implementation MemoryMessageBus is an SPI adapter.

Application services are the conductors which glue domain and ports performing use case scenarios. They are supposed to orchestrate the whole transaction, e.g. load data for the domain to use, call functions which perform domain logic, persist the results and publish events.

There is one important rule to remember about hexagonal architecture: "Dependencies are directed from the outer layers to the inner center." In practice this means that adapters are aware of domain objects, but domain should not know of adapters or ports. It's also important to clarify that since adapters belong to the Adapters layer, they may call each-other directly. The cases for this are:

API adapter calls SPI adapter. - Think of HTTP GET requests which end up querying a database: It doesn't make sense to reach database through the domain layer making tons of objects mappings on the way. It's much more efficient for a controller to call the specific SPI port directly.
SPI adapter calls SPI adapter. - Think of an database adapter which calls other adapters and aggregates all the results before returning it.
API adapter calls API adapter. - Probably the least useful. Handy, if you need to redirect calls from one API adapter to the other.
SPI adapter calls API adapter - AVOID. Such calls is unnecessary and creates call loops in the application.

And... THAT IS IT! The basic principles of Hexagonal architecture of ports and adapters are surprisingly simple. This kind of architecture works well in application with complex problem domain. But it is an overkill, if everything you need is an "HTTP interface for a database".

We are now ready to dive into building a Hexagonal architecture -based Django applications. Stay tuned for part II.

References

[1]	Though PEP-544 Protocols are about structural subtying. Another option to mimic interfaces is to either use abstract classes with abstract methods.

[2]	Cohesion and coupling: Devopedia: Cohesion vs. Coupling Cohesion and Coupling: the difference

Turn the Ship Around!: A True Story of Turning Followers into Leaders - book review

2021-10-22T12:00:00+03:00

Whatever sense we have of thinking we know something is a barrier to continued learning.

(David Marquet, "Turn The Ship Around!")

"What does it take to turn a nuclear submarine around?" sprung up in my head, when I saw the cover of the book for the first time. It turns out, the book wasn't literally about turning the ship around. Or rather the definition of a "ship" was not a "nuclear submarine".

What does it take to challenge the decision making and leadership mechanisms in an organization as old and solid as U.S. Navy? Any why should these mechanisms change in the first place?

In December 1998, Captain David Marquet was ordered to take command over the SSN Santa Fe - a nuclear-powered fast attack submarine. She and her crew were to be ready for combat deployment in six months. That was not an ordinary task. Not only Captain Marquet previously spent half a year getting ready to command SSN Olympia - a submarine completely different from technical perspective, he was also assigned to command "The Santa Fe" - the worst-performing submarine in the navy at a time.

However, that rapid change of plan was not coincidental nor random. During his earlier career and training Marquet has experienced and identified the limitations of "One leader commands them all" or broadly "Leader - Follower" approach. The follower learns to rely on the leader to make all decisions rather than to fully engage with the work process to help make the organization run as efficiently as possible. One of the factors that makes this top-down leadership approach appealing is the induced numbness. It absolves subordinates of the hard work of thinking, making decisions, and being responsible and accountable. Santa Fe was a ship where everyone below the captain and the department heads had their brain shut off.

Captain Marquet was committed to "turn Santa Fe around". He describes how he needed to let go of old ideas to make room for new ones. He would make small evolutionary steps to build the bridge from "leader-follower" to the supreme "leader-leader" approach. The core of the leader-leader model is giving employees control over what they work on and how they work. It means letting them make meaningful decisions. The two enabling pillars are competence and clarity.

Marquet discovered that leader-leader structures are significantly more resilient, and they do not rely on the designated leader always being right. Moreover, they spawn additional leaders throughout the organization naturally.

"Turn The Ship Around!" is a story of how a mediocre crew that focused on avoiding mistakes became a crew thriving to achieve excellence. Marquet does a fantastic storytelling, fusing the narrative with practical analysis and guide. Every chapter focuses on a set small set of challenges that the crew and the captain of Santa Fe had to face and the solution they've come up with. All solutions are further described as practical "mechanisms".

Ultimately, this book is a call to action, a manifesto, for all those frustrated workers and bosses for whom the current leadership structure just isn’t working.

(David Marquet, "Turn The Ship Around!")

A crew? - A Tribe!

What I personally find great about Santa Fe's journey is that it resembles a journey through the stages of tribal relationships as described in "Tribal Leadership" [1]. The crew of Santa Fe, was at stage 2: "My life sucks" when Captain Marquet took over. The crew wanted change, even if they didn’t know quite how to do it. And since the leader-follower approach in place, they always had the boss, the ultimate decision maker - merely the captain to blame for every problem.

Captain Marquet was able to to tap into the existing energy of the command, discover the strengths, and remove barriers to further progress. Marquet's determination, commitment and leadership helped the crew to reach the highest tribal stages: Stage 4 "We're great" - and ultimately Stage 5 "Life is Great".

Thoughts

A good leader always remembers what needs to be done, follows up the execution to make sure that it was unimpeachable. A great leader doesn't have to remember everything or follow up every step the team makes. Instead, the leader employs practices which ensure that the team can perform flawlessly without a leader being present. He or she are the part of the team where every member trusts each-other to do their job. That trust comes from the team being competent, being able to make decisions, clearly understanding the goals on all levels - from personal to organizational. That trust means that every single member of the crew is a leader.

What kind of a leader are you? A leader that tells what should be done, controls the execution hides the bigger picture since "it's not their concern"? A leader who wants to be followed? Or a leader who clarifies the bigger picture, helps building competence and lets the team figure out the best way to perform the job? A leader who trusts and is trusted in return? A leader who spawns leaders?

Thanks to David Marquet I made my mind. How about you?

[1]	Tribal Leadership: Leveraging Natural Groups to Build a Thriving Organization (Dave Logan, John King, Halee Fischer-Wright)

Accelerate: The Science of Lean Software and DevOps - book review

2021-06-23T12:00:00+03:00

Longing for improvement

Are you familiar with the gut feeling that the software development practices around you could be better? Sometimes the desired improvements are small and obvious. They are not very hard to implement and have a quick return of investment.

For example, adding a vulnerability scanner to a CI pipeline prevents using dependencies with known CVEs. A code formatter which applies the rules for every source file save operation - brings determinism and keeps team members from nagging about style. Even a low disk space alarm configured long time ago could save you from major embarrassment and business losses.

Other improvements are not that easy. Core dependencies being dramatically outdated since new versions break compatibility? Painful deployment with downtime lasting for days? Features being developed by single developers for months and living in a branch for all the that time? Backlog growing indefinitely. Bugs, bugs and even more bugs pouring in every day? I'd be glad to tell that the above is a horror story, and it never happens in reality. But you and I both know the sad truth. Worse - when everyone in the development department agrees that something should be done, but the time is never right. Thousands of software businesses have to deal with stories like this every day.

There is no silver bullet to fix all the issues in your organization. But we are not armless in the matter. What if you don't need to guess what should or could be improved? What if there is a solid scientific study on practices of high performing technology organizations?

Accelerate

Accelerate by Nicole Forsgren, Jez Humble and Gene Kim is a book concluding a four-year research into capabilities and practices of software development practices. There were over 23000 survey responses from over 2000 unique and diverse organizations from around the world collected during the research. It uncovered 24 key capabilities that drive improvement in software delivery performance. Some of these capabilities are: continuous delivery and trunk-based development, loosely coupled architecture and customer feedback, work-in-progress limits and transformational leadership.

"Accelerate" doesn't have all the answers and is not a guidebook. But it clearly describes the differences between low and high -performing organizations and teams. It explains in detail how capabilities are related and how each capability affects performance and how leadership is essential for transformation. Being said, not every capability is easy to measure. You can quickly estimate how long it takes to deliver a committed piece of code to the customer. But how do you measure culture?

The second part of the book may help you find the answer. While the first part is about the study results, it's the second part which explains the methods used in the research and dives deep into the science behind conducted surveys.

The third part is a life example story of ING Netherlands. It tells how leadership, management, and team practices work together bringing a traditional banking business into the digital age.

Verdict

There is no verdict. This book is truth based on facts. Although not a long one, my career has been among low, average and great performing teams. I mentally applied everything said in the book to my experience, and boy it is accurate! There is really no excuse for an organization to ignore this study or delay applying the practices ASAP. If you want to stay in the game, you have to Accelerate.

Alarms and dashboards

2021-06-04T12:00:00+03:00

Dashboard is a great tool to show information in a human-friendly manner. Yet, it is useless as a monitoring tool. Hanging a big shiny screen in the middle of the room, with bars, plots, charts and lots of other cool stuff is spectacular. But it won't help you to catch an issue in a timely manner. That is because you can't watch the screen all the time, don't you?

The solution is obvious - alarms. Alarms are meant to warn you when the situation is exceptional or something is about to go south. When a service fails. When your disk space is running low. When a request processing time takes too long. When there is an error in logs.

And then, you actually need a dashboard to query the history of the parameters in trouble. You'd want to see the trend of free disk space - did it happen suddenly, or it's been filling up in a linear manner? Maybe you have to wipe some old data? Perhaps there was a temporal network error? How many times that error happened lately?

Alarm can be dangerous. When you get too many alarms they slip through. Too many "usual" alarms turn into annoying buzz which is soon ignored. The solution is to temporarily disable the unnecessary alarms. Do keep in mind that an alarm could be disabled only when you are actively fixing the problem.

This is nothing new. Have you seen Ron Howard's "Apollo 13" film? Remember the CO2 alarm indicator lighting up and the crew checking the CO2 levels on the gauge?

Can you imagine the crew constantly checking those values to make sure that CO2 levels are acceptable instead? What about other critical indicators? Of course not.

There are alarms to draw attention and dashboards to help dealing with the alarm as fast as possible.

Concrete Settings: a new way to manage configurations in Python projects

2020-07-11T13:06:00+03:00

After two years of developing a hobby project, I am proud to announce Concrete Settings - a new configuration management library for Python projects.

Concrete Settings was born as an effort to improve configuration handling in a huge decade-old Django application with exceedingly bloated settings.py. Remember that settings.py starts with Django configuration only? What could go wrong if you start adding application settings to the file? One setting, two settings, four...

FEATURE_X = 123
FEATURE_Y_ENABLED = True
FEATURE_Z = 'square'

In time there were so many settings, that developers loose track of them. The documentation was scattered around issues tracking tool and outdated wiki pages. There was little or no settings validation - sometimes customers were getting HTTP 500 - Internal Server Error due to typo in a setting value.

Concrete Settings does its best to tackle these problems.

Why?

I was thinking a lot of a developer's and end-user's experience. How would Concrete Settings library differ from the existing solutions?

After months of experimenting and juggling ideas, the following ideas laid the foundation:

Settings with their validation rules and documentation are defined in classes.
Validation can happen independently from the application, allowing early fails.
Settings definitions can be nested and mixed.
Application can read settings from developer-defined sources, like yaml / json / python files, environmental variables etc.
All these are based on Python metaclassing and other capabilities to minimize the amount of code written by an end-developer.

From an end-user's perspective:

Settings are defined in sources. User works with application.yml or configuration.json environmental variables or custom solutions instead of overbloated settings.py.
An immediate and verbose feedback is received if settings values are invalid.

Enough words, show me the code!

Let's take a look at a simple example and see the bells and whistles (aka cool features) that are in the box.

# app_settings.py

from concrete_settings import Settings
from concrete_settings.contrib.sources import EnvVarSource


class AppSettings(Settings):

    #: Whether debug mode is enabled
    DEBUG: bool = False

    #: HTTP server host
    HOST: str = '127.0.0.1'

    #: HTTP server listening port
    PORT: int = 8080

# Verify settings definition and construct settings object
app_settings = AppSettings()

# Read settings from sources
app_settings.update('settings.yml')
app_settings.update(EnvVarSource())

app_settings.is_valid(raise_exception=True)

print(f"Server HOST is {app_settings.HOST} ({AppSettings.HOST.__doc__})")
print(f"Server PORT is {app_settings.PORT} ({AppSettings.PORT.__doc__})")
print(f"DEBUG is {app_settings.DEBUG} ({AppSettings.DEBUG.__doc__})")

An end-user is happy with the default HOST value, but wants to override the rest of the configuration in settings.yml

PORT: 8081

and set DEBUG=true via an environmental variable.

The output in this case would be:

Server HOST is 127.0.0.1 (HTTP server host)       # default value
Server PORT is 8080 (HTTP server listening port)  # settings.yml
DEBUG is True (Whether debug mode is enabled)     # environmental variable

Document via sphinx-style docstrings

Concrete Settings allows documenting settings via #: Sphinx-style comments:

#: HTTP server host
#: Format: IPv4 address encoded in a string
HOST: str = '127.0.0.1'

Documentation can be also passed in an explicit Setting definition.

Let's compare these explicit and implicit definitions:

class AppSettings(Settings):

    #: HTTP server port
    PORT: int = 8080

    HOST = Setting(
        '127.0.0.1',
        type_hint=str,
        doc='HTTP server host'
    )

Explicit or implicit - the extracted docstring, is stored to Setting.__doc__:

print(AppSettings.PORT.__doc__)
print(AppSettings.HOST.__doc__)

However, wouldn't you agree that PORT documentation is way more pleasant to read and maintain?

Validate settings early and...

What if a user makes a typo and the supplied port is not an integer?

Let's change a value in settings.yml from the first example:

PORT: 8081

PORT: "8081"

Running the first example again would raise an exception:

concrete_settings.exceptions.ValidationError: PORT: Expected value of type `<class 'int'>` got value of type `<class 'str'>`.

The validation error message is generated by the default ValueTypeValidator.

... add custom validators with style!

Let's craft a validator which checks that port number is equal or greater than 8000:

from concrete_settings import Settings, ValidationError, validate

def port_validator(value: int, **ignore):
    if not 8000 <= value <= 65535:
        raise ValidationError('Expected value in range 8000..65535')

class AppSettings(Settings):

    #: HTTP server listening port
    PORT: int = 8080 @validate(port_validator)  # <--- I know you are scrolling, but have you noticed this?

app_settings = AppSettings()
app_settings.update('settings.yml')

print(app_settings.is_valid())
print(app_settings.errors)

Here we use a decorator-like syntax of so-called behaviors (actually it's a matrix multiplication operator in this case :).

Let's test it out by changing PORT value in settings.yml to 80:

PORT: 80

The result of running the snippet above is

False
{'PORT': ['Expected value in range 8000..65535']}

If you are still uncomfortable with @behaviors - there is an explicit way to add validators to settings. Simply pass validators to Setting constructor:

class AppSettings(Settings):

    #: HTTP server listening port
    PORT: int = Setting(8080, validators=(port_validator,))

Helpful hierarchies

I never liked settings names like DB_HOST_ADDRESS. Why have flat settings names, with feature, sub-feature, configuration, sub-configuration... in them?

Concrete Settings prefers composition over flat-style settings declaration, though it provides both extension and grouping mechanism for settings. For example, let's define database and logging settings in separate classes:

from concrete_settings import Settings

class DBSettings(Settings):
    USER = 'alex'
    PASSWORD  = 'secret'
    SERVER = 'localhost@5432'

class LoggingSettings(Settings):
    LEVEL = 'INFO'
    FORMAT = '%(asctime)s %(levelname)-8s %(name)-15s %(message)s'

class AppSettings(Settings):
    DB = DBSettings()
    LOG = LoggingSettings()

app_settings = AppSettings()
print(app_settings.LOG.LEVEL)

At first glance, there is nothing special about this code. What makes it special and somewhat confusing is that class Settings is a subclass of Setting! Hence, nested Settings behave and can be treated as Setting descriptors - have validators, documentation or bound behavior.

Additionally, validating top-level settings automatically cascades to all nested settings. The following example ends up with a validation error:

from concrete_settings import Settings

class DBSettings(Settings):
    USER: str = 123
    ...

class AppSettings(Settings):
    DB = DBSettings()
    ...

app_settings = AppSettings()
app_settings.is_valid(raise_exception=True)

Traceback (most recent call last):
    ...
concrete_settings.exceptions.ValidationError: DB: Expected value of type `<class 'str'>` got value of type `<class 'int'>`

Finally, the settings can be read from a similarly nested structure. For example settings.json:

"DB": {
    "USER": "admin"
}

or environmental variable DB_USER.

In a retrospective

This project took a long time to develop. What I did right was no releasing an unfinished and buggy library. That is probably also what I did wrong. Trying to polish everything before the first public release without getting any users feedback is not the best way to go. Hopefully, there will be feedback and the project would steer towards its users needs and wishes.

Let's start!

Install it via pip:

1	pip install concrete-settings

and check out the documentation!

Is classical music performed too fast?

2018-04-25T21:00:00+03:00

Every now and then, being at home with kids I put on some music they've never heard before. I always try different genres - rock, blues, jazz, even hip hop. This time I've chosen one of my favorites, the renowned classic of classics: Antonio Vivaldi's Four Seasons.

But the first notes of the "Spring" made me shiver. "This doesn't sound right! It sounds... too fast?" Seems that the first search results in Spotify yielded the recording of 1999, conducted by Fabio Biondi, Europa Galante orchestra.

But there were more to discover. "Sinkovsky Plays and Sings Vivaldy" was the next search result. Oh my! What is your rush man? It is played so fast that makes enjoying the music impossible. It felt like I've been given the best of the world's delicacies and have only two minutes to taste them all. Don't get me wrong - the technique is impressive! But where is the soul in all that?

And there are many more performances that sound too paced. For example La Serenissima / Adrian Chandler (2015), and even Nigel Kennedy (1989).

I made a simple experiment: searched for "four seasons summer g minor" in Spotify and sorted the results by duration. After manually narrowing the results to the third part of the "Summer" - I found myself gazing upon a 45 seconds difference between the slowest (Takako Nishizaki, Capella Istropolitana, Stephen Gunzenhauser) and the fastest (Giuliano Carmignola, Venice Baroque Orchestra) performances! From 3m 07s to 2m 22s!

My overall impression of listening to the faster-paced performances was somewhat of standing in the middle of a stock exchange at a rush hour. Instruments are screaming, vulgarly interrupting each-other, not letting one another to finish the parts.

Is it the speed of our age that influences tempo that much? Or is it me, being used to the slower performance of "Four Seasons"?

Blogging with Pelican

2017-04-25T21:00:00+03:00

Pelican is a static site generator, that requires no database or server-side logic. It is also an amazing blogging platform and as of April 2017, my blog is powered by Pelican. At this point, people stare at me with "ORLY?" expression on their faces. Why not Wordpress, Tumblr, Facebook or any other 3rd-party blogging platform?

I tried Google Blogger a decade ago. Back then, the editor was extremely basic and sometimes one had to edit HTML manually to get the required visual representation. Then there was Wordpress. It was rather nice, with all its integrated features and thousands of free themes and plugins. However, self-hosted Wordpress can be a painful experience. Even with automatic updates I ended up with hacked blog sending spam emails in background.

Around 2010 I wrote a simple blogging engine in Python/Django. That was a nice programming exercise (you've got to try it at least once in a lifetime!) but it required even more maintenance: upgrading Django, fixing bugs, adding new features, fixing styles in theme etc.

Free time was becoming luxury, so I started looking for a solution which would take care of all the routine, which requires little or no maintenance at all and which would be preferably written in a familiar programming language.

Tinkerer was the first static blog generator I encountered. It works fine, however there are some configuration limitations e.g. the output files and directory structure is generated in a certain hard-coded way. Pelican on the other hand has a very wide range of configurable options and provides great extensibility API for developers. As of April 2017 there are over 120 various 3rd-party plugins and over 100 user-made themes. Needless to say, Pelican has a large and active community!

That is something one might expect from a good blogging platform. Is there something what makes blogging via Pelican a remarkable experience for computer geeks?

For starters, you write the content in reStructuredText, Markdown, or AsciiDoc formats (btw, this blog is written in reStructuredText). The code highlight comes right out of the box. You can publish the content in different languages. You can even have multiple authors in a blog! Static output means, that there is very little or no maintenance and no more blog engine upgrades because of yet-another-critical-security-issue. Thanks to the community, I don't have to worry about the themes and once the static output is generated, it can be published literally anywhere, on any web server.

So begins the new era of my blog. Powered by Pelican!

P.S.

Thanks to Alexandre Vicenzi for Flex theme!

The ultimate guide to Python decorators

2017-01-22T23:00:00+02:00

Contents

Introduction to decorators
Decorators with arguments
Decorating methods
- Self and method decorators
Decorating classes
- Example: Registering a plugin
Finale
References

Introduction to decorators

Historical background

It all started with discussion on how-to turn the following syntax into something nice:

class C(object):
    def info():
        return 'This is class C'

    info = staticmethod(info)

This is an absolutely valid code, however declaring info as a static method requires wrapping it as info = staticmethod(info). Inconvenient, isn't it?

In case you are not familiar with staticmethod() [1]: it is a builtin function which accepts a function as an argument and creates a static method for the class. This is all done via descriptors protocol, but no need to go that deep today. At this point, info() can be called as

>>> C.info()
This is class C

After months of discussion, analysis and research PEP 318 -- Decorators for Functions and Methods was finally accepted. Since Python 2.4 decorators turned the code above into

class C(object):

   @staticmethod  # this is a decorator
   def info():
       return 'This is class C'

Geting to know decorators

A decorator is a syntactic sugar of calling a function which accepts another function as an argument and returns either the same function or another callable. Note: this definition is not complete and will be expanded further in the article.

Take a look at these nice decorators examples:

# Django web framework view decorators
@login_required
@require_GET
def my_view(request):
    # I can assume now that only GET requests
    # from a logged in user make it this far.
    # ...


# Flask web microframework routing
@app.route("/")
def hello():
    return "Hello World!"


# Standard library unit tests
class MyTestCase(TestCase):

    @skipUnless(sys.platform.startswith("win"), "requires Windows")
    def test_windows_support(self):
        # windows specific testing code
        # ...

No doubt, decorators improve the readability of code dramatically.

Closures

Let's take a look at a practical example to see how decorators work from the inside. Imagine that you want to log all the arguments' values and the return value of a send_message(sender, receiver, text='') function calls to stdout. This can be simply done via print() as follows:

def send_message(sender, receiver, text=''):
    print('send_message was called')
    print('The arguments are sender={}, receiver={}, text={}'
          .format(sender, receiver, text))
    ...
    print('The return value is "SENT"')
    return 'SENT'

However, consider that you now have to do the same for dozen other functions, all with different arguments and return values. First, let's do this without decorators:

def send_message(sender, receiver, text=''):
    ...

def send_message_with_log(*args, **kwargs):
    ret = send_message(*args, **kwargs)
    print('send_message was called')
    return ret

# Important substitution
send_message = send_message_with_log

To make send_message_with_log more generic, a technique called closure is used as follows:

def send_message(sender, receiver, text=''):
    ...

def log_fn(f):
    def wrapper(*args, **kwargs):
        ret = f(*args, **kwargs)
        print('{} was called'.format(f.__name__))
        return ret
    return wrapper

send_message_with_log = log_fn(send_message)
send_message = send_message_with_log

Here, value of f is stored in lexical scope of wrapper, which means that after send_message_with_log = log_fn(send_message), send_message_with_log variable has a value of the wrapper function with send_message stored in f variable. Finally, the original send_message is substituted. In a single line:

send_message = log_fn(send_message)

Calling send_message will produce the following output:

>>> send_message()
send_message was called

Note, that log_fn is now generic and can be used to log any functions' calls.

Your first decorator

Now we are ready to turn log_fn into a decorator. Let's write it down and analyze its code line-by-line:

def log_fn(f):
    def wrapper(*args, **kwargs):
        ret = f(*args, **kwargs)
        print('{} was called'.format(f.__name__))
        ...
        return ret
    return wrapper

@log_fn
def send_message(sender, receiver, text=''):
    ...
    return 'SENT'

1. def log_fn(f): - a decorator is defined. It is a function which accepts a single argument - another function.

2. def wrapper(*args, **kwargs): - an internal function (a closure technique) which wraps calls to function f with custom logic. For a moment let's skip the body of wrapper() and proceed to the end of log_fn().

7. return wrapper - the return value of the decorator. It returns a callable, which is usually an internal wrapper function.

9. @log_fn - at this point decorator is called with send_message as an argument. This is equivalent to:

def send_message(sender, receiver, text=''):
    ...

# send_message is substituted
send_message = log_fn(send_message)

Pay attention to the wrapper function, which is returned by log_fn() and assigned to send_message. What happens to the original send_message? Let's get back to the body of wrapper.

3. ret = f(*args, **kwargs) - here the original f function is called. At this point, f == send_message.

4. print('{} was called'.format(f.__name__)) - a simple logging to stdout, which prints "send_message was called" each time the function is called

5. return ret - the return value of the original function call is returned

Finally, let's update the decorator to make it more useful and fulfill the original requirements of logging the function call with the arguments and the return value:

def log_fn(f):
    def wrapper(*args, **kwargs):
        ret = f(*args, **kwargs)
        sargs = ', '.join(repr(arg) for arg in args)
        skwargs = ', '.join('{}={}'.format(k, repr(v))
                            for k, v in kwargs.items())

        print('{name}( {sargs}, {skwargs} ) => {ret}'.format(
              name=f.__name__,
              sargs=sargs,
              skwargs=skwargs,
              ret=repr(ret)
        ))
        return ret
    return wrapper

So, the output of calling the decorated send_message is:

>>> send_message('alice', 'bob', text='Hello!')
send_message( 'alice', 'bob', text='Hello!' ) => SENT

Examples

Decorator syntax allows quick and clear extension of the wrapped functions. Its beauty is in simplicity: by writing a single line declaration one can embed powerful functionality while keeping the code clean and neat.

Let's explore decorators possibilities by writing a couple of decorators which might be handy in development.

debug_on_error

import pdb;

def debug_on_error(f):
    """Drop to pdb debugger on exception"""
    def wrapper(*args, **kwargs):
        try:
            return f(*args, **kwargs)
        except Exception as e:
            pdb.post_mortem()
    return wrapper

# usage
@debug_on_error
def div(x, y):
    return x / y

This decorator allows dropping into debugger the moment an error occurs in a function call. For example running the following code in ipython3:

>>> div(5, 0)
<ipython-input-2-cd786e30d343>(3)div()
-> return x / y
(Pdb) ?

timeit

import time

def timeit(f):
    """Measure an execution time of the wrapped function"""
    def wrapper(*args, **kw):
        tick = time.time()
        ret = f(*args, **kw)
        tock = time.time()

        print('{}() execution time: {} s.'.format(f.__name__, tock - tick))
        return ret

    return wrapper

# example
@timeit
def sum_up_to(x):
    res = 0
    for i in range(0, x):
        res += i
    return res

This decorator prints the amount of seconds passed between function call and return events. It is helpful to measure a function's execution time and find performance bottlenecks. Let's run an example:

>>> sum_up_to(int(10e6))
sum_up_to() execution time: 0.0465855598449707 s.

Chaining decorators

Imagine that you want to log the function call with log_fn and at the same time profile it with timeit decorators. Just chain the decorators! For example:

@timeit
@log_fn
def sum_up_to(x):
    ...

This is the same as:

def sum_up_to(x):
    ...

sum_up_to = timeit(log_fn(sum_up_to))

There are virtually no limits on the amount of decorators decorating a function (beside the stack limit) however you may want to keep that number low for code readability.

Decorators with arguments

So far we've been exploring simple decorators and their usage, which is just the tip of the iceberg. For example, decorators accept arguments the same way as any ordinary function does. This makes them even more powerful and versatile. But before jumping there, let's take a look at an important technique which always follows decorators source.

Wrapping functions properly

Let's call sum_up_to() from the last example, where it has been decorated with both timeit and log_fn. The printed result may look a little bit unexpected:

>>> sum_up_to(int(10e6))
sum_up_to( 10000000,  ) => 49999995000000
wrapper() execution time: 0.4475877285003662 s.

Did you notice that the name of the function in the output is not sum_up_to but wrapper? This behaviour is not a bug. Take a minute to find out why this happens.

Let's unwrap the decorator call:

# Python unwraps the decorator call into this:
sum_up_to = timeit(log_fn(sum_up_to))

# First, log_fn(sum_up_to) is executed, which returns
# log_fn's internal `wrapper` function:
log_fn_wrapper = log_fn(sum_up_to)

# Hence, `timeit` is called with `log_argument_wrapper`
# as an argument:
sum_up_to = timeit(log_fn_wrapper)

The problem is that wrapper does not mimic the original function. To overcome this, attributes like __name__, __doc__, __module__ etc. from the original function should be copied to wrapper:

def timeit(f):
    def wrapper(*args, **kw):
        ...

    wrapper.__name__ = f.__name__
    wrapper.__doc__ = f.__doc__
    wrapper.__module__ = f.__module__
    return wrapper

Though this code works fine, it means that one would have to write the same attributes copying routine in every decorator. Sounds familiar? Indeed, why not write yet another decorator, which does the attributes copying? Guess what, the standard library already contains a function wraps() [2] which gracefully handles this issue:

from functools import wraps

def timeit(f):
    @wraps(f)
    def wrapper(*args, **kw):
       ...
    return wrapper

def log_fn(f):
    @wraps(f)
    def wrapper(*args, **kwargs):
        ...
    return wrapper

@timeit
@log_fn
def sum_up_to(x):
    ...

>>> sum_up_to(int(10e6))
sum_up_to( 10000000,  ) => 49999995000000
sum_up_to() execution time: 0.9093782901763916 s.

Wonderful! Now timeit() prints the name of the decorated function. One thing you have probably noticed about wraps is that it actually accepts an argument! Now that you know how-to wrap a decorator properly, let's find out how to pass arguments to decorators.

Decorators and arguments

Decorators are functions, so they accept arguments the same way as any other function does. For example, let's update the timeit decorator to log the function calls which take more than N seconds to complete:

@timeit(0.1) # 100 milliseconds
def sum_up_to(x):
    ...

How to implement such a decorator? Remember that by definition, a decorator accepts a single argument only (i.e. the decorated function), thus accepting an integer in the example above breaks the rules. The trick is that it is not timeit which decorates sum_up_to, but rather the result of timeit(0.1) call:

def timeit(limit):
    def decorator(f):
        @wraps(f) # wrap properly
        def wrapper(*args, **kw):
            ...
        return wrapper
    return decorator

What happens when a function e.g. sum_up_to is decorated as @timeit(0.1)? First, Python calls timeit(0.1) which builds a decorator and returns it. Then, Python decorates sum_up_to with the obtained decorator. It all can be decomposed as follows:

def timeit(limit):
    ...
    return decorator

timeit_decorator_100ms = timeit(0.1)

@timeit_decorator_100ms
def sum_up_to(x):
    ...

There are no limits on decorators' arguments design. For example, a version of timeit which accepts two arguments: a limit and a printer function:

import logging
log = logging.getLogger(__name__)

def timeit(limit, printer_fn=print):
    def decorator(f):
        @wraps(f)
        def wrapper(*args, **kw):
            ...
            if diff > limit:
                printer_fn('{}() execution time: {:.2} s.'.format(f.__name__, diff))
            return ret
        return wrapper
    return decorator

@timeit(100, printer_fn=log.warning)
def sum_up_to(x):
    ...

Class as a decorator

Python allows taking the idea of decorators with arguments even further. Classes can be effectively used to reduce the amount of nested functions and to improve the code of complex decorators.

There is no magic in using classes: first, Python creates an object from a class and then calls it (i.e. invokes object's __call__() method) to decorate a function.

Let's rewrite timeit(limit, printer_fn) as a class. The arguments are passed through __init__() method and are stored as class attributes. The __call__() method returns a wrapped function:

class timeit:
    def __init__(self, limit, printer_fn=print):
        self.limit = limit
        self.printer_fn = printer_fn

    def __call__(self, f):
        @wraps(f)
        def wrapper(*args, **kw):
            tick = time.time()
            ret = f(*args, **kw)
            tock = time.time()
            diff = tock - tick
            if diff > self.limit:
                self.printer_fn('{}() execution time: {:.2} s.'.format(f.__name__, diff))
            return ret
        return wrapper

@timeit(0.01)
def sum_up_to(x):
    res = 0
    for i in range(0, x):
        res += i
    return res


  sum_up_to(int(10e6))
  >>> sum_up_to() execution time: 1.0 s.

Function-decorators vs. class-decorators

PEP-20 states that

There should be one-- and preferably only one --obvious way to do it. Although that way may not be obvious at first unless you're Dutch.

Unless writing decorators on a regular basis, one would wonder, what are the benefits of using function-decorators vs. class-decorators and vice-verse? Function-decorators are

Simple and clear when no decorator arguments are required.
Even simpler when a decorator returns the original function, not a wrapper.

However class decorators

Allow better decomposition of complex decorators.
Provide clearer syntax to store state of a decorator both local and global.

The choice whether to write a decorator as a function or a class always depends on the case or personal preferences. As a rule of thumb - if the logic behind the decorator is complex, then go on with a class. Otherwise, pick a function approach.

Decorating methods

The syntax of decorating class methods is similar to decorating functions:

class Calculator:

    @log_fn
    @timeit
    def sum(self, x, y):
        ...

You are most probably familiar with the built-in property [6], classmethod [7] and staticmethod [1] decorators. For example here a calculator's epsilon is a read-only property:

class Calculator:
    _eps = 0.00001

    @property
    def eps(self):
        return self._eps

calc = Calculator()
print(calc.eps)
>>> 0.00001

calc.eps = 0.0001
>>> Traceback (most recent call last):
>>> File "<stdin>", line 1, in <module>
>>> AttributeError: can't set attribute

Self and method decorators

Method decorators may have an explicit access to self. Consider this:

def decorator(m):
    # WARNING, do not do this!
    def wrapper(self, *args, **kwargs):
        return m(self, *args, **kwargs)
    return wrapper

However this is a dangerous construction. A generic decorator should not access self argument of the decorator method. As a matter of fact, a generic decorator should not know anything about the decorated callable, whether it is a function, a method, or a wrapper returned by another decorator. Otherwise it breaks the universal protocol and for example makes impossible chaining the decorators in an arbitrary order.

Decorating classes

When function decorators were originally debated for inclusion in Python 2.4, class decorators were seen as obscure and unnecessary thanks to metaclasses. After several years' experience with the Python 2.4.x series of releases and an increasing familiarity with function decorators and their uses, Guido van Rossum aka BDFL and the community re-evaluated class decorators and recommended their inclusion in Python 3.0 [3].

The use cases however are not that obvious [4]. That is because which could be done via class decorators, could be as well done via good old metaclasses. It seems that the usage case is registering a class in any kind of chain like a plugins system.

A hint of such usage, is the standard library's unittest.skip* [5] functions, for example:

@unittest.skipUnless(settings.LOGGING_ENABLED):
class LoggingTest:
    def test_smoke(self):
        ...

This test will be executed, only if a Logging feature is enabled in application's settings.

Consider another example: a media player application which supports plugins:

from player.plugins import Plugin, register


class AACPlugin(Plugin):
    ...

register(AACPlugin)

# This plugin is still experimental and is not
# registered in Player's plugins system
class AACExperimentalPlugin(Plugin):
    ...

Wouldn't it be easier to write

@register
class AACPlugin(Plugin):
    ...

Also, why not automatically register the plugins which inherit from Plugin class? Gotcha! What if you do not want to register the plugin yet, whether it is experimental, or incomplete, or for any other reason? That's where decorators allow expressing developer's intention in a clear and non-ambiguous manner.

Example: Registering a plugin

Let's consider the case described above : A media player application has a plugins system which extend its basic capabilities. A player.plugins.register() call is used to register an arbitrary class as a plugin. An API user should not care, about register() 's internals, however in this case, register() simply validates the classes and stores them in a global _plugins list:

 # --- in player/plugins.py --- #

 _plugins = []

 class Plugin:
     '''Plugin base class'''

 def register(cls):
     if not isinstance(cls, Plugin):
         raise TypeError('Cannot register a class as a Plugin: wrong type {}'.format(type(cls)))

     _plugins.append(cls)

     return cls

# --- usage in 3d-party module --- #

from player.plugins import Plugin, register

@register
class AACPlugin(Plugin):
    ...

Finale

This is it! Thank you for reading, I hope you enjoyed! Please leave a comment, ask a question, or just share this article with anyone who is still lost in the world of Python decorators :)

References

[1]	(1, 2) staticmethod

[2]	@functools.wraps

[3]

PEP 3129

[4]	Class decorators in Python: practical use cases

[5]	unittest.skip

[6]

property

[7]	classmethod

My story of caffeine withdrawal

2015-05-31T12:00:00+03:00

It all started...

It all started long before I turned two years old. I started drinking tea! Properly saying, I drank whatever was in the bottle and no one asked my opinion. Drinking tea is an Azeri tradition, you will be offered tea everywhere, at any meeting, gathering or reception, party or mourning and many other occasions. Speaking of, that is not a tea from a teabag, no sir! It is a properly brewed tea almost at a boiling point of water. The taste is great and is fully-flavoured.

I have been drinking coffee since university. Tea was not enough to sustain my brain during the nights before exams (how stupid that sounds today!). Then, I turned to energy drinks. What a perfect example of how cheap caffeine tablets are sold in a form of overpriced liquid!

Nevertheless, tea was my favourite in the family of hot drinks. I drank coffee occasionally and had an energy drink or two, couple times a month.

Getting hooked up

Lately I noticed obtaining a new "drinking coffee at work" habbit. I had a cup of coffee in the morning right after logging in. I had another cup right after lunch and usually the third cup just before leaving home.

The side effects came out rather quickly: Drinking coffee for productivity turned into drinking to feel normal. I could not start a day without coffee anymore. I started getting sleepy in the evenings, as I did not drink any coffee at home.

Finally, I decided to stop drinking anything with caffeine in it.

Detox

There are great caffeine detoxing advises on caffeineinformer.com. I chose the The Cold Turkey Method i.e. ceased consuming caffeine at all. In a retrospective, that was a naughty move. The detox process went as follows:

Day I: Running nose and irritating headache. Hard to think and hard to move. I had to take a doze of ibuprofen painkiller to ease the symptoms.
Day II: Headache and running nose continues. Severe muscles pain started to build around back of the neck. Another single doze of ibuprofen. I'd take paracetamol instead, but most of those painkillers are mixed with caffeine.
Day III: Headache is not as bad, muscles are still suffering. No painkillers today.
Day IV: Headache is almost over. Nose is lightly running. Muscles pain starting to fade.
Day V: Feeling much better. Headache is completely gone, I had to do some exercise to completely get rid of muscles pain, or rather turn it into a rewarding post-exercise itch.

That was it! My body got back to normal, I did not need coffee or tea to wake up or feel good. My brain was completely cured from the caffeine addiction.

Afterwards

That was not the end of the it yet! I opted to drink only water for three more weeks. It was almost summer, the weather was warm thus, no need to warm up by drinking something hot. These weeks went fine. I only noticed consuming more water than before. Eventually I started drinking tea again. Coffee was left as a "brain doping" which I'm willing to take only when it is really required.

Do you have any withdrawal experience? I'd be glad to hear your story! If you want to try (though I'd hardly recommend it :) check caffeineinformer.com for great advises and more valuable information on human body and caffeine interaction.

Python nightmares: Implicit this

2015-05-11T22:00:00+03:00

My first line of defense against such heresy is PEP 20 - The Zen of Python, which states that Explicit is better than implicit. Next comes the fact that Python class methods are in functions, bound to a class instance via descriptor protocol. If this sounds complex, I strongly encourage you to check the great Descriptor HowTo Guide from the official documentation. In general, passing function arguments through magic variables is awful (think of Perl with its $)! Why pass an object through a magical this?

Finally, it is possible to hide self and pass this implicitly. The easiest way is to do it by decorating a method:

# NOTE: all code examples are done in Python 3

def add_this(f):
    def wrapped(self, *args, **kwargs):
        f.__globals__['this'] = self
        return f(*args, **kwargs)
    return wrapped

class C:
    name = 'Alex'

    @add_this
    def say(phrase):
        print("{} says: {}".format(this.name, phrase))

c = C()
c.say('Can you believe it? There is no `self` here!')

Output:

1	Alex says: Can you believe it? There is no `self` here!

As you can see, there is no self argument in say() method, but there is an implicit this! What happens in add_this() decorator is that we modify function's __globals__ dictionary, adding this variable with value of self to the scope. Recall that __globals__ is

A reference to the dictionary that holds the function’s global variables — the global namespace of the module in which the function was defined [1].

Thus, modifying it, we also modify the global scope of the current module. This is certainly not the way to go, but is enough for simple demonstration.

If that is not crazy enough, let's write a metaclass which takes care of decorating methods automatically:

import types

class AddThisMeta(type):
    def __new__(cls, name, bases, classdict):
        new_classdict = {
            key: add_this(val) if isinstance(val, types.FunctionType) else val
            for key, val in classdict.items()
        }
        new_class = type.__new__(cls, name, bases, new_classdict)
        return new_class

class D(metaclass=AddThisMeta):
    name = 'Daniel'

    def say(phrase):
        print("{} says: {}".format(this.name, phrase))

    def run():
        print("{} runs away :)".format(this.name))

d = D()
d.say('And now, there is only AddThisMeta!')
d.run()

Output:

Daniel says: And now, there is only AddThisMeta!
Daniel runs away :)

Here, the metaclass does the same job we did above: it wraps all the methods which have a plain function type in class dictionary via add_this() decorator.

As you can see, it is not hard at all to introduce an implicit this in your code. But please, for all the good we have in Python, don't even think about doing it!.

References

[1]	https://docs.python.org/3/reference/datamodel.html

A micro Lisp interpreter experiment

2014-06-21T12:00:00+03:00

Let's define the language grammar and evaluation forms:

list := (item0, item1, ...)
item := list | atom
atom := stringliteral|numliteral

The evaluation rules are similar to any Lisp: the first atom is a function, the rest - are the arguments:

fn = list[0]
args = list[1:]

Notice that a list is written in a form of Python tuple. This is in a way a cheat which allows decreasing the interpreter's code dramatically (i.e. Python does the lexical and syntax analysis). Also note that the interpreter does not include any built-in operators, functions or special forms. They can all be created to extend the language separately.

Let's write some examples before jumping to the code of the interpreter and its extension functions:

(quote, 1, 2, 3) ; >>> (1, 2, 3)
(plus, 1, 2, 3)  ; >>> 6
(inc, 10)        ; >>> 11

Allright, enough sentimental talk, let's get to the interpreter!

The micro Lisp interpreter

def eval(list_or_atom):
    if isinstance(list_or_atom, tuple):
        fn, *fn_args = [eval(item) for item in list_or_atom]
        return fn(*fn_args)
    else:
        return list_or_atom

That is it! Here is how it works: First, we check whether the input is a list (Python tuple) or an atom. In case if it is an atom we return it immediately. Thus, eval(1) returns 1. If the argument is a tuple, we extract a function as its first element, and supply the rest elements as function arguments, recursively evaluating them in-place. Let's write various functions to extend this basic interpreter.

quote

Quoting is used to separate expressions from data in Lisp. For example, in Emacs-Lisp it is (quote 1 2 3), which is usually written via quotation prefix as '(1 2 3). Without quotation, Lisp will interprete this as a function call, where the first symbol (1) is a function name and 2 3 - are function arguments. Because we are limited by Python syntax, it is impossible to introduce Lisp-alike special quote forms, i.e. '(1 2 3). Thus, quote should be written as a function:

def quote(*args):
    """Returns a list without evaluating it."""
    return tuple(args)

eval((quote, 'x', 3, 0.7))
>>> ('x', 3, 0.7)

Please be aware, that this is still a rudimentary quotation, which does not work correctly for quoted sub-lists. For example, in any Lisp the following evaluation takes places:

'(1 2 '(3 4))
>>> (1 2 (quote 3 4))

However it evaluates to the following code in this mini-interpreter:

(1, 2, (3, 4))

plus

Let's write a mathematical function. Usually the + operator is used in various Lisp dialects for summing, but we are still limited with Python's syntax, which would not allow writing (+, 2, 3).

def plus(*args):
    """Sums up the input arguments."""
    return sum(args)

eval((plus, 3, 4, 5))
>>> 12

And here come two nice examples with recursion:

eval((plus, 3, 4, (plus, 5, 6)))
>>> 18

eval((plus, (plus, 3, 4), (plus, 5, 6)))
>>> 18

apply

What happens if you try to plus a list, like (plus, (quote, 1, 2, 3))? The interpreter will crash because it will end up calling Python's sum as sum([(1, 2, 3), ]). A typical Lisp dialect deals with this problem via apply function:

def apply(*args):
    """Applies a function to a list of arguments."""
    fn = args[0]
    fn_args = args[1]
    return fn(*fn_args)

eval((apply, plus, (quote, 1, 2, 3)))
>>> 6

map and inc

The map function takes another function and a list as input, applies the function to each element of this list and returns the results in a new list. For example: (map, inc, (quote, 1, 2, 3)) returns (2, 3, 4).

Here, inc - is a simple function which returns the value of it's argument + 1. For example, (inc, 10) returns 11.

def map(*args):
    """Apply the function to each element of the list and return
       the results in a new list."""
    fn = args[0]
    fn_args = args[1]
    return tuple(fn(item) for item in fn_args)

def inc(arg):
    """Increases the argument by 1."""
    return arg + 1

eval((map, inc, (quote, 1, 2, 3)))
>> (2, 3, 4)

lambdas

I was looking for a nice way of writing lambdas without modifying the base interpreter. Unfortunately using the Python lambdas directly would require putting an explicit eval() in the lambda body, e.g.:

eval((map, lambda x: (plus, x, 1), (quote, 1, 2, 3)))

Would not work because (plus, x, 1) is never evaluated. To make this work we have to write it as:

eval((map, lambda x: eval(plus, x, 1), (quote, 1, 2, 3)))

which destroys the consistency of the syntax. Ruby code blocks would be indeed helpful here :)

I will stop extending the interpreter now. As you see the interpreter is not complete and it is quite primitive. But the aim of this article is to show another way of demonstrating and teaching recursion in Python through the beauty of Lisp :) I hope you enjoyed it! Looking forward for your comments and solutions!

ECCO online

2014-02-01T12:00:00+02:00

Why?

My old boots' life was coming to an end. They are 5 years old and have served me very well. Considering the winter in Finland, investing into warm, water-proof and well-breathing winter boots with a tough iced surface grip is a must.

When?

Christmas sales? After New year discounts? Nonsense!

I believe that the best time to buy cloths is a time close to the end of a season. It is hard to call the end of January an "end of the season", nevertheless you may find that huge discounts on various seasonal cloths have already started. Certainly, there are even bigger discounts after the season ends - but then, the choice is extremely limited. If you get lucky - the discount can be dramatic. For example: we bought new roller skates for my wife, with a 75% discount in September. It was the last pair of the required size :)

Where?

I consider ordering shoes online a risky business. Time is money and it takes time for the shoes to arrive and if they don't fit - it takes time for the backward process. On the other hand, online stores have better price offers and discounts. The best ones include a "send it back" post tag into the original package which can be used to send the shoes back to the shop if they don't fit.

Who?

Extreme Nordic weather requires solid Nordic shoes to deal with it. Some of you have probably heard of Danish ECCO. I've been buying ECCO shoes, sandals, pumps, boots, trainers for years. Finally, I've got so assured with them, that I ordered new winter boots online. They came with a nice 40% discount, which decreased the price from 225 to 135 EUR. Now, 135 EUR may seem quite a price, but hey, if you expect at least 3-4 years of service, it drops to ~39 EUR/year. Isn't that a bargain?! Hell, my monthly Internet bill is about that amount of money. So, here it is ECCO Roxton:

The official ad says:

Warm up to Winter, in these sheepskin-lined 100% waterproof boots.

Lace up, high-cut boot style with metal D-rings and speed lacing system
Designed for premium comfort, warmth and insulation
Uppers of oiled nubuck, suede and textile - for durability and a comfortable fit
Warm lining of lambskin and wool provides superior, natural insulation
Gore-tex® lining provides a 100% waterproof guarantee, ensuring dry feet at all times
Direct-injected sole provides lightweight durability, flexibility and won't weigh you down

It took me a week to wear them in. The boots are a bit heavier than my old ones and have a thicker sole. After a week I can sincerely say, that these are the best winter boots I owned ever! Thank you, ECCO!

Kidomi - a JSON-based templating library

2014-01-05T12:00:00+02:00

Certainly, ClojureScript plays a major role in why the templating syntax seems so natural and pleasant, e.g.:

(node
[:span
  {:style
    {:color "#aaa"
     :text-decoration "line-through"}}
  "hello world!"])

But with JavaScript arrays and objects, there is a way to create something similar

kidomi(['span',
       {'style':
         {'color': '#aaa',
          'text-decoration': 'line-through'}},
       "hello world!"])

Which outputs a HTMLElement with the following nested structure:

<span style="color:#aaa; text-decoration:line-through">
  hello world!
</span>

Kidomi is written in CoffeeScript. It is covered by unit tests via QUnit and can be used by a Google Closure compiler in an ADVANCED_MODE compilation or separately (e.g. to produce a minified output).

Usage

The kidomi(data) function returns a HTMLNode constructed from a data, for example:

elem = kidomi(
     ['div#main.content',
         ['span', {'style': {'color': 'blue'}}, 'Select file'],
         ['form', {
             'name': 'inputName',
             'action': 'getform.php',
             'method': 'get'},
         'Username: ',
         ['input', {'type': 'text', 'name': 'user'}],
         ['input', {'type': 'submit', 'value': 'Submit'}]]])

The generated HTML element is:

<div id="main" class="content">
  <span style="color: blue;">Select file</span>
  <form name="inputName" action="getform.php" method="get">
    Username:
    <input type="text" name="user"></input>
    <input type="submit" value="Submit"></input>
  </form>
</div>

Syntax

The general syntax of kidomi is: node = kidomi(parsableObject). Here, node is a HTMLElement or in a more generic case a DOM node.

The parsableObject is:

A string. The returned object is a Text node.
A number. It is automatically converted to string and the returned object is a Text node.
A node. The returned object is the same node.
An array. This should be discussed a bit thoroughly:

The syntax of the parsableObject array is simple and very flexible. It consists of at least one item, which is:

['element#id.class1.class2.classN']

Here, id - is the id attribute of the node, class1.class2.classN - CSS classes of the node, i.e. class="class1 class2 classN".

For example:

['div']                 // <div></div>
['div#content']         // <div id="content"></div>
['span#user.username']  // <span id="user" class="username"></span>
['span.password']       // <span class="passwordd"></span>
['div.main.dialog']     // <div class="main dialog"></div>
// etc.

The second item is either an attributes object, or a sub-parsableObject. The attributes object has the following syntax:

{'class': ['class1', 'classN'],
 'style': {'prop1': 'val1', 'propN': 'valN'},
 'attribute1': 'value1',
 'attributeN': 'valueN'}

{'class': 'class1 classN',
 'style': 'prop1:val1; propN:valN;',
 'attribute1': 'value1',
 'attributeN': 'valueN'}

The class and style key-value pairs or strings are optional.

The class key-value pair is an array or a string with CSS classes' names applied to the node. It is appended to the classes found in the first item of the parsableObject array.
The style key-value pair is an object or a string of CSS style properties of the node.

The attributeX key-value pairs are the attributes of the node.

For example:

['a', {'class': ['biglink'],
       'style': {'color': 'red'},
       'href': 'http://github.com'}]

// <a href="http://github.com" class="biglink" style="color:red;"></a>

The rest of the array items are nested parsableObjects or in a special case - an array of arrays with parsableObjects. For example:

// Nested elements
['div', ['div', ['span.name', 'Name: '], ['span.lastname', 'Last name: ']]]
/*
   <div>
     <div>
       <span class="name">Name: </span>
       <span class="lastname">Last name: </span>
     </div>
   </div>
*/

// Expandable nested array
['tr', [['td', 'First'], ['td', 'Second'], ['td', 'Third']]]
/*
 <tr>
   <td>First</td>
   <td>Second</td>
   <td>Third</td>
 </tr>
*/

Building and testing

You will need the following tools to build and test kidomi:

GNU Make. This is used to run the Makefile script.
CoffeeScript compiler. This is enough to build the library.
Google Closure compiler. This is used to build the optimized version of the library. The CoffeeScript code is written with the Closure restrictions in mind.
PhantomJS is used to run the unit tests from a shell. You can as well run them in a normal browser.

Advanced usage

Referencing elements

One of the patterns where kidomi might be especially handy is when you have to create certain HTML elements before adding them in a DOM structure. For example:

button = kidomi(['button']);
button.onclick = function(){ alert('Hello world'); };

myDiv = kidomi(
            ['div',
              ['span', 'Click me:'],
                button]);

document.body.appendChild(myDiv);

List comprehensions in CoffeeScript

List (array) comprehensions are very handy to use as the expandable array elements, for example:

['tr', [['td', '1'], ['td', '2'], ['td', '3']]]

# can be written as:

['tr', (['td', "#{i}"] for i in [1..3])]

Is it a string?

2013-09-23T12:00:00+03:00

The most straightforward method of getting a variable type in javascript is using the typeof operator:

typeof 'abc'
>> "string"

In a simple case

s = 'abc'
typeof(s) == 'string'

But what do we know about javascript strings?

The String global object is a constructor for strings, or a sequence of characters. String literals take the forms [1]:

  'string text'
  "string text"

Or, using the ``String`` global object directly:

String(thing)
new String(thing)

Now that is where it becomes a bit tricky:

s = String('abc')
>> 'abc'
typeof s
>> 'string'
so = new String('abc')
>> { '0': 'a',
     '1': 'b',
     '2': 'c' }
typeof so
>> 'object'  // it is not a 'string' anymore!

And there is nothing wrong with the latter statement as because first of all so is Object created via the new operator. Fortunately the instanceof operator can handle this scenario:

so instanceof String
>> true

The code

isString = function(s) {
  return typeof(s) == 'string' || s instanceof(String);
}

isString('abc')
>> true
isString(new String('abc'))
>> true
isString(321)
>> false

References

[1]	Mozilla Developer Network: String

Kaylee v0.3

2013-06-20T12:00:00+03:00

What is new in Kaylee 0.3

Kaylee environment and project management scripts
Built-in debug server
Native Werkzeug framework support
Demo projects repository
Updated unit tests and massive code improvements

Kaylee environment and project management scripts

This is the sweetest feature of Kaylee. Since the first release I felt that although Kaylee may be an interesting project to an end user (meaning a programmer), it has a pretty big fence which one has to jump over to start using it. The tutorial was written in a plain and simple language, yet a user had to create a Makefile, learn about Kaylee's recursive-make-based build system and perform some other simple but inconvenient operations.

The best way out of this were management scripts inspired by the Django framework. Now, a user can simply run:

1	python kaylee-admin.py startenv myenv

to create a Kaylee development environment directory with a management script inside it. The management script helps creating, building and testing the projects, e.g.:

1	python klmanage.py startproject FastPI

- creates a fastpi directory with the project's server and client-side skeletons.

Built-in debug server

This feature is about testing and debugging your project without installing a 3d party web server. Simply execute

1	python klmanage.py run

after building the environment. The command starts a local web server, and launches the first available application defined in the environment's settings.py. The application is then by-default accessible via http://127.0.0.1:5000

Native Werkzeug framework support

Now you can easily integrate Kaylee into your Werkzeug -based web application via native API. This is the third front-end framework which is supported by Kaylee out-of-the box. The other two are Django and Flask.

Demo projects repository

At last, all Kaylee demo projects, including the tutorial project are gathered under a separate repository on github: https://github.com/BasicWolf/kaylee-demo-projects In addition to the projects, the repository includes a special build script which automatically creates a Kaylee demo environment, builds the projects and starts the built-in debug server.

Updated unit tests and massive code improvements

Finally, there are lots of code improvements covered with unit tests. All to make sure that Kaylee would be solid-stable the day it reaches the maturity version :)

Microsoft Wireless Mouse 5000: The Good, the Bad and the Ugly.

2013-05-28T12:00:00+03:00

Why MS Wireless 5000?

There is an odd thing people notice, looking and my working place: the mouse is on the left side. Usually the following question is: "Are you left-handed?" - "I'm not!". Long ago my father pointed out that a right-handed person can have a free hand for other tasks (like writing with a pen) if the mouse remains in the left hand. I am now more confident having the mouse on the left side, that on the right. But guess what is the trick in this situation? Most modern ergonomical mice are created for right-handed persons. For example, this nice Logitech M705:

One would find very hard times using it with the left hand. You are probably asking "why not use symmetrical mouse?". And that is exactly the kind of mouse I've been using since 2005. This is my old Defender (the blue one):

A funny story about it: Back in 2005 I went to a store looking for a USB mouse, as the PS/2 mouse was not working with the Linux Mandrake distro (early 2.6 kernel) on my dorm roommate's PC.

All these years the mouse was working perfectly. But then, I married and started sharing the computer with my wife. Guess how happy she was about the mouse on the left side :) Dragging the wired mouse here and there created a mess with all the adjacent cables. It was clear that a time to buy a wireless mouse had finally come.

Of course, my choice would have been dramatically influenced by the old mouse. I was looking for something:

wireless,
symmetrical,
big enough to fit my palm,
with side buttons
and laser sensor.

I have considered several models, which dropped out of the list one-by-one (for example the Logitech M525 was not big enough for my palm). In the end, the only mouse that seemed to fit all these requirements was:

The Good

My palm lies firmly on the mouse, the dimensions are 7x12 cm (2.8x4.6) inches. Two AA batteries add weight which gives the overall feeling of a solid device. The side panels are made from rubbery material. The BlueTrack laser is perfect, it literally works on any surface (I was able to use it on my kneecap :). Finally the wireless signal range is at least 3-4 meters (10-13 feet).

The Bad

I have been extensively using the side buttons on the Defender for convenient forward and back navigation in browsers. The side buttons on Defender are big and located directly under the thumb and the little finger. The side buttons on MS 5000 are small and very hard to click as they are located above of where thumb and little finger are placed:

I'd wish that was the only problem with the buttons, but there is something much worse: the middle button. The middle button has a "horizontal scrolling" feature, which can be used by pushing the button sideways. In practice, 80% of "middle click" ends up in horizontal scrolling, because the button is very stiff to click. I use middle click a lot, especially when browsing (e.g. for opening the links in background tabs). And it became a real headache to perform that operation with this mouse. Luckily...

The Ugly

Luckily, I am a Debian GNU/Linux user :) With Debian you can easily override the keymap of any X11 device. For example, it is possible to map the horizontal scroll buttons as a middle-click! To do a quick temporal change to the buttons map, run xinput. This is the output on my computer:

zaur@z:~% xinput
⎡ Virtual core pointer                            id=2        [master pointer  (3)]
⎜   ↳ Virtual core XTEST pointer                      id=4    [slave  pointer  (2)]
⎜   ↳ HID 044e:3012                                   id=10   [slave  pointer  (2)]
⎜   ↳ PS/2 Mouse                                      id=17   [slave  pointer  (2)]
⎜   ↳ AlpsPS/2 ALPS GlidePoint                        id=18   [slave  pointer  (2)]
⎜   ↳ Microsoft Microsoft® 2.4GHz Transceiver v7.0    id=12   [slave  pointer  (2)]
⎜   ↳ Microsoft Microsoft® 2.4GHz Transceiver v7.0    id=13   [slave  pointer  (2)]
⎣ Virtual core keyboard                       id=3    [master keyboard (2)]
    ↳ Virtual core XTEST keyboard                     id=5    [slave  keyboard (3)]
    ↳ Sony Vaio Keys                                  id=6    [slave  keyboard (3)]
    ↳ Power Button                                    id=7    [slave  keyboard (3)]
    ↳ UVC Camera (05ca:183b)                          id=8    [slave  keyboard (3)]
    ↳ HID 044e:3013                                   id=9    [slave  keyboard (3)]
    ↳ AT Translated Set 2 keyboard                    id=16   [slave  keyboard (3)]
    ↳ Microsoft Microsoft® 2.4GHz Transceiver v7.0    id=11   [slave  keyboard (3)]
    ↳ USB2.0 Camera                                   id=14   [slave  keyboard (3)]
    ↳ Topre Corporation Realforce 105U                id=15   [slave  keyboard (3)]

For some reason there are two Microsoft Transceivers devices. To find the required one, run xinput get-button-map device_id, where device_id is either 12 or 13, e.g.:

zaur@z:~% xinput get-button-map 13
1 2 3 4 5 6 7 8 9 10 11 12 13

In order to set the desired keymap of the device, run the xinput command as follows:

1	xinput set-button-map 12 1 2 3 4 5 2 2 8 9 10 11 12 13

Here, 2 is the code of the middle button which overrides the codes of "scroll left" and "scroll right" (6 and 7).

To make the change permanent, create a config file (e.g. 10-ms5000.conf) in /etc/X11/xorg.conf.d/ directory, with the following content:

Section "InputClass"
        Identifier      "MS 5000"
        MatchProduct    "Microsoft Microsoft® 2.4GHz Transceiver v7.0"
        Option          "ButtonMapping" "1 2 3 4 5 2 2 8 9 10 11 12 13"
EndSection

The conclusion

Microsoft Wireless Mouse 5000 is 75% worth its money (35EUR), especially for users who are looking for a big and symmetric mouse.

It has a minor (small side buttons) and a major (stiff middle click) glitches, which are not that bad to ignore this mouse.

I hope you enjoyed the review. Feel free to comment and share you mouse experience!

PyRegs: The Python Regex Debugger

2013-05-22T12:00:00+03:00

PyRegs v0.1

Features:

Written in Python 3
Tkinter widgets, which are available on most of the Python-supported platforms.
Python 3 regular expressions with Unicode support.
Easy matches and grouped values navigation.
Regex analysis in separate thread.
Quick regex reference.
Licensed under GPLv3.

Pyregs v0.1 is ready, you are encouraged to give it a try!

Installation

Run:

1	pip install pyregs

Or clone the repository from https://github.com/BasicWolf/pyregs

Screenshots

Match tab:

Grouping by name:

Thoughts about XML structure: elements vs. attributes

2013-05-18T12:00:00+03:00

XML defines no rules about using elements or attributes in paricular case. For example:

<user firstname="Alex" lastname="Black"/>

is as valid as

 <user>
   <firstname>Alex</firstname>
   <lastname>Black</lastname>
</user>

Are there any general guidelines on when attributes or elements should be used? Let's explore various cases to find out the best solution of the problem.

Consider the example above. Should one use the attributes or elements in this case? Unless you care about the size of the data, it doesn't matter at all! And that is true for this particular case without a context. We have no knowledge about how the <user> data is processed.

Processing matters

XML is meant to be both human and machine readable. Would you agree that "machine readable" matters more, because every day machines automatically process tons of XML data?

For example, the first steps of processing XHTML [1] may look like:

|----------|   |------------|   |-----------------|
| XML data |-->| XML parser |-->| XHTML processor |--> ...
|----------|   |------------|   |-----------------|

Note, that the XML parser knows nothing about the data except that it is XML. It doesn't care whether the data is described via the elements or attributes. In fact, the XML parser is able to parse any data, whether it is an XHTML document, RSS channell, OpenDocument container and so on. The XML parser transforms the input data into objects in memory which are passed to the XHTML processor which contains the logic of dealing with these objects.

XML data

Consider an ordered list which starts with a "100" bullet:

<ol start="100">
  <li>Coffee</li>
  <li>Milk</li>
</ol>

The XHTML processor, knows that the <ol> tag represents an ordered list, and the nested <li> tags are the list elements with the data. The processor also expects that the <ol> tag data may contain specific attribute which describe the list, e.g. "start" Note, that changing the value of the start attribute does not interfere with the data (the list items).

Now, let's create an unordered list:

<ul>
  <li>Tee</li>
  <li>Soda</li>
</ul>

The XHTML processor will also parse this data as a list, because it is aware of the <ul> tag and its contents. The HTML standard has the third <dt> [2] tag which describes a definision list.

Let's think of an interesting design task: how would you design an XML format which stores any kind of lists unknown to the processor module?

|----------|   |------------|   |----------------|
| XML data |-->| XML parser |-->| Our  processor |--> ...
|----------|   |------------|   |----------------|

And that is where attributes can be very useful:

<users type="list" item="user">
  <user>
    <name>Alex</name>
    <lastname>Black</lastname>
  </user>

  <user>
    <name>John</name>
    <name>Brown</name>
  </user>
</users>

The processor is not aware of the <users> tag. But it knows that if an element contains the type="list" attribute (and value), then the item attribute would describe the list items to look for. What if we wanted to make a list with multiple selected items? The attributes can handle this situation as well:

<users type="list" item="user">
  <user selected="true">
    <name>Alex</name>
    <lastname>Black</lastname>
  </user>

  <user>...</user>
  <user selected="true">...</user>
  <user>...</user>
</users>

Attributes as metadata

Metadata is a data which describes some other data. In all the examples above, the attributes contained the metadata information. Removing the type="list" and item="user" attributes from the <users> tag will affect the way how the data is processed, but the original data is intact.

In my opinion the attributes are perfect for metadata, and a person designing an XML document format should simply take the discussion of "Attributes vs. elements" to the "Data and metadata" level.

Eager to hear your comments!

Footnotes

[1]	HTML is not a subset of XML, while XHTML is.

[2]	http://www.w3schools.com/html/html_lists.asp

Kaylee

2012-08-28T12:00:00+03:00

Kaylee is

Kaylee is a Python and JavaScript framework for developing in-browser distributed and volunteer computing applications that allows programmers to concentrate on business logic by leaving all the computational nodes handling to Kaylee.

Why?

Do you have a wast amount of data to be analyzed? Are you deciphering a life-and-death matter e-mail? Do you have a smart algorithm to find the question with the known "42" answer? Do you have computing resources to do any of it? If the answer to the last question is "No! :(" then Kaylee is the right solution.

With Kaylee you are able to distribute the computation between millions of Internet users and they even don't have to know that! This is how it works:

A user opens a link / enters a web site via a modern browser.
Kaylee is started in background.
The application and the data are downloaded from the server.
The application solves the given tasks.
The results are sent back to the server.
Getting and solving the tasks continues as long as the user stays on the page or there are no more tasks left.

All this happens via JavaScript code executed in a separate so-called Web Worker thread. That's how Kaylee does not interfere with the browser's main JavaScript loop and the user interface is intact.

Get it!

The repository is located at http://github.com/basicwolf/kaylee

Release history

TODO: Kaylee v0.3 <kaylee_0_3>

I have graduated

2012-07-08T12:00:00+03:00

Ladies and Gentlemen!

I have graduated! Four years of master studies at the University of Eastern Finland and finally, it is done. I'd like to thank everyone who helped and supported me these years, my family, my friends, my colleagues at Blancco and first of all my supervisor Dr. Tomi Kinnunen.

The thesis is about Voice activity detection (VAD). You may be surprised to hear that you use it every day when speaking by mobile phone or Skype. It is a small module that detects whether the input signal from a microphone contains speech or not. Consider a typical phone conversation: you are speaking for half of the time and listen for another half. That's when a VAD decides that there is no need to transfer any data from your microphone to another end of the conversation. It saves your battery and network bandwidth a lot.

So, without any further hesitation, here is the Download link. I hope you will my thesis interesting and will enjoy reading it.

-- Zaur.

C's heritage: bitwise and equality operators

2011-12-29T12:00:00+02:00

I've stumbled in a Javascript situation where bitwise operators were used. Logically, a bitwise operator should have a higher precedence than an equality operator, e.g:

if x | 10 == y
    alert("We've got it!")

But it looks like that this code would work the other way, as in JavaScript bitwise operators have lower precedence than equality operators do (see Mozilla's JS reference). The code above would always return 0 for any valid numerical val, because the result of val & true is 0. So, the proper way would be to put parenthesis around bitwise expression:

if (x | 10) == y
   alert("We've got it!")

I dug up the history of the question and it seems like this behaviour comes from the age of K&R's C:

Early C had no separate operators for & and && or | and ||. (Got that?) Instead it used the notion (inherited from B and BCPL) of "truth-value context": where a Boolean value was expected, after "if" and "while" and so forth, the & and | operators were interpreted as && and || are now; in ordinary expressions, the bitwise interpretations were used. It worked out pretty well, but was hard to explain. (There was the notion of "top-level operators" in a truth-value context.)

The precedence of & and | were as they are now. ...

In retrospect it would have been better to go ahead and change the precedence of & to higher than ==, but it seemed safer just to split & and && without moving & past an existing operator. (After all, we had several hundred kilobytes of source code, and maybe 3 installations....)

—Dennis Ritchie [1].

In terms of logical statement in C:

if (x == 1 & y == 0) {
    /* ... */
}

Makes perfect sense. But it doesn't make any in terms of bitwise logic.

C++, Java, Objective-C, PHP, C# and finally Javascript have it the same way. Python, Ruby, Go have it the other way around.

Do you know any reasons (apart from the one that comes from C's heritage) which made programming languages' designers to follow C's precedence rules?

References

[1]	http://www.lysator.liu.se/c/dmr-on-or.html

Python: comparing to None

2011-08-01T12:00:00+03:00

Consider a class:

class Queue(object):
    def __init__(self):
        self._len = 0

    def __len__(self):
        return self._len

Next, the usage of the class is:

q = Queue()

# at this point, you want to check if q is None
if not q:
   doSomething()

The confusing thing is that doSomething() is actually called! And that is because len(q) == 0!

Instead, use the is None comparison:

if q is None:
    doSomething()

Inside Python: understanding os.listdir()

2011-05-08T12:00:00+03:00

To answer this question, one has to get inside Python's source. The stable Python 3.2 could be found here. The os.py module in source archive's Lib directory doesn't contain the listdir() function. Yet, the very place to look comes from:

from posix import *

Let's take a look on Modules/posixmodule.c:

Note

Pay attention to the comments!

static PyObject *
posix_listdir(PyObject *self, PyObject *args) /* line 2323 */
{
    /* POSIX-related code, supposed to start from line 2574 */
    /* ... */
    dirp = opendir(name); /* Opening directory for which os.listdir() was called */
    /* ... */

The opendir() function opens a directory stream corresponding to the directory name, and returns a pointer to the directory stream. The stream is positioned at the first entry in the directory.

—Linux opendir() man page

/* continuing posix_listdir() */
/* ... */
for (;;) {
    ep = readdir(dirp); /* A crucial readdir() call */
    /* ... */
    /* ... */

    if (ep->d_name[0] == '.' && /* skipping '.' and '..' */
        (NAMLEN(ep) == 1 ||
         (ep->d_name[1] == '.' && NAMLEN(ep) == 2)))
        continue;
    if (arg_is_unicode)
        v = PyUnicode_DecodeFSDefaultAndSize(ep->d_name, NAMLEN(ep));
    else
        v = PyBytes_FromStringAndSize(ep->d_name, NAMLEN(ep));
    if (v == NULL) {
        Py_CLEAR(d);
        break;
    }
    if (PyList_Append(d, v) != 0) { /* appending found path to the return list */
        Py_DECREF(v);
        Py_CLEAR(d);
        break;
    }
    /* ... */
}

The readdir() function returns a pointer to a dirent structure representing the next directory entry in the directory stream pointed to by dirp. It returns NULL on reaching the end of the directory stream or if an error occurred.

—Linux readdir() man page

In Linux, the dirent structure is defined as follows:

struct dirent {
   ino_t          d_ino;       /* inode number */
   off_t          d_off;       /* offset to the next dirent */
   unsigned short d_reclen;    /* length of this record */
   unsigned char  d_type;      /* type of file; */
   char           d_name[256]; /* filename */
};

As you can see, readdir() loops through a list of dirent structures, and there is no quarantie that the structures will be somehow sorted.

So, how one can act when a sorted os.listdir() behaviour is required? Pretty simple:

lst = sorted(os.listdir(path))

# sorted files only
files = sorted(f for f in os.listdir(path)
               if os.path.isfile(os.path.join(path, f)))

# sorted dirs only
dirs = sorted(d for d in os.listdir(path)
              if os.path.isdir(os.path.join(path, d)))

Another Python mystery revealed!

Opera extensions: YouTube suite

2010-12-31T12:00:00+02:00

So, the number of extensions is growing rapidly. But how can one benefit from an extension? Well, if you are a user-js geek, first, think of Opera extensions as of always up-to-date 3d party user-js scripts. The provided browser communication API for managing windows, tabs, toolbars, icons etc. makes a complete picture. Yet, a user-js is just a script that can be initially triggered at various page loading stages. Every extension, for starters, has it's own process constantly running in background... But why discuss technical issues? I'm sure a geek can find a way to create an extension in 30 minutes. (Hopefully all geeks have followed that link :)

I believe, there is no need to explain, what YouTube is. The widely-used video hosting service already got it's fair share of Opera extensions. Would you like to find out what features these wonderful extensions are capable of?

Downloading videos

Easy Youtube Downloader for Opera [⇩]

The extension provides an easy and convenient way to download YouTube videos in all available formats, including mp3 (through the videodroid.org service).

The extension creates a Download As: drop-down list with one or more direct links to video files. If you're not familiar with "p" notation, "p" stands for horizontal scan lines of display resolution (also known as number of vertical resolution pixels) [1]. The very basic idea here is: the bigger - the better.

In case you don't know what should be chosen from 360p options [2] [3], a simple answer would be: "360p FLV has better resolution". Consequently, the file size will differ a bit too. For example, the Opera Unite intro video's size is 4.4 MB (MP4) versus 5.2 MB (FLV).

Bored of watching low-res movies? Hit highest available HD option (720p or 1080p) and download the clip in High Definition.

Easy Youtube Downloader is not the one of it's kind, yet is the best available. Even with minor advantages (an icon on download button and "MP3" download option) it beats the Fastest Youtube Video Downloader and Youtube Downloader 2 extensions.

Watching videos

Focus on Videos [⇩]

Focus on Videos is a nice extension that deems everything but the video you're watching. As a video starts playing, the extension turns this

Into this:

You can also toggle that by clicking a control button with a green lamp icon on it, or hitting Escape button (yeah, the one that's in the right top corner of the keyboard :).

Note

This focus effect is especially perfect if browser is running in Full screen mode.

YouTube AdsFree [⇩]

An extension that makes user's dream come true. As a user, I'm very embarrassed with these kind of ads:

Fortunately, YouTube AdsFree extension solves the problem once and for all. Eventually Google will have to invent another way of showing YouTube ads to Opera users.

Note

You have to re-open the page with a video to make extension work. (In this case, Ctrl+W; Ctrl+Z will save you a couple of mouse clicks!)

LoopTube [⇩]

A minor, yet often required feature that a standard player lacks: video playback loop. LoopTube is a small extension that makes our live easier by providing that feature. You can manage it's behaviour via additional control button:

Youtube Autoplay Killer [⇩] Another heavily demanded feature. Not only the extension disables auto-play, it also holds the download process (very useful, e.g. if you just want to read the comments, yet very annoying, if you have a slow connection and want to cache the video before watching it).

A small drawback: the extension doesn't work with HTML5 videos (which are discussed below), but that's version 1.0 we're talking about!

Searching for videos

YouTube WebM Enablr [⇩] Do you know that the <video> tag was proposed by Opera back in 2007? That was almost 4 years ago! Nowadays we're slowly moving from Adobe Flash-embedded videos to HTML5 video system.

WebM is a multimedia container format designed to provide a royalty-free, high-quality open video compression format for use with HTML5 video [4].

In my opinion, the key words here are: royalty-free, high-quality, open. There were loud debates (I wonder if they are finished) about the format [5]. Still, what concerns Opera users is that WebM is fully supported by Opera, and if you have a trouble running Adobe Flash plug-in, then WebM Enablr is just what the doctor ordered.

The steps are simple: 1. Install the extension. 2. Search for videos. 3. Watch video without Flash plugin involved.

Note

WebM videos only! There are plenty of YouTube clips that haven't been encoded in WebM yet. And don't rely on playlist-based search results. Those still might require Adobe Flash plugin.

So many of them...

Of course, this is not the complete list of YouTube-specific extensions. Hopefully this small suite will suit your basic needs. Don't hesitate a moment to try them all !

References

[1]	Wikipedia: 720p

[2]	Wikipedia: Flash Video

[3]	Wikipedia: MPEG-4

[4]	Wikipedia: WebM

[5]	Wikipedia: Default HTML5 video format debate

Assembler for beginners

2010-08-08T12:00:00+03:00

As a part of my work in Blancco, I encounter a very low-level code. By low-level I mean Assembler (AT&T syntax) for x86 processors family. A bit shame for computer scientist: I have NO experience in assembly. Thus, I spent a lot of time looking for good tutorials. Sounds interesting?

Let me introduce you to Programming from the Ground Up Book by Jonathan Bartlett. What is so special about it?

This is an introductory book to programming and computer science using assembly language. It assumes the reader has never programmed before, and introduces the concepts of variables, functions, and flow control.

I've spent less than half an hour of reading it and already know much much more about processor's registers, interrupts, assembly language basics. The code examples are clear and explained very well. The language of the book is easy to understand and very interesting to read. I'd like to thank the author for this achievement. Great work!

Linux, it's time to wake up!

2010-05-22T12:00:00+03:00

What is RTC and ACPI?

A Real Time Clock (RTC) alarm is a feature that can be used to allow a computer to 'wake up' after shut down to execute tasks every day or on a certain day. It can sometimes be found in the 'Power Management' section of a motherboard's BIOS setup [1].

In computing, the Advanced Configuration and Power Interface (ACPI) specification provides an open standard for unified operating system-centric device configuration and power management. ACPI, first released in December 1996, defines platform-independent interfaces for hardware discovery, configuration, power management and monitoring [2].

ACPI allows control of power management from within the operating system. The previous industry standard for power management, Advanced Power Management (APM), is controlled at the BIOS level. APM is activated when the system becomes idle: the longer the system idles, the less power it consumes (e.g. screen saver vs. sleep vs. suspend). In APM, the operating system has no knowledge of when the system will change power states [3].

ACPI can typically be configured from within the operating system. This is unlike APM where configuration often involves rebooting and entering the BIOS configuration screens to set parameters.

The ACPI specification defines seven states (so-called global states) for an ACPI-compliant computer-system [4]. Some of them are:

G0 (S0) Working.
G1 Sleeping (subdivides into the four states S1 through S4).
G1/S3 Commonly referred to as Standby, Sleep, or Suspend to RAM. RAM remains powered
G1/S4 Hibernation or Suspend to Disk. All content of main memory is saved to non-volatile memory such as a hard drive, and is powered down.
G2 (S5) Soft Off. G2, S5, and Soft Off are synonyms. G2 is almost the same as G3 Mechanical Off, but some components remain powered so the computer can "wake" from input from the keyboard, clock, modem, LAN, or USB device.

If implemented, the Real Time Clock alarm must generate a hardware wake event when in the sleeping state. The RTC can be programmed to generate an alarm. An enabled RTC alarm can be used to generate a wake event when the system is in a sleeping state.

Setting up the wakeup alarm

Open a terminal and switch to root su root.
Check if rtc is available (kernel 2.6.22 and higher): ls /sys/class/rtc/rtc0/
Look for the wakealarm file.
Initialize the alarm via echo 0 > /sys/class/rtc/rtc0/wakealarm.
Check if it has been initialzed: cat /proc/driver/rtc. At this point, the output should be similar to:

rtc_time      : 09:30:05
rtc_date      : 2010-05-22
alrm_time     : 09:34:43
alrm_date     : ****-**-22
alarm_IRQ     : no
alrm_pending  : no
24hr          : yes
periodic_IRQ  : no
update_IRQ    : no
HPET_emulated : yes
DST_enable    : no
periodic_freq : 1024
batt_status   : okay

The rtc alarm is now ready to be set up. Wakealarm accepts the number of seconds since Jan 1, 1970 (this is known as "unix time", "POSIX time" or "epoch time").

You must make sure that your BIOS clock is set to UTC time - not localtime - otherwise it will wakeup at the wrong time. However, it is still possible if the BIOS clock is set to localtime (likely if you also run windows); see the section below for how to set the alarm correctly when the BIOS clock is in localtime. If you want to change the wakealarm time, you will need to write the new wakealarm time to the BIOS.

A simple way to set the alarm to current time + 1 minute is: echo "+60" > /sys/class/rtc/rtc0/wakealarm.
Check if the alarm has been set up: cat /proc/driver/rtc

...
alarm_IRQ      : yes
...

Now, suspend or shut down your computer. And wait until it wakes up in less than a minute :)

You can find more information about the topic at MythTV.org.

Be careful with the experiments!

References

[1]	Wikipedia.org: Real-time alarm

[2]	Wikipedia.org: ACPI

[3]	Wikipedia.org: ACPI power states

[4]	About ACPI by Emma Jane Hogbin

Why Opera?

2010-04-28T12:00:00+03:00

A real hacker?

Opera is my major browser since version 5.02. I discovered it in "Хакер" (russian word for "Hacker") monthly magazine. It was described as an innovative browser with a "jet engine" inside. With the "A real hacker should use Opera!" motto I launched the program... And for ten years Opera never stopped surprising me.

You can find lots of "Opera vs. Firefox" or "Opera vs. Chrome" or "Opera vs. Something-something-something". Most of those articles will lead you to a "speed test" results. Do you really care, if your browser runs a benchmark in 10.1 seconds, with top competitor running it in 9.812 s.? I don't. What I really care of is how the browser is easy-to-use, is compatible with standards and well.. how good it serves my needs.

Features?

Even if you're an experienced user, you may find lots of features you've never heard about. For example, do you know that Opera has a "Show" mode? No a common feature though.

What about the common features?

Trash bin for closed tabs. If you close a tab accidentally, Ctrl + z restores it immediately.
Navigation via mouse or keyboard is very fast in Opera. There are mouse gestures, back-and-forward clicks (press left button while holding right button to navigate back), Wand password manager (just press Ctrl + Enter or the Wand button on toolbar to login automatically), keyboard navigation (e.g. Shift + arrow keys), middle-click panning.. have I forgotten something?
Zooming is another great usability feature. A user can combine it with the Fit to width feature - that allows zooming into page and being able to navigate without using horizontal scroll bar at all!
Search engine customization allows one to create a "search engine" from any text box on any website. For example, I often use Google translate's Finnish-to-English page. With custom "search engine" in Opera , all I have to do is to type "gfi [word or phrase]" in address bar and press "Enter" - Opera will take me to a page with translated text.
Opera Unite is a technology that turns a browser into a web server. One can share files, music, pictures, serve static html pages, create chatrooms.. you name it!

This is a very short incomplete list and there are lots and lots of other great features. But I'm sure you get the idea :)

Portability?

You see, with Opera you can get it's all wonderful features in one place, without messing with add-ons, extensions, plug-ins etc. I get very annoyed when I have to use a computer at the university with a "blank" Firefox on it. "Blank" means - no particular extensions that could ease user's work. I can have a perfectly customized Firefox on my own computer.. but as soon as I reach a "blank" installation or even another persons customized environment - I fail.

Sorry for mentioning Firefox, but I wanted to explain, why "add-ons" approach might not work for a person who seeks a complete solution. I respect Mozilla foundation and understand why the "add-ons" way was chosen. Moreover, this way has proved it's consistency and our days Firefox has a much bigger market share than Opera.

But let's continue with Opera. Opera is truly cross-platform software. Opera remains the same on Linux, Windows, MacOS, *BSD. A user gets the same features on all platforms!. Does your mobile phone supports Java? There is Opera Mini to serve you. Here comes another feature that was introduced in Opera 9.5 (back in 2007): synchronization. You add a bookmark in your mobile phone and it appears in your desktop browser. You type an address in Opera's address bar at work, and then get a clue from the browser's "typed addresses history" at home. Will you agree that all this makes Opera extremely portable?

Give it a try!

I hope that I convinced you to give Opera another try :) Enjoy!

Inside Python subprocess communication

2010-04-24T12:00:00+03:00

Before we begin

I love Python and try using the latest versions when possible. But this article is about the subprocess module in Python 2.6. Is there a good reason for that? For starters, you can find this comment in Python 3.1's subprocess module:

# XXX Rewrite these to use non-blocking I/O on the
# file objects; they are no longer using C stdio!

Then, there is a pending Asynchronous I/O For subprocess. Popen PEP-3145. Last, but not least, a

temporary moratorium (suspension) of all changes to the Python language syntax, semantics, and built-ins for a period of at least two years from the release of Python 3.1

was proposed and accepted in PEP 3003. Yet,

..As the standard library is not directly tied to the language definition it is not covered by this moratorium.

A simple experiment

import subprocess

proc = subprocess.Popen(['ls', '-l'],
                        cwd='/',
                        stdout=subprocess.PIPE)
out, err = proc.communicate()
print(out)

This one is really easy: the ls command with the -l switch is executed, the root is set as a current working directory and a pipe is created to get the data written by ls to stdout.

Why use communicate? Why not write the data directly to Popen.stdin and read from Popen.stdout?

The official documentation says:

Use communicate() rather than Popen.stdin.write, Popen.stdout.read or Popen.stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process.

Let's try communicate() with a long-term shell process:

import subprocess

proc = subprocess.Popen('bash',
                        cwd='/',
                        stdin=subprocess.PIPE,
                        stdout=subprocess.PIPE,
                        stderr=subprocess.PIPE)

out, err = proc.communicate('ls')
print(out)

out, err = proc.communicate('ls -l')

With successful first communicate() call, you'll receive ValueError: I/O operation on closed file trying communicate the second time.

Would you like to know, why an error is raised? It's time to dive deeper into the code.

Inside Popen.communicate()

You can find the original and complete code of subprocess module in e.g. /usr/lib/python2.6/subprocess.py.

Note the comments!

def communicate(self, input=None):
   """Interact with process: Send data to stdin.  Read data from
   stdout and stderr, until end-of-file is reached.  Wait for
   process to terminate.  The optional input argument should be a
   string to be sent to the child process, or None, if no data
   should be sent to the child.

   communicate() returns a tuple (stdout, stderr)."""

   # Optimization: If we are only using one pipe, or no pipe at
   # all, using select() or threads is unnecessary.
   if [self.stdin, self.stdout, self.stderr].count(None) >= 2:
      stdout = None
      stderr = None
      if self.stdin:
         if input:
            self.stdin.write(input)
         self.stdin.close()
      elif self.stdout:
         # This happens in the experiment we ran above
         stdout = self.stdout.read()

         # Note, stdout is closed!
         self.stdout.close()
      elif self.stderr:
         stderr = self.stderr.read()
         self.stderr.close()

      # Waiting until process terminates!
      self.wait()

      return (stdout, stderr)

   # The most interesting case, two or more pipes opened
   # Remember, self is an instance of subprocess.Popen class
   return self._communicate(input)

Python was made to be cross-platform. On the other hand Python is generally used on POSIX-compatible operating systems. Let's skip the if mswindows: part and get to the POSIX methods block:

def _communicate(self, input):
    # there is a historical reason calling this variables "sets"
    # see select.select() (below)
    read_set = []
    write_set = []

    # returned variables
    stdout = None
    stderr = None

    # the tricky part starts right here
    if self.stdin:
        # Flush stdio buffer.  This might block (!), if the user has
        # been writing to .stdin in an uncontrolled fashion.
        self.stdin.flush()

        # Data to be sent to the process through the pipe
        if input:
            write_set.append(self.stdin)
        else:
            self.stdin.close()

    if self.stdout:
        read_set.append(self.stdout)
        stdout = []
    if self.stderr:
        read_set.append(self.stderr)
        stderr = []

    input_offset = 0

    # ...

This part was not hard at all. It'll be a little bit harder in the next block: please read the select.select()``[1]_ documentation if you're not familiar with the ``select() system call.

#...
# while read_set contains self.stdout
# or write_set contains self.stdin
while read_set or write_set:
    try:
       # man select
       # .. select() allows a program to monitor multiple file descriptors,
       # waiting until one or more of the file descriptors become "ready"
       # for some class of I/O operation (e.g., input possible).
       # A file descriptor is  considered ready if it is possible to perform
       # the corresponding I/O operation (e.g., read) without blocking
       rlist, wlist, xlist = select.select(read_set, write_set, [])
    except select.error, e:

       # EINTR means "This call did not succeed because it was interrupted.
       # However, if you try again, it will probably work."
       # In other words, EINTR is not a fatal error, it just means
       # you should retry whatever you were attempting.
       if e.args[0] == errno.EINTR:
           continue
       raise

    if self.stdin in wlist:
       # When select has indicated that the file is writable,
       # we can write up to PIPE_BUF bytes without risk
       # blocking.  POSIX defines PIPE_BUF >= 512
       chunk = input[input_offset : input_offset + 512]
       bytes_written = os.write(self.stdin.fileno(), chunk)
       input_offset += bytes_written
       if input_offset >= len(input):
          # stdin is closed! It's not possible communicate(input) any more.
          self.stdin.close()
          write_set.remove(self.stdin) # write_set is empty now

    if self.stdout in rlist:
        data = os.read(self.stdout.fileno(), 1024)
        if data == "":
            self.stdout.close()
            read_set.remove(self.stdout)
        stdout.append(data)

    if self.stderr in rlist:
        data = os.read(self.stderr.fileno(), 1024)
        if data == "":
           self.stderr.close()
           read_set.remove(self.stderr)
        stderr.append(data)

    #...

Note, that os.write() and os.read() functions are being used. This functions are intended for low-level I/O. If the end of the file referred to by file descriptor (e.g. self.stdout.fileno()) has been reached, an empty string is returned (the if data == "": conditions).

#...
# The while read_set or write_set: loop ends here
# All data exchanged.  Translate lists into strings.
if stdout is not None:
    stdout = ''.join(stdout) # (stdout) is a list
if stderr is not None:
    stderr = ''.join(stderr)

# Translate newlines, if requested.  We cannot let the file
# object do the translation: It is based on stdio, which is
# impossible to combine with select (unless forcing no
# buffering).
if self.universal_newlines and hasattr(file, 'newlines'):
    if stdout:
       stdout = self._translate_newlines(stdout)
    if stderr:
       stderr = self._translate_newlines(stderr)

self.wait() # wait until process terminates
return (stdout, stderr)

If Python was built with the --with-universal-newlines option in configure (the default), the file.newlines read-only attribute exists, and for files opened in universal newline read mode it keeps track of the types of newlines encountered while reading the file. The _translate_newlines() method just replaces the Windows-style (\r\n) and Mac-style (\r) newlines with \n.

The last pieces of the puzzle:

def wait(self):
    """Wait for child process to terminate.  Returns returncode
    attribute."""
    if self.returncode is None:
        # Try calling a function os.waitpid(self.pid, 0)
        # Ignore Interrupted System Call (errno.EINTR) errors
        pid, sts = _eintr_retry_call(os.waitpid, self.pid, 0)
        self._handle_exitstatus(sts)
    return self.returncode

References

[1]	http://docs.python.org/library/select.html#select.select

A very long nightmare

2010-04-01T12:00:00+03:00

Few tools I can use right now:

Ubuntu 8.10 Live CD (without Ext4 suppor!)
A usb flash drive
A two month-old backup (agh!)

I'm writing this post in Ubuntu loaded from LiveCD. Right now, I'm downloading Ubuntu 10.04 and RIPLinux distribution. I hope everything will be fine :)

Update@3.00 AM

Some files are corrupted, but I was able to backup projects and master thesis materials. Now, backing up thunderbird directory.

Update@6.55 AM

There are more than 2 millions bad sectors on the drive. I'm lucky to have two external USB SATA drives. I hope that the only thing I've missed this night - were sweet dreams :)

Update@8.39 AM

Ubuntu 10.04 Beta is installed and updated. Too tired to write how BAD I felt about it. Looks like I have to move on. Long ago, when I lived and studied in Saint Petersburg, me and my roommate Vova had frequent conversations about Linux world. Ubuntu was new, still, as Vova said, "There is always will be Debian". I'm downloading Debian Testing right now. Thank you for everything you've taught me, Ubuntu!

Update@21.00

What the HELL! I never thought that Debian will cause so many problems. First, there is no iwlagn4945 firmware in first installation CD. And I don't have an Ethernet cable to connect and download everything required for normal network support. Agh. I'm getting back to Ubuntu :)

Python classes: Dynamic properties

2010-03-10T12:00:00+02:00

I love Python 3

Even Python 2.x is still used everywhere, it is time to move to Python 3. I made the code fully Python2.6 - compatible, but if you'd like to have a nice output of print(..) function please use the

from __future__ import print_function

statement.

A simple class

This is a simple class with one property:

class Man(object):
    def __init__(self, name):
        self._name = name

    @property
    def name(self):
        return self._name

    @name.setter
    def name(self, value):
        self._name = value

    m = Man('Alex Black')
    print(m.name)

    m.name = 'Kyra Brown'
    print(m.name)

The output is:

Alex Black
Kyra Brown

We keep the local values in attributes that start with the underscore, e.g. self._name. Usually this is a way I separate "private" and "public" members of a class/object [1] (of course this "private" members remain "public").

This code does the same as the code above:

class Man(object):
    def __init__(self, name):
        self._name = name

    def _name_get(self):
        return getattr(self, '_name')

    def _name_set(self, value):
        setattr(self, '_name', value)

    name = property(fget = _name_get, fset = _name_set)

There are two core changes:

The _name attribute of an object is read and set by the getattr(..) and setattr(..) functions.
The @property decorator is replaced by the built-in property(...) function [2] (which is actually "behind" that decorator).

Dynamic properties class

Finally, let's write a class called "Properties" that will allow adding dynamic properties. The properties will require local ("private") fields. We can use the same scheme as above, e.g. for property name, the private member of a class is _name:

class Properties(object):
    def add_property(self, name, value):
        # create local fget and fset functions
        fget = lambda self: self._get_property(name)
        fset = lambda self, value: self._set_property(name, value)

        # add property to self
        setattr(self.__class__, name, property(fget, fset))
        # add corresponding local variable
        setattr(self, '_' + name, value)


    def _set_property(self, name, value):
        setattr(self, '_' + name, value)

    def _get_property(self, name):
        return getattr(self, '_' + name)

The trick in add_property(..) is that we create two lambda objects (those could also be anonymous functions) which use the self._get_property and self._set_property methods with particular value of name argument.

Let's play with this class:

po = Properties()
po.add_property('user', 'noname')
po.add_property('speed', 50)

print(po.user, po.speed)

po.speed = 100
po.name = 'Alex Black'

print(po.user, po.speed)
print(po._user, po._speed)

The output is:

noname 50
Alex Black 100
Alex Black 100

A practical usage

After all, what is a practical usage of dynamic properties? I'm sure you may have thought of that if you're reading this post now :) Here is a small example of a class which is able to "lock" the properties' setters:

class PropertiesWithLock(object):
    def __init__(self, lock = False):
        self.lock = lock

    def add_property(self, name, value):
        fget = lambda self: self._get_property(name)
        fset = lambda self, value: self._set_property(name, value)

        setattr(self.__class__, name, property(fget, fset))
        setattr(self, '_' + name, value)


    def _set_property(self, name, value):
        if not self.lock:
            setattr(self, '_' + name, value)
        else:
             print('Cannot change "{0}": properties are locked'
                    .format(name))

    def _get_property(self, name):
        return getattr(self, '_' + name)

And the usage:

po = PropertiesWithLock()
po.add_property('user', 'noname')
po.add_property('speed', 50)

print(po.user, po.speed)
>>> noname 50

po.user = 'a user'
po.speed = 200
print(po.speed)
>>> a user 200

po.lock = True
po.user = 'a user'
>>> Cannot change "user": properties are locked

References

[1]	Reserved classes of identifiers

[2]	property([fget[, fset[, fdel[, doc]]]])

Back to life

2009-08-19T12:00:00+03:00

Lots of things happened. I have recently finished my IT Project (as a part of my studies).

VADpy is a universal, modular, easy-to-run and easy-to-extend Voice Activity Detection algorithms testing framework that puts different speech/sound databases, VADs, error metrics and performance reports in one place. VADpy saves developers time by providing convinient MATLAB interface for MATLAB-based VADs. A powerful modular architecture lets developers integrate any kind of file databases and VADs into the framework. The Python programming language provides convinient environment to develop input, error metric and output modules.

—VADpy adtext.

I learned lots of new things about Python while working on this project. I hope to share this knowledge in my blog.

Also, I got rid out of static html, now the blog runs on Django. The "engine" is still in development (comments with captcha are on their way) yet, the blog is fully-functional and you're welcome to read it.

Beginner's guide to creating a GNOME 2 applet with Python (Part II)

2009-08-19T12:00:00+03:00

Part II: GNOME applets and PyGTK

Before we begin...

If you have read part I of the tutorial, and have a test applet running in both "panel" and "debug" mode, you would have no problems running all the code described below. All terms described in part I, keep their meanings here so, if you have not read the article, you can do it now.

1. Launching an applet

Generally, a GNOME applet could be treated as an application "attached" to the panel through Bonobo component model system [2]. We can launch the sample applet in a standard GNOME window with the -d key [1]. Yet, for example creating and attaching a menu to an applet is not the same task of creating a menu for a windowed application. Or, an applet will always be vertically-oriented" in debug-mode. So, this time the applet will be added to a panel during all test-runs.

Note

If the sample applet doesn't appear on a panel, it means, that something went wrong (a possible bug in a script). Try launching the applet in debug mode to see the output and catch possible bugs.

2. Popup Context Menu

Every applet has a basic context popup-menu with three (at least in GNOME v2.26) items:

According to the GNOME documentation library [3], the only way to extend this menu is to call the following function:

void panel_applet_setup_menu (PanelApplet *applet,
                              const gchar *xml,
                              const BonoboUIVerb *verb_list,
                              gpointer user_data);

The corresponding method in Python is Applet object's setup_menu(xml, verbs, user_data) method.

The required xml string is in a very easy-to-understand XML format:

<popup name="button3">
<menuitem name="ItemPreferences"
          verb="Preferences"
          label="_Preferences"
          pixtype="stock"
          pixname="gtk-preferences"/>
<separator/>
<submenu name="Submenu" _label="Su_bmenu">
<menuitem name="ItemAbout"
          verb="About"
          label="_About"
          pixtype="stock"
          pixname="gtk-about"/>
</submenu>
</popup>

The example above creates a menu of two items, a separator between them and a single sub-item:

As you can see, the Preferences and About icons are set from the current environment's theme. So, pixtype="stock" and pixname="gtk-about" attributes mean that an icon from GTK stock collection named "gtk-about" [4] will appear in a menu item. The name and label attributes define menuitem object's name and label properties. The underscore (_) character defines a shortcut to a menu.

The verb attribute is crucial for each menuitem node. The name of each verb links a menuitem and a callback function.

verbs = [('About', show_about), ('Preferences', show_preferences)]

To link the menus and the verbs list to an applet, the setup_menu() method is called:1

applet.setup_menu(xml, verbs, None)

Where None is a "user data" (could be any python data object or None).

Finally, all snippets described above, plus callback functions will look as follows:

def create_menu(applet):
   xml="""<popup name="button3">
          <menuitem name="ItemPreferences"
              verb="Preferences"
              label="_Preferences"
              pixtype="stock"
              pixname="gtk-preferences"/>
          <separator/>
          <submenu name="Submenu" _label="Su_bmenu">
          <menuitem name="ItemAbout"
              verb="About"
              label="_About"
              pixtype="stock"
              pixname="gtk-about"/>
          </submenu>
          </popup>"""

   verbs = [('About', show_about), ('Preferences', show_preferences)]
   applet.setup_menu(xml, verbs, None)


def show_about(*arguments):
   print(arguments)


def show_preferences(*arguments):
   print(arguments)


def applet_factory(gnome_applet, iid):
   create_menu(gnome_applet)
   ...
   ...

Try launching the applet in debug mode, and printing out the *arguments list in show_about() and show_preferences() functions.

The result should be similar to:

(<bonobo.ui.Component object at 0x963f5cc (BonoboUIComponent at 0x969dac0)>, 'About'

Also try changing the None argument in setup_menu() method to e.g. 'Hello!' string and then printing out the *arguments list one more time. This time, the result should look like:

(<bonobo.ui.Component object at 0x963f5cc (BonoboUIComponent at 0x969dac0)>, 'About', 'Hello!')

Now, you can change the callbacks' definitions

def show_about(*arguments):

def show_about(obj, label, *data):

which slightly clarifies the code.

3.1 Applet's orientation

An applet could be set up both on a horizontal or vertical panel. This should be taken into account before creating the visual elements of an applet. Also an applet's orientation could be changed by user (dragging an applet from a horizontal panel to vertical panel), in this case a callback for the change-orient signal should be created [5].

def change_orientation(applet, orient, user_data):
    pass

def applet_factory(applet, iid):
    applet.connect('change-orient', change_orientation)
    ...
    ...

The code below demonstrates how change-orient signal could be used:

def applet_factory(applet, iid):
    # initialize an orientation-dependent label
    orientation = applet.get_orient()
    if (orientation == gnomeapplet.ORIENT_UP or
        orientation == gnomeapplet.ORIENT_DOWN):
        label = gtk.Label("Vertical")
    else:
        label = gtk.Label("Horizontal")

    # label is the user data
    applet.connect('change-orient', change_orientation, label)
    ...
    ...


def change_orientation(applet, orient, user_data):
    orientation = applet.get_orient()
    label = user_data

    if (orientation == gnomeapplet.ORIENT_UP or
        orientation == gnomeapplet.ORIENT_DOWN):
        label.set_label("Vertical")
    else:
        label.set_label("Horizontal")

The applet was added to a vertical panel:

Then, dragged to a auto-hidden horizontal one:

3.2 Applet's background

The GNOME panel has three background options:

And an applet can detect these options' changes via the change-background signal [6].

But in practical situations, an applet doesn't respond to the panel's background changes. What really matters is an applet's background transparency. This is where the power of GNU helps: the sources of GNOME trash applet contain a "hack" line of code that makes an applet transparent. In Python it is:

def applet_factory(applet, iid):
    applet.set_background_widget(applet) # /* enable transparency hack */
    ...
    ...

Now, the applet should look like this:

That's it! This is the end of the second part of the tutorial. There is not much left to say about the applet techniques. But there is much to learn about PyGTK (creating dialogs, different widgets, signals, etc.). A great tutorial is available at the official site http://www.pygtk.org.

References

[1]	Beginner's guide to creating a GNOME applet with Python (Part I)

[2]	Bonobo component model

[3]	The PanelApplet object

[4]	GTK stock items

[5]	www.pygtk.org - GNOME applet with python

[6]	The "change-background" signal

Big and Little endianness quest

2009-08-01T12:00:00+03:00

I was working with G.729 Annex B 's source code <http://www.itu.int/rec/T-REC-G.729/en> to extract the codec's Voice Activity Detector (VAD).

Voice activity detection is a technique used in speech processing wherein the presence or absence of human speech is detected in regions of audio (which may also contain music, noise, or other sound).

—Wikipedia [1]

The result of my work was a tool that produced 0 and 1 binary output for a speech and non-speech 10ms-long sections of a sound file with sampling frequency of 8000 Hz and 16 bits PCM data.

The Aurora speech recognition experimental framework [2] data was used to test the program. That was the point where the problems began.

The VAD was not working. Almost every part of every file was marked as "speech". But that was not true! And the G729 Annex B VAD could not work so awful. Where was the problem?

The original G.729 reads the input file as follows:

while( fread(new_speech, sizeof(Word16), L_FRAME, f_speech) == L_FRAME)

Where new_speech is a L_FRAME-long array of Word16 (short) elements. Then the new_speech data is pre-processed, passed through and compressed via the codec. I removed the compression part and the code looked like:

Pre_Process(new_speech, L_FRAME);
Vad = Coder_ld8a(prm, frame);

Probably the error was in the Coder_ld8a() function. But how could it be? I just removed the compression stuff, and added

return 1 for speech sections
return 0 for non-speech sections

I could have spent a lot of time hunting the "ghost" bug, if my supervisor Rahim Saedi didn't ask, whether the files were read correctly. OMG! A possible bug error in fread() function?

I checked the values of the new_speech array for a random file from the Aurora database and found something weird: the values sequence was -1 -1 0 0 256 -1 0 0 0 256 ... Something clicked in my mind.

In computing, endianness is the byte (and sometimes bit) ordering used to represent some kind of data. Most modern computer processors agree on bit ordering "inside" individual bytes (this was not always the case). This means that any single-byte value will be read the same on almost any computer one may send it to.

Integers are usually stored as sequences of bytes, so that the encoded value can be obtained by simple concatenation. The two most common of them are:

increasing numeric significance with increasing memory addresses or increasing time, known as little-endian, and

its opposite, most-significant byte first, called big-endian.

Well known processor architectures that use the little-endian format include x86.

—Wikipedia [3]

I was using a x86 processor that uses the little-endian format, trying to read a file that contained 16-bit integers in big-endian format!

It's easier to imagine what happened, with binary representation of the decimal 1 in 16-bit big-endian and little-endian formats:

00000000 00000001 - Big endian
00000001 00000000 - Little endian

And vice-versa the decimal 256 is:

00000001 00000000 - Big endian
00000000 00000001 - Little endian

Now you can see: the program read the big-endian data and interpreted it as little-endian. So, all values were actually multiplied by 256!

Finally, a small piece of code helped to eliminate the problem:

short reverseShort(short s)
{
    unsigned char c1, c2;
    c1 = s & 255;
    c2 = (s >> 8) & 255;

    return (c1 << 8) + c2;
}

References

[1]	http://en.wikipedia.org/wiki/Voice_activity_detection

[2]	http://aurora.hsnr.de/index.html

[3]	http://en.wikipedia.org/wiki/Endianness

Emacs, LaTeX and pdf viewer

2009-04-14T12:00:00+03:00

My favourite editor Emacs provides a great support for LaTeX document preparation system. There are several modes that extend the default latex-mode. My favourite ones are: AUCTex (with Preview-latex) and CDLaTeX.

By-default LaTeX outputs DVI files and uses xdvi viewer with "View" command. However it's possible to configure LaTeX to outpud pdf files. In Emacs, you have to execute (M-x tex-pdf-mode) command to set the LaTeX's output to pdf. If you'd like to have a pdf output for all LaTeX documents, add

(setq TeX-PDF-mode t)

to your .emacs.

Now, let's add a pdf-viwer to be executed on "View" command (C-c C-v). I use GNOME's default Evince document viewer for pdf files. So, the code that should be added to .emacs is:

(defun pdfevince ()
   (add-to-list 'TeX-output-view-style
                 '("^pdf$" "." "evince %o %(outpage)")))

(add-hook  'LaTeX-mode-hook  'pdfevince  t) ; AUCTeX LaTeX mode

That's it!

Byte-compiled ChangeLog

2009-04-03T12:00:00+03:00

My dot emacs file is divided into .el pieces like this:

(load "~/.emacs/conf/env_internal.el")
(load "~/.emacs/conf/env_external.el")
(load "~/.emacs/conf/ui.el")
; ...

It's much easier to deal with functions grouped in different files rather than functions in one big .emacs.el.

The following "trick" automatically byte-compiles .el files as they're being saved in Emacs:

(defun autocompile nil
  (interactive)
  (require 'bytecomp)
  (if (numberp (string-match ".el" buffer-file-name))
        (byte-compile-file (buffer-file-name))))

(add-hook 'after-save-hook 'autocompile)

It worked perfect for me, until once I was updating a single ChangeLog file via Emacs. When I saved it, Emacs tried to byte-compile the file! That was weird.

The error should have not been in Emacs (that's true for 99.9% Emacs-related "bugs"), but somewhere in one of my configuration .el files. And probably there was something wrong with the byte-compile-on-save hook. The only function that could cause it was:

(if (numberp (string-match ".el" buffer-file-name))

Let's see what Emacs says about the string-match() function:

(string-match regexp string & optional start)

Returns the index of start of the first match for regexp in string, or nil.

So, string-match() works with regexps! And I have the "." symbol in the pattern. But period stands for a special character in regexps:

'.' (Period) is a special character that matches any single character except a newline. Using concatenation, we can make regular expressions like 'a.b', which matches any three-character string that begins with 'a' and ends with 'b'.

—Emacs help

So, (string-match ".el" buffer-file-name) function call is wrong! For example:

(string-match ".el" "ChangeLog")

returns 4.

I had to use a backslash to quote that period, like this:

(string-match "\.el" "ChangeLog")

But somehow, that didn't work either. An additional documentation exploration had shown that

When you use regular expressions in a Lisp program, each '\' must be doubled.

—Emacs help

Finally, I tried:

(string-match "\\.el" "ChangeLog")

And it worked! The result was nil. And the result for any file with .el extension was the start position of the ".el" string in file's name.

Here is the final version for automatic byte-compile when saving a file with .el extension:

(defun autocompile nil
  "compile itself if contains .el"
  (interactive)
  (require 'bytecomp)
  (if (numberp (string-match "\\.el" buffer-file-name))
      (byte-compile-file (buffer-file-name))))

(add-hook 'after-save-hook 'autocompile)

That was a funny problem and it took me about half an hour to deal with. Emacs help helps :)

Beginner's guide to creating a GNOME 2 applet with Python (Part I)

2009-02-25T12:00:00+02:00

Programming is an art. To be more specific, it's an everlasting art. There is no program that could completely satisfy a user. And I doubt that there is a program that could completely satisfy a programmer.

I spend hours in front of my computer. Of course that is very unhealthy for the eyes. I wanted to create a simple reminder applet - an eyes icon that changes to "bloody" eyes - indicating that it's time to relax.

This was my first GNOME applet, my first GTK and more or less serious Python experience and I'd like too share it with everyone, who deals with his or her first GNOME applet.

I hope, this tutorial will help you.

Part I: GNOME applet basics

Before we begin... Some technical information about my working environment. I used Ubuntu 9.04 and Python 2.6. So, when I'm saying that "this library loads files from here", I'm speaking of what I've found in my working environment. It could be different in yours.

1. Project's directory

First, lets create a directory for the applet. /home/user/applet/ (~/applet/) will do, of course you can create it as a symbolic link to your designated path. Here, /home/user/ is a path to your home directory. The source code of the applet will be located in /home/user/applet/src/ (~/applet/src/) directory.

2. How GNOME interacts with applets

In GNOME, an applet is a small application, designed to sit in the GNOME panel, providing quick and easy access to a control, such as a volume control, a clock, a network status display, or even a weather gauge [1].

Technically, applets are Bonobo controls [2] embedded in the Gnome panel. This means that there are few slight differences from stand-alone GNOME programs. The first difference is that each applet requires a 'server' file, which contains a description of the Bonobo capabilities [1].

GNOME searches for applet server files in /usr/lib/bonobo/servers directory.

The basic part of a simple server file contains server's id and location of an executable file (our file is a python script):

<oaf_info>
  <oaf_server iid="OAFIID:SampleApplet_Factory"
              type="exe"
              location="/home/user/applet/src/applet.py">
    <oaf_attribute name="repo_ids" type="stringv">
      <item value="IDL:Bonobo/GenericFactory:1.0"/>
      <item value="IDL:Bonobo/Unknown:1.0"/>
    </oaf_attribute>
    <oaf_attribute name="name"
                   type="string"
                   value="Sample Applet Factory"/>
    <oaf_attribute name="description"
                   type="string"
                   value="Sample Applet's factory that launches the applet"/>
  </oaf_server>

  <oaf_server iid="OAFIID:SampleApplet"
              type="factory"
              location="OAFIID:SampleApplet_Factory">
    <oaf_attribute name="repo_ids" type="stringv">
      <item value="IDL:GNOME/Vertigo/PanelAppletShell:1.0"/>
      <item value="IDL:Bonobo/Control:1.0"/>
      <item value="IDL:Bonobo/Unknown:1.0"/>
    </oaf_attribute>
    <oaf_attribute name="name" type="string" value="Sample Applet"/>
    <oaf_attribute name="description" type="string" value="description"/>
    <oaf_attribute name="panel:category:" type="string" value="Utility"/>
    <oaf_attribute name="panel:icon" type="string" value="no-picture-yet.png"/>
  </oaf_server>
</oaf_info>

For simplicity, the 'factory' is called SampleApplet_Factory and home/user/applet/src/applet.py is the absolute path to the executable Python script.

So, the fist step is to create a server file in /usr/lib/bonobo/servers. Let's name it SampleApplet.server.

Note

The root privileges are required to work with the SampleApplet.server file. Remember to correct the /home/user/ path

After that, the GNOME session could be restarted (restarting X-server via Ctrl+Alt+Backspace will do). Try adding the new applet to a panel. The dialog should look like this:

The applet's image is absent, because it doesn't exist in /usr/share/pixmaps. The applet engine searches for an image according to our .server file:

<oaf_attribute name="panel:icon" type="string" value="no-picture-yet.png"/>

The value="no-picture-yet.png" string should be changed to value="gnome-laptop.png". After restarting GNOME session the "Add to panel" dialog should look like this:

If by accident the gnome-laptop.png is absent, it could be saved to /usr/share/pixmaps from here:

It's time to write a few lines of code.

3.1 Launching the applet

Let's start with creating an applet.py script file in ~/applet/src.

The next step, is to define the interpreter that'll handle the script (this is where programming starts):

1	#!/usr/bin/env python

The /usr/bin is a standard directory on Unix-like operating systems that contains most of the executable files (i.e., ready-to-run programs) that are not needed for booting (i.e., starting) or repairing the system [3]

Some modules should be imported: The pyGTK module is needed to specify the GTK version used (2.x in this article), the gnome module, that contains all the useful classes and methods about the GNOME desktop environment, i.e. the applet class, and the gtk module, Python bindings for the GTK toolkit [4].

import sys

import gtk
import pygtk
import gnomeapplet

pygtk.require('2.0')

def applet_factory(applet, iid):
    label = gtk.Label("It works!")
    applet.add(label)
    applet.show_all()
    print('Factory started')
    return True

if __name__ == '__main__':   # testing for execution
    print('Starting factory')
    gnomeapplet.bonobo_factory('OAFIID:SampleApplet_Factory',
                               gnomeapplet.Applet.__gtype__,
                               'Sample Applet', '0.1',
                               applet_factory)

The applet_factory function receives the object to be initialized (the applet) and the bonobo activation ID that the new factory will implement. It returns True if no errors were reported. Then the bonobo_factory function is called [4].

Note

Remember to change the mode of the applet.py file to +x (execute/search), e.g. by running chmod 755 ~/applet/src/applet.py

The bonobo_factory(IID,Type,Description,Version,Callback) arguments are: * IID: The bonobo-activation id of the factory. * Type: the type of the created object. * Description * Version * Factory callback: the name of the factory function

"It works!" label should appear on a panel:

3.2 Debugging routine

The print() function calls should print the text to the output stream. But where is the output? A common issue when developing an applet is the debug process which means, knowing what it's failing and why. As an applet is nothing but a GTK+ application we can use an additional command line argument like "run-in-window" to put it in window-mode by creating a GTK+ window and inserting the applet in it [4].

It could be done making some changes in the code:

if __name__ == '__main__':   # testing for execution
    print('Starting factory')

    if len(sys.argv) > 1 and sys.argv[1] == '-d': # debugging
        mainWindow = gtk.Window()
        mainWindow.set_title('Applet window')
        mainWindow.connect('destroy', gtk.main_quit)
        applet = gnomeapplet.Applet()
        applet_factory(applet, None)
        applet.reparent(mainWindow)
        mainWindow.show_all()
        gtk.main()
        sys.exit()
    else:
        gnomeapplet.bonobo_factory('OAFIID:SampleApplet_Factory',
                                   gnomeapplet.Applet.__gtype__,
                                   'Sample applet', '0.1',
                                   applet_factory)

A GTK window is created, its title is set and the 'destroy' signal is connected to the gkt.main_quit callback. It means that gtk.main_quit() will be automatically called, when the window is destroyed (closed). We create GNOME applet's instance and make it a child control (widget) of the window we've just created. The gtk.main() function runs the main loop until the gtk.main_quit() function is called [5]. The GTK+ main loop's primary role is to listen for events on a file descriptor connected to the X server, and forward them to widgets [6].

Now the applet can be launched from a console with the -d key, e.g. ./applet.py -d. The applet should appear in a window. It's a common GNOME window:

References

[1]	(1, 2) http://projects.gnome.org/ORBit2/appletstutorial.html

[2]	http://en.wikipedia.org/wiki/Bonobo_(component_model)

[3]	http://www.linfo.org/usr_bin.html

[4]	(1, 2, 3) http://www.pygtk.org/articles/applets_arturogf

[5]	http://www.pygtk.org/docs/pygtk/gtk-functions.html

[6]	http://developer.gnome.org/doc/GGAD/sec-mainloop.html

The legacy of the initramfs optimization

2009-01-11T12:00:00+02:00

I was shocked receiving the Target filesystem doesn't have /sbin/init message. I've surfed the Internet for possible solutions - none of them helped. I had to remember everything I've changed in my system during past 3 months. Finally, I remembered..

I'm a Linux person. I'm a Linux person since March 2008. My current OS is Ubuntu Linux (x86). But I don't like to be "generic". My x64 processor is not generic, it's Core2. I have one particular Ethernet, wi-fi and bluetooth adapter, one sound and video card. I'll never use the 95% of Linux-supported hardware..

So, I've compiled my custom "sony-vaio" kernel, still always downloaded the generic kernel upgrades (to have a native kernel full of modules in a case of emergency).

This happened when I've updated the system.

Ubuntu has downloaded the linux-image-2.6.11-generic kernel. I wanted to check, If my motioneye camera works under it, so, I rebooted the computer and selected the Ubuntu 8.10, kernel 2.6.27-11-generic option. Bump!

Target filesystem doesn't have /sbin/init

I had no ideas of what did that mean. I rebooted the laptop via Ctrl+Alt+PrintScreen+B. Fortunately my custom kernel was working fine. After an hour of surfing the Internet I already knew, that possibly the kernel can't mount a root file system. So, the difference was between my custom kernel and the generic kernel. I was annoyed. There was nothing in my kernel, that generic kernel didn't have. But wait! While creating my custom kernel, I've compiled the ext3 filesystem support statically into it. But why the generic kernel can't load the ext3 module?

Suddenly it dawned upon me.

There is initial ram disk stuff at the beginning of system load process.

The initial ramdisk, or initrd is a temporary file system commonly used by the Linux kernel during boot. The initrd is typically used for making preparations before the real root file system can be mounted [1]

So, possibly the initrd image created for the generic kernel didn't contain the ext3 module. How could that happen? It could, if you're a person who likes to optimize everything that could be optimized. For example, the system's boot-time.

There is a minimal set of modules enough to load a system both from hibernate (suspend2disk) and powered-off conditions. By default, the initrd contains "most" modules (see /etc/initramfs-tools/initramfs.conf file). In my case the MODULES option was changed like this:

#MODULES=most
MODULES=list

I used next steps to generate the modules list:

Boot kernel with init=/bin/sh option
Execute sudo lsmod | tail -n +2 | sort | awk '{print $1;}' > /etc/initramfs-tools/modules
Execute update-initramfs -v -d -k \`uname -r` && update-initramfs -v -c -k \`uname -r`

The second line copies all loaded modules' names to /etc/initramfs-tools/modules file. The third line updates the current kernel's initramfs file.

As you might already found out, there was no ext3 module in the list, because my custom kernel didn't need an ext3 module!

But Ubuntu's generic kernel has the ext3 as a module! So, to boot a system with generic kernel, I should have had an "ext3" line in /etc/initramfs-tools/modules.

It didn't take long. I updated the file and called update-initramfs -c -k all to regenerate the init ram disks for both kernels (actually I didn't need to do that for my custom kernel :).

At last, I could load the system with new 2.6.27.11 kernel.

[1]	http://en.wikipedia.org/wiki/Initrd

Zaur's Thoughts

The "Handover" teams

Working together

I was there, Gandalf

Fixing low display brightness on LUKS password prompt

The Power of Atomic Habits

xkbcommon and my custom Russian phonetic layout

Custom layout - the naive way

Bits about XKB

Historical reference

Custom layout - the smart way

References

Framework Laptop is awesome!

A rant about a broken screen

Imagine a laptop

The chassis

Using GNU/Linux

Shall you join the club?

My setup specs and other numbers for nerds

Hexagonal architecture and Python - Part IV: Lightweight integration tests

1. Pick a behaviour unit

2. Write a test for its API

3. Write a minimal implementation to pass the test

4. Write a test to verify downstream interaction

Mocking Django model persistence

5. Write a minimal implementation to pass the test

7. Refactor

Mocks are for humans, not machines!

Remove duplication in tests

1. Pick a behaviour unit

2. Write a test for its API

6. Fix failing tests

Mocking existing users

7. Refactor BIG

Peopleware

Testified Documentation

Specification by Example and Cucumber

Testified Documentation

Feedback

The dark patterns of seat selection in Turkish Airlines

Ticket prices gone wrong

Seat selection gets weird

Seat selection gets worse

Support can't do anything

Five stages of grief

Bad luck?

Aftermath

Building Microservices - more than just another "microservices book"

Teams Topologies - Book Review

The Mythical Man-Months - thoughts

1 - The Tar Pit

2 - The Mythical Man-Months

3 - The Surgical Team

4 - Aristocracy, Democracy and System Design

5 - The second-system effect

6 - Passing the word

7 - Why did the tower of Babel fail?

8 - Calling the shot

9 - Ten Pounds in a Five-Pound sack

10 - The Documentary Hypothesis

11 - Plan To Throw One Away

12 - Sharp tools

13 - The Whole and The Parts

14 - Hatching a catastrophe

15 - The other face

17 - No Silver Bullet

16 - EPILOGUE

The DevOps Handbook - review

The curious case of denied repository access in Github Actions workflow

Clean Architecture - book review

Inside the book

The hidden gem

Thoughts

Good practices: Software deployment and feature releases

The problem

A good practice

Specification by example - book review

Key process patterns in a tiny nutshell

Thoughts

Related