Planet Grep

Planet'ing Belgian FLOSS people

Planet Grep is maintained by Wouter Verhelst. All times are in UTC.

September 24, 2021

I published the following diary on isc.sans.edu: “Keep an Eye on Your Users Mobile Devices (Simple Inventory)“:

Today, smartphones are everywhere and became our best friends for many tasks. Probably your users already access their corporate mailbox via a mobile device. If it’s not yet the case, you probably have many requests to implement this. They are two ways to achieve this: you provide corporate devices to all users. From a risk perspective, it’s the best solution: you select the models and control them. But it’s very expensive and people don’t like to carry two devices (a personal and a corporate one). Hopefully, if you use a Microsoft Exchange platform, there are ways to authorize personal devices to access corporate emails with a software component called ActiveSync. ActiveSync allows deploying basic security policies like forcing the device to be locked with a password, force a minimum password length, etc. However, it’s not a real MDM (“Mobile Device Management”)… [Read more]

The post [SANS ISC] Keep an Eye on Your Users Mobile Devices (Simple Inventory) appeared first on /dev/random.

September 23, 2021

I published the following diary on isc.sans.edu: “Excel Recipe: Some VBA Code with a Touch of Excel4 Macro“:

Microsoft Excel supports two types of macros. The legacy format is known as “Excel4 macro” and the new (but already used for a while) is based on VBA. We already cover both formats in many diaries. Yesterday, I spotted an interesting sample that implements… both!

The malicious file was delivered through a classic phishing email and is called “Document_195004540-Copy.xls” (SHA256:4f4e67dccb3dfc213fac91d34d53d83be9b9f97c0b75fbbce8a6d24f26549e14). The file is unknown on VT at this time. It looks like a classic trap… [Read more]

The post [SANS ISC] Excel Recipe: Some VBA Code with a Touch of Excel4 Macro appeared first on /dev/random.

September 20, 2021

Vous pouvez dès à présent précommander la version audiolivre de Printeurs et donner votre avis sur la voix à choisir. Ce qui me fait réfléchir à la voix, au bruit, au marketing, au crowdfunding et aux inondations…

Mon roman Printeurs va prendre de la voix et sera bientôt produit sous forme d’un audiolivre. Un format avec lequel je ne suis pas du tout familier (je suis un lecteur visuel), mais dont je me réjouis d’écouter le résultat. Je suis d’ailleurs curieux d’avoir les avis des gros consommateurs de livres audio sur ce qui fait un « bon » audiolivre. Qu’aimez-vous ? À quoi doit-on faire attention ? Et qu’est-ce qui vous fait arrêter votre écoute à tous les coups ?

Afin de financer cette entreprise, mon éditeur a mis en place une campagne de crowdfunding au cours de laquelle vous pouvez précommander la version audio de Printeurs. Vous aurez même la possibilité de donner votre avis sur des voix présélectionnées. Je suis vraiment curieux de lire l’avis des amateurs du genre.

Précommander la version audio de Printeurs :https://fr.ulule.com/ludomire/?reward=752654
Voter pour votre voix préférée (lien réservé aux souscripteurs) : https://fr.ulule.com/ludomire/news/decouvrez-le-casting-de-voix-pour-le-livre-audio-p-312900
Explications techniques sur l’adaptation audio : https://fr.ulule.com/ludomire/news/les-adaptations-en-livres-audio-312264/

La voix

La voix est un médium particulier. Lorsqu’on parle, le charisme et les intonations ont souvent plus d’importance que le contenu lui-même. Les incohérences sont gommées par le rythme. Un exemple parmi tant d’autres : j’ai été récemment interviewé par Valentin Demé pour le podcast Cryptoast afin de parler des monopoles et de la blockchain.

Pendant une heure, je parle en laissant mes idées vagabonder. Des idées bien moins formées que ce que j’écris d’habitude, des intuitions, des explorations. D’après les réactions, ce que je dis semble intéressant. Mais il faudrait garder à l’esprit que, à l’exception d’un discours entièrement préparé (un cours par exemple), les informations sont beaucoup plus aléatoires et toujours à prendre avec un grain de sel. Paradoxalement, la voix est plus convaincante alors qu’elle est moins rigoureuse. On apprend et réfléchit dans les livres, on se fait convaincre par les discours. La politique est une affaire de voix. La science est une affaire d’écrit.

Ploum sur Cryptoast : https://www.youtube.com/watch?v=vq6o_30LxJM

Le crowdfunding en question

Cette campagne de crowdfunding ne concerne pas que Printeurs. C’est avant tout une campagne englobant toutes les nouveautés de la collection SFFF Ludomire notamment la version papier en quatre volumes du One Minute de Thierry Crouzet. One Minute est un roman de SF se déroulant durant… une seule et unique minute, comme le dit le titre. Chacun des 365 chapitres dure… une minute. J’ai beaucoup apprécié la version Wattpad et je me réjouis de lire cette version entièrement retravaillée.

Encore de la pub pour une campagne de crowdfunding ? Autant je suis enthousiaste sur le contenu, autant je vous comprends.

La campagne crowdfunding de Printeurs m’a laissé un souvenir assez amer. Certes, elle a été un incroyable succès (grâce à vous qui me lisez) mais j’ai eu l’impression de spammer sans arrêt mes réseaux. De produire le bruit contre lequel je me bats tellement. J’en suis sorti lessivé et ceux qui me suivent également. Le problème, comme me l’a fait remarquer mon éditeur, c’est que le spam… ça fonctionne !

Ces campagnes sont désormais beaucoup plus nombreuses. Il faut se différencier, se professionnaliser. Bref, le marketing redevient essentiel alors que, dans mon esprit, l’un des buts initiaux du crowdfunding était de se passer de cette étape. Ironiquement, le marketing se concentre, non plus sur le produit lui-même, mais sur la promotion… de la campagne de financement ! Alors que cette méthode est censée rapprocher le créateur du consommateur, elle l’éloigne paradoxalement.

C’est un questionnement que se pose également Lionel, mon éditeur. Comment se faire connaitre et se financer sans pour autant tomber dans le spam ? Thierry lui-même m’a confié ne pas avoir la moindre envie de promouvoir la campagne liée à la parution de son roman.

La campagne Ludomire 2021 :https://fr.ulule.com/ludomire/
Crouzet raconte One Minute : https://tcrouzet.com/2021/09/14/de-lecriture-de-la-vie-du-roman/
Réflexions sur le crowdfunding : http://ludom.cc/index.php/2021/09/08/levolution-du-crowdfunding-selon-mon-experience/

Le prix libre ?

La problématique n’est pas uniquement limitée au crowdfunding. Le prix libre est également impacté. Il y a quelques années, je fus l’un des pionniers francophones du Prix Libre sur le Web à travers un billet au titre provocant : « Ce blog est payant ! ». Force est de constater que le concept s’est largement popularisé, au point d’avoir sa page Wikipédia.

Un peu trop popularisé peut-être. Désormais, le prix libre est partout et, comme par magie, se fédère sur quelques plateformes centralisées. Alias parle justement de son questionnement à propos de Tipeee, plateforme que j’ai également quittée.

Il y’a une fatigue indéniable du public : nous sommes sollicités tout le temps pour financer tous les projets imaginables, depuis les aiguilles à tricoter connectées révolutionnaires à l’installation de pots de fleurs sur la voirie de notre quartier. Outre les sous, il s’agit de jongler entre les différentes plateformes, les sommes, récurrentes ou non. J’ai également le sentiment que ce sont toujours les mêmes qui contribuent à tout, pas spécialement les plus aisés.

J’en ai déduit une sorte de loi générale sur Internet qui fait que toutes les bonnes idées sont soit pas assez populaires pour être largement utiles, soit tellement populaires que ça en fait des mauvaises idées. Les réseaux sociaux, la mobilité en sont les illustrations les plus marquantes. Le prix libre est-il en train de suivre cette voie ?

Les alternatives que nous construisons ne sont-elles séduisantes que parce qu’elles sont des alternatives justement ? Le succès n’entraîne-t-il pas obligatoirement un excès inexorable ? Je pense par exemple au réseau minimaliste Gemini dont je vous ai parlé.

Le prix libre sur Wikipedia :https://fr.wikipedia.org/wiki/Prix_libre
Alias quitte Tipeee : https://erdorin.org/il-est-temps-de-changer-de-tipeee/
Le drama tipeee (lien gemini) : gemini://lord.re/fast-posts/62-le-drama-tipee-2021/index.gmi

Le livre suspendu et les inondations

Face à ce constat, j’ai décidé de retirer tous les appels aux dons sur mon blog et encourager l’achat de livres. Je trouve que les livres sont parmi les objets les plus symboliques de l’humanité. Un livre n’est jamais inutile. Il peut dormir des années voire des siècles sur des étagères avant de renaître et d’illuminer une journée ou une vie. Le livre, y compris au format électronique, c’est le cadeau par excellence : un monde à découvrir, un objet à transmettre, des explorations intellectuelles à partager, dans le présent et le futur.

Acheter mes livres :  https://ploum.net/livres/

Le livre papier ne connait que deux dangers : le feu et l’eau. C’est malheureusement ce qui est arrivé cet été dans mon pays. Si je n’ai pas été personnellement touché, ce fut bien le cas de ma ville (Ottignies) et surtout de la région d’où sont originaires mon épouse, mes parents et mes ancêtres (vallée de la Vesdre).

Si vous avez perdu votre bibliothèque suite aux inondations ou si vous connaissez quelqu’un dans le cas, envoyez-moi un petit mot, je vous ferais parvenir un exemplaire de Printeurs. Je dispose également de plusieurs ouvrages de la collection Ludomire que j’enverrai volontiers aux bibliothèques qui cherchent à se reconstruire. N’hésitez pas à prendre contact et à faire l’intermédiaire pour des personnes à qui cela pourrait apporter un petit sourire. C’est toujours bon à prendre dans cette période difficile de reconstruction où la vie, comme la Vesdre, semble avoir repris son cours normal. Sauf pour ceux qui ont tout perdu, qui vivent dans l’humidité, qui sont nourris par la Croix-Rouge et dont le cœur s’étreint d’angoisse à chaque nouvelle pluie un peu drue.

Il me reste quelques exemplaires du livre « Les aventures d’Aristide, le lapin cosmonaute ». Ils sont normalement en vente, mais je les offre avec plaisir aux familles avec enfant (idéalement 5-9 ans) qui sont en manque de livre, que ce soit à cause des inondations ou pour des raisons qui ne me regardent pas.

Envoyez-moi un mail en précisant quel livre vous ferait plaisir (ou bien les deux) à l’adresse suspendu at ploum.net.

Bonne lecture et bonne écoute !

Si la relation entre les humains et la technologie vous intéresse, abonnez-vous par mail ou par RSS pour ne rater aucun billet (max 2 par semaine). Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, offrez et partagez ses livres.

Ce texte est publié sous la licence CC-By BE.

September 19, 2021

playbook

I wrote a few articles:

on my blog on how to use cloud images with cloud-init on a “non-cloud” environment.

I finally took the time to create an Ansible role for it. You’ll find the READE.md below.

Virt_install_vm 1.0.0 is available at: https://github.com/stafwag/ansible-role-virt_install_vm

Have fun!


Ansible Role: virt_install_vm

An Ansible role to install a libvirt virtual machine with virt-install and cloud-init. It is “designed” to be flexible.

An example template is provided to set up a Debian system.

Requirements

The role is wrapper around the following roles:

Install the required roles with

$ ansible-galaxy install -r requirements.yml

this will install the latest default branch releases.

Or follow the installation instruction for each role on Ansible Galaxy.

https://galaxy.ansible.com/stafwag

Supported GNU/Linux Distributions

It should work on most GNU/Linux distributions. cloud-cloudds is required. cloud-clouds was available on Centos/RedHat 7 but not on Redhat 8. You’ll need to install it manually to use it role on Centos/RedHat 8.

  • Archlinux
  • Debian
  • Centos 7
  • RedHat 7
  • Ubuntu

Role Variables and templates

Variables

See the documentation of the roles in the Requirements section.

  • virt_install_vm: “namespace”

    • skip_if_deployed: boolean default: false.

                            When true:
                              Skip role if the VM is already deployed. The role will exit successfully.
                            When false:
                              The role will exit with an error if the VM is already deployed.
      

Templates.

  • templates/simple_debian: Example template to create a Debian virtual machine.

This template use cloud_localds.cloudinfo to configure the cloud-init user-data.

See the Usage section for an example.

Usage

Create a virtual machine template

This is a file with the role variables to set set up a virtual machine with all the common settings for the virtual machines. In this example vm.hostname and vm.ip_address can be configured for each virtual machine.

  • debian_vm_template.yml:
qemu_img:
  dest: "/var/lib/libvirt/images/.qcow2"
  format: qcow2
  src: /Downloads/isos/debian/cloud/debian-10-generic-amd64.qcow2
  size: "50G"
  owner: root
  group: kvm
  mode: 660
cloud_localds:
  dest: "/var/lib/libvirt/images/_cloudinit.iso"
  config_template: "templates/simple_debian/debian.j2"
  network_config_template: "templates/simple_debian/debian_netconfig.j2"
  cloud_config:
    system_info:
      default_user:
        name: ansible
        passwd: ""
        ssh_authorized_keys:
          - ""
    network:
      dns_nameservers:
        9.9.9.9
      dns_search:
        intern.local
      interface:
        name:
          enp1s0
        address:
          ""
        gateway:
          192.168.123.1
    disable_cloud_init: true
    reboot:
      true
virt_install_import:
  wait: 0
  name: ""
  os_type: Linux
  os_variant: debian10
  network: network:default
  graphics: spice
  disks:
    - "/var/lib/libvirt/images/.qcow2,device=disk"
    - "/var/lib/libvirt/images/_cloudinit.iso,device=cdrom"

Playbook

Playbook to setup a virtual machine:

- name: Install tstdebian2
  hosts: kvmhost
  become: true
  vars:
    vm:
      hostname:
        tstdebian2
      ip_address:
        192.168.123.2/24
  pre_tasks:
    - name: Load the vm template
      include_vars: debian_vm_template.yml
    - name: display qemu_img
      debug:
        msg: 
          - "qemu_img: "
  roles:
    - stafwag.virt_install_vm

September 17, 2021

I published the following diary on isc.sans.edu: “Malicious Calendar Subscriptions Are Back?“:

Did this threat really disappear? This isn’t a brand new technique to deliver malicious content to mobile devices but it seems that attackers started new waves of spam campaigns based on malicious calendar subscriptions. Being a dad, you can imagine that I always performed security awareness with my daughters. Since they use computers and the Internet, my message was always the same: “Don’t be afraid to ask me, there are no stupid questions or shame if you think you did something wrong”… [Read more]

The post [SANS ISC] Malicious Calendar Subscriptions Are Back? appeared first on /dev/random.

September 15, 2021

De VRT radio streams, zoals deze (http://icecast.vrtcdn.be/radio1-high.mp3) hebben sinds kort de neiging om reklame te spelen als je ze start. Niet altijd, maar wel regelmatig.

Zonet (21u en enkele seconden) wou ik naar het nieuws luisteren , maar er begon reklame te spelen. Ik laadde de stream opnieuw en de reklame begon opnieuw (terwijl het nieuws bezig was, maar dat kreeg ik dus niet).

Een paar dagen geleden had ik dat ook al met de Radio 1 stream, telkens reklame als je de stream start. Dit is irritant.

Tijd dus om het script aan te passen en de eerste 60 seconden van de stream te muten. De cronjob om het nieuws te spelen kan dan een minuut eerder starten.

 

UPDATE 2021-09-15: dit werkt

#!/bin/bash

export HOME=/var/www
pkill mplayer

mplayer -volume 0 -slave -input file=/var/www/master http://icecast.vrtcdn.be/radio1-high.mp3 &

sleep 60

echo volume 100 1 > /var/www/master
exit

Naming conventions. Picking the right naming convention is easy if you are all by yourself, but hard when you need to agree upon the conventions in a larger group. Everybody has an opinion on naming conventions, and once you decide on it, you do expect everybody to follow through on it.

Let's consider why naming conventions are (not) important and consider a few examples to help in creating a good naming convention yourself.

Naming conventions imply standardization

When you settle on a naming convention, you're effectively putting some standardization in place which you expect everybody to follow, and which should also cover 100% of the cases. So, when assessing a possible naming convention, first identify what standards you need to enforce and are future proof.

Say you are addressing database object naming conventions. Are you able to enforce this at all times? You might want to start tables with tbl_ and views with vw_, but when you are dealing with ISV software, they generally do not allow such freedom on 'their' database definitions. Your DBAs thus will learn to deal with setups that are more flexible anyway.

Using a naming convention for internal development is of course still a possible path to pursue. But in that case, you will need to look at the requirements from the development teams (and related stakeholders).

Standardization does not imply naming conventions

The inverse isn't true: even though you might have certain standards in place, it doesn't mean that the object names need to reflect the standards. If your company standardizes on two operating systems (like Red Hat Enterprise Linux and Microsoft Windows), it doesn't mean that server names have to include an identifier that maps to Linux or Windows.

I personally often fall into this trap - I see standards, so I want to see them fixed in the naming convention because that allows better control over following the standards. But naming conventions aren't about control, they are about exposing identifiable information.

Structure it for readability

Trying to add too much information in a naming convention makes it more complex for users to deal with. You might be able to read and understand the naming convention immediately upon seeing it, but are all the other stakeholders equally invested in understanding the naming conventions?

Say that you have a hostname that looks like so:

sppchypkc05m01.reg1.internal.company.com

While I can tell you that this name comes from the following convention, it might be overdoing things:

  • s to identify it is a server
  • p to identify it is a physical server
  • p to identify it is hosted in a production environment
  • c to identify it is a cattle-alike managed server
  • hypk to identify the ownership (in this case, hypervisor usage, KVM)
  • c05 to identify it is the fifth cluster
  • m01 to identify it is the first master node
  • reg1 to identify the first region (location)

Even if you still want to include this information, using separators might make this more obvious. For instance, for the given name, I would suggest splitting this as follows:

sppc-hypk-c05m01.reg1.internal.company.com

The first two parts are then global naming convention requirements, with the first set being about the type of system whereas the second is about ownership, and the third is then a naming convention specific to that owner.

Choose what information to expose easily

Assets that follow a certain naming convention provide information about that asset that a reader can immediately assume, without having to do additional lookups. The intention here is that you want to define important information that many stakeholders will need immediately to support their work (and thus their efficiency). Insights that are useful for a select set of stakeholders might not be suitable for a naming convention (or at least not a global one).

You should consider every stakeholder that comes in contact with the name of the asset, and how that stakeholder would obtain the information they need. If you have a central, easily accessible configuration management system, it might be possible to have many structured insights exposed through that interface, but is that useful when you are dealing with lists of assets?

Suppose you do not include the host class for hostnames, with the host class being what class of system the host is (server, workstation, router, firewall, appliance, ...). Does your SOC team need this insight every time they are going through events? Does your helpdesk need that information? What about the resource managers?

If all these stakeholders do need that information over and over again, it might be sensible to include it in the naming convention. If, however, only a few stakeholders need that information, you might want to expose that easily through different means. For instance, resource managers might be able to easily join that information with the asset management system information.

Choose what information NOT to expose easily

Sometimes, you want to have some information about objects easily available, but not for everybody. It might be information that can be abused for nefarious purposes. In that case, you want this information to be shielded and only offered to authenticated and authorized users. For instance, if you use separate accounts for administering systems, you might not want to add information about what type of admin account it is, as account enumeration might reveal too much immediately and provide attackers with better insights.

So, rather than having ken_adadmin for Ken's Active Directory administration account, stick to a nonsensible account identification like ua1503 (user account 1503). Stakeholders that need information about accounts, in this case, can still notice it is a user account rather than a system or machine account and will need to query the central repositories for more information (such as AD to get information about the user - and don't forget to add sensitive users to, for instance, the Protected Users group in AD).

Use layered naming conventions

With "global naming convention" I am suggesting the ability to add naming conventions for specific purposes, but leave that open in general. A server name could, for instance, require an indication of the environment (production or not) and the fact that it is a server (and not a workstation), but leave a part of the name open for the administrators. The administrators can then add their local naming convention to it.

An active directory group, for instance, might have a standard global naming convention (usually the start of the group name) and leave the second part open, whereas specific teams can then use that part to add in their local naming convention. Groups that are used for NAS access might then use a naming convention to identify which NAS share and which privileges are assigned, whereas a group that is used for remote access support can use VPN naming conventions.

The University of Wisconsin has their Campus Active Directory - Naming Convention published online, and the workstation and server object part is a good example of this: while the objects in AD have to follow a global naming convention (because Active Directory is often core to an organization) it leaves some room for local department policies to assign their own requirements: <department><objectfunction>-<suffix> only has the first two fields standardized globally, with the <suffix> field left open (but within certain length constraints).

Consider the full name for your naming conventions

If you do want to add information in a naming convention, do not consider this purely on a single object type, but at the full name. A hostname by itself is just a hostname, but when you consider the fully qualified hostname (thus including domain names) you know that certain information points can be put in the domain name rather than the hostname. The people over at Server Density have a post titled "Server Naming Conventions and Best Practices" where they describe that the data center location (for the server) is a subdomain.

Another example is for databases, where you not only have a table, but also the database in which the table is located. Hence, ownership of that table can easily be considered on the database level.

Learn from mistakes or missing conventions

As you approach naming conventions, you will make mistakes. But before making mistakes yourself, try looking out for public failures that might have been due to (bad or missing) naming conventions. Now, most public root cause analysis reports do not go in-depth on the matter completely, but they do provide some insights we might want to learn from.

For instance, the incident that AWS had on February 28th, 2017, has a Summary of the Amazon S3 Service Disruption in the Northern Virginia (US-EAST-1) Region. While there is no immediate indication about the naming conventions used (mainly that a wrong command input impacted more servers than it should), we could ask ourselves if the functional purpose of the servers was included in the name (or, if not in the name, if it was added in other labeling information that the playbook should use). The analysis does reveal that AWS moved on to implement partitions (which they call cells), and the cell name will likely become part of the naming convention (or other identifiers).

Also internally, it is important to go over the major incidents and their root causes, and see if the naming conventions of the company are appropriate or not.

Still need examples?

While most commercial companies will not expose their own naming conventions (as there is no value for them to receive, and it exposes information that malicious users might abuse), many governmental agencies and educational institutions do have this information publicly available, given their organization public nature. Hence, searching for "naming convention" on *.gov and *.edu already reveals many examples.

Personally, I am still a stickler for naming conventions, but I am slowly accepting that some information might be better exposed elsewhere.

Feedback? Comments? Don't hesitate to drop me an email, or join the discussion on Twitter.

My laptop is a 2011 MacBook Air. I’m not a huge Apple fan, it’s just that at the time it had the most interesting hardware features compared to similar laptops. And it’s quite sturdy, so that’s nice.

Over the years I have experimented with installing Linux in parallel to the OS X operating system, but in the end I settled on installing my favorite Linux tools inside OS X using Homebrew, because having two different operating systems on one laptop was Too Much Effortâ„¢. In recent times Apple has decided, in it’s infinite wisdom (no sarcasm at all *cough*), that it will no longer provide operating system upgrades for older hardware. Okay, then. Lately the laptop had become slow as molasses anyway, so I decided to replace OS X entirely with Ubuntu. No more half measures! I chose 20.04 LTS for the laptop because reasons. 🙂

The laptop was really slow…

According to the Ubuntu Community Help Wiki, all hardware should be supported, except Thunderbolt. I don’t use anything Thunderbolt, so that’s OK for me. The installation was pretty straightforward: I just created a bootable USB stick and powered on the Mac with the Option/Alt (⌥) key pressed. Choose EFI Boot in the Startup Manager, and from there on it’s all a typical Ubuntu installation.

screenshot
Startup Manager

I did not bother with any of the customizations described on the Ubuntu Wiki, because everything worked straight out of the box, and besides, the wiki is terribly outdated anyway.

The end result? I now have a laptop that feels snappy again, and that still gets updates for the operating system and the installed applications. And it’s my familiar Linux. What’s next? I’m thinking about using Ansible to configure the laptop.

To finish, I want to show you my sticker collection on the laptop. There’s still room for a lot more!

sticker collection on my laptop. Photo copyright: me.

The post Installing Ubuntu 20.04 LTS on 2011 MacBook Air appeared first on amedee.be.

September 10, 2021

Cover Image

Cultural Assimilation, Theory vs Practice

The other day, I read the following, shared 22,000+ times on social media:

"Broken English is a sign of courage and intelligence, and it would be nice if more people remembered that when interacting with immigrants and refugees."

This resonates with me, as I spent 10 years living on the other side of the world. Eventually I lost my accent in English, which took conscious effort and practice. These days I live in a majority French city and neighborhood, as a native Dutch speaker. When I need to call a plumber, I first have to go look up the words for "drainage pipe." When my barber asks me what kind of cut I want, it mostly involves gesturing and "short".

This is why I am baffled by the follow-up, by the same person:

"Thanks to everyone commenting on the use of 'broken' to describe language. You're right. It is problematic. I'll use 'beginner' from now on."

It's not difficult to imagine the pile-on that must've happened for the author to add this note. What is difficult to imagine is that anyone who raised the objection has actually ever thought about it.

mines

Minesweeper

Consider what this situation looks like to an actual foreigner who is learning English and trying to speak it. While being ostensibly lauded for their courage, they are simultaneously shown that the English language is a minefield where an expression as plain as "broken English" is considered a faux pas, enough to warrant a public correction and apology.

To stay in people's good graces, you must speak English not as the dictionary teaches you, but according to the whims and fashions of a highly volatile and easily triggered mass. They effectively demand you speak a particular dialect, one which mostly matches the sensibilities of the wealthier, urban parts of coastal America. This is an incredibly provincial perspective.

The objection relies purely on the perception that "broken" is a word with a negative connotation. It ignores the obvious fact that people who speak a language poorly do so in a broken way: they speak with interruptions, struggling to find words, and will likely say things they don't quite mean. The dialect demands that you pretend this isn't so, by never mentioning it directly.

But in order to recognize the courage and intelligence of someone speaking a foreign language, you must be able to see past such connotations. You must ignore the apparent subtleties of the words, and try to deduce the intended meaning of the message. Therefor, the entire sentiment is self-defeating. It fell on such deaf ears that even the author seemingly missed the point. One must conclude that they don't actually interact with foreigners much, at least not ones who speak broken English.

The sentiment is a good example of what is often called a luxury belief: a conviction that doesn't serve the less fortunate or abled people it claims to support. Often the opposite. It merely helps privileged, upper-class people feel better about themselves, by demonstrating to everyone how sophisticated they are. That is, people who will never interact with immigrants or refugees unless they are already well integrated and wealthy enough.

By labeling it as "beginner English," they effectively demand an affirmation that the way a foreigner speaks is only temporary, that it will get better over time. But I can tell you, this isn't done out of charity. Because I have experienced the transition from speaking like a foreigner to speaking like one of them. People treat you and your ideas differently. In some ways, they cut you less slack. In other ways, it's only then that they finally start to take you seriously.

Let me illustrate this with an example that sophisticates will surely be allergic to. One time, while at a bar, when I still had my accent, I attempted to colloquially use a particular word. That word is "nigga." With an "a" at the end. In response, there was a proverbial record scratch, and my companions patiently and carefully explained to me that that was a word that polite people do not use.

No shit, Sherlock. You live on a continent that exports metric tons of gangsta rap. We can all hear and see it. It's really not difficult to understand the particular rules. Bitch, did I stutter?

Even though I had plenty of awareness of the linguistic sensitivities they were beholden to, in that moment, they treated me like an idiot, while playing the role of a more sophisticated adult. They saw themselves as empathetic and concerned, but actually demonstrated they didn't take me fully seriously. Not like one of them at all.

If you want people's unconditional respect, here's what did work for me: you go toe-to-toe with someone's alcoholic wine aunt at a party, as she tries to degrade you and your friend, who is the host. You effortlessly spit back fire in her own tongue and get the crowd on your side. Then you casually let them know you're not even one of them, not one bit. Jawdrops guaranteed.

This is what peak assimilation actually looks like.

Ethnic food

The Ethnic Aisle

In a similar vein, consider the following, from NYT Food:

"Why do American grocery stores still have an ethnic aisle?

The writer laments the existence of segregated foods in stores, and questions their utility. "Ethnic food" is a meaningless term, we are told, because everyone has an ethnicity. Such aisles even personify a legacy of white supremacy and colonialism. They are an anachronism which must be dismantled and eliminated wholesale, though it "may not be easy or even all that popular."

We do get other perspectives: shop owners simply put products where their customers are most likely to go look for them. Small brands tend to receive obscure placement, while larger brands get mixed in with the other foods, which is just how business goes. The ethnic aisle can also signal that the products are the undiluted original, rather than a version adapted to local palates. Some native shoppers explicitly go there to discover new ingredients or flavors, and find it convenient.

More so, the point about colonialism seems to be entirely undercut by the mention of "American aisles" in other countries, containing e.g. peanut butter, BBQ sauce and boxed cake mix. It cannot be colonialism on "our" part both when "we" import "their" products, as well as when "they" import "ours". That's just called trade.

Along the way, the article namedrops the exotic ingredients and foreign brands that apparently should just be mixed in with the rest: cassava flour, pomegranate molasses, dal makhani, jollof rice seasoning, and so on. We are introduced to a whole cast of business owners "of color," with foreign-sounding names. We are told about the "desire for more nuanced storytelling," including two sisters who bypassed stores entirely by selling online, while mocking ethnic aisles on TikTok. Which we all know is the most nuanced of places.

I find the whole thing preposterous. In order to even consider the premise, you already have to live in an incredibly diverse, cosmopolitan city. You need to have convenient access to products imported from around the world. This is an enormous luxury, enabled by global peace and prosperity, as well as long-haul and just-in-time logistics. There, you can open an app on your phone and have top-notch world cuisine delivered to your doorstep in half an hour.

For comparison, my parents are in their 70s and they first ate spaghetti as teenagers. Also, most people here still have no clue what to do with fish sauce other than throw it away as soon as possible, lest you spill any. This is fine. The expectation that every cuisine is equally commoditized in your local corner store is a huge sign of privilege, which reveals how provincial the premise truly is. It ignores that there are wide ranging differences between countries in what is standard in a grocery store, and what people know how to make at home.

Even chips flavors can differ wildly from country to country, from the very same multinational brands. Did you know paprika chips are the most common thing in some places, and not a hipster food?

paprika chips by lays

Crucially, in a different time, you could come up with the same complaints. In the past it would be about foods we now consider ordinary. In the future it would be about things we've never even heard of. While the story is presented as a current issue for the current times, there is nothing to actually support this.

To me, this ignorance is a feature, not a bug. The point of the article is apparently to waffle aimlessly while namedropping a lot of things the reader likely hasn't heard of. The main selling point is novelty, which paints the author and their audience as being particularly in-the-know. It lets them feel they are sophisticated because of the foods they cook and eat, as well as the people they know and the businesses they frequent. If you're not in this loop, you're supposed to feel unsophisticated and behind the times.

It's no coincidence that this is published in the New York Times. New Yorkers have a well-earned reputation for being oblivious about life outside their bubble: the city offers the sense that you can have access to anything, but its attention is almost always turned inwards. It's not hard to imagine why, given the astronomical cost of living: surely it must be worth it! And yes, I have in fact spent a fair amount of time there, working. It couldn't just be that life elsewhere is cheaper, safer, cleaner and friendlier. That you can reach an airport in less than 2 hours during rush hour. On a comfortable, modern train. Which doesn't look and smell like an ashtray that hasn't been emptied out since 1975.

But I digress.

"Ethnic aisles are meaningless because everyone has an ethnicity" is revealed to be a meaningless thought. It smacks headfirst into the reality of the food business, which is a lesson the article seems determined not to learn. When "diversity" turns out to mean that people are actually diverse, have different needs and wants, and don't all share the same point of view, they just think diversity is wrong, or at least, outmoded, a "necessary evil." Even if they have no real basis of comparison.

graffiti near school in New York

Negative Progress

I think both stories capture an underlying social affliction, which is about progress and progressivism.

The basic premise of progressivism is seemingly one of optimism: we aim to make the future better than today. But the way it often works is by painting the present as fundamentally flawed, and the past as irredeemable. The purpose of adopting progressive beliefs is then to escape these flaws yourself, at least temporarily. You make them other people's fault by calling for change, even demanding it.

What is particularly noticeable is that perceived infractions are often in defense of people who aren't actually present at all. The person making the complaint doesn't suffer any particular injury or slight, but others might, and this is enough to condemn in the name of progress. "If an [X] person saw that, they'd be upset, so how dare you?" In the story of "broken English," the original message doesn't actually refer to a specific person or incident. It's just a general thing we are supposed to collectively do. That the follow-up completely contradicts the premise, well, that apparently doesn't matter. In the case of the ethnic aisle, the contradictory evidence is only reluctantly acknowledged, and you get the impression they had hoped to write a very different story.

This too is a provincial belief masquerading as sophistication. It mashes together groups of people as if they all share the exact same beliefs, hang-ups and sensitivities. Even if individuals are all saying different things, there is an assumed archetype that overrules it all, and tells you what people really think and feel, or should feel.

To do this, you have to see entire groups as an "other," as people that are fundamentally less diverse, self-aware and curious than the group you're in. That they need you to stand up for them, that they can't do it themselves. It means that "inclusion" is often not about including other groups, but about dividing your own group, so you can exclude people from it. The "diversity" it seeks reeks of blandness and commodification.

In the short term it's a zero-sum game of mining status out of each other, but in the long run everyone loses, because it lets the most unimaginative, unworldly people set the agenda. The sense of sophistication that comes out of this is imaginary: it relies on imagining fault where there is none, and playing meaningless word games. It's not about what you say, but how you say it, and the rules change constantly. Better keep up.

Usually this is associated with a profound ignorance about the actual past. This too is a status-mining move, only against people who are long gone and can't defend themselves. Given how much harsher life was, with deadly diseases, war and famine regular occurences, our ancestors had to be far smarter, stronger and self-sufficient, just to survive. They weren't less sophisticated, they came up with all the sophisticated things in the first place.

When it comes to the more recent past, you get the impression many people still think 1970 was 30, not 51 years ago. The idea that everyone was irredeemably sexist, racist and homophobic barely X years ago just doesn't hold up. Real friendships and relationships have always been able to transcend larger social matters. Vice versa, the idea that one day, everyone will be completely tolerant flies in the face of evidence and human nature. Especially the people who loudly say how tolerant they are: there are plenty of skeletons in those closets, you can be sure of that.

* * *

There's a Dutch expression that applies here: claiming to have invented hot water. To American readers, I gotta tell you: it really isn't hard to figure out that America is a society stratified by race, or exactly how. I figured that out the first time I visited in 2001. I hadn't even left the airport in Philadelphia when it occurred to me that every janitor I had seen was both black and morbidly obese. Completely unrelated, McDonald's was selling $1 cheeseburgers.

Later in the day, a black security guard had trouble reading an old-timey handwritten European passport. Is cursive racist? Or is American literacy abysmal because of fundamental problems in how school funding is tied to property taxes? You know this isn't a thing elsewhere, right?

In the 20 years since then, nothing substantial has improved on this front. Quite the opposite: many American schools and universities have abandoned their mission of teaching, in favor of pushing a particular worldview on their students, which leaves them ill-equipped to deal with the real world.

Ironically this has created a wave of actual American colonialism, transplanting the ideology of intersectionality onto other Western countries where it doesn't apply. Each country has their own long history of ethnic strife, with entirely different categories. The aristocrats who ruled my ancestors didn't even let them get educated in our own language. That was a right people had to fight for in the late 1960s. You want to tell me which words I should capitalize and which I shouldn't? Take a hike.

Not a year ago, someone trying to receive health care here in Dutch was called racist for it, by a French speaker. It should be obvious the person who did so was 100% projecting. I suspect insecurity: Dutch speakers are commonly multi-lingual, but French speakers are not. When you are surrounded by people who can speak your language, when you don't speak a word of theirs, the moron is you, but the ego likes to say otherwise. So you pretend yours is the sophisticated side.

All it takes to pierce this bubble is to actually put the platitudes and principles to the test. No wonder people are so terrified.

September 09, 2021

It's been long overdue, but Planet Grep now does the https dance (i.e., if you try to use an unencrypted connection, it will redirect you to https). Thank you letsencrypt!

I hadn't previously done this because some blogs that we carry might link to http-only images; but really, that shouldn't matter, and we can make Planet Grep itself be a https site even if some of the content is http-only.

Enjoy!

September 08, 2021

Cette interdépendance que l’on essaie d’oublier afin de camoufler l’apport essentiel de l’oisiveté et de la réflexion ouverte.

En 2014, alors que je parlais beaucoup du prix libre, j’ai reçu un gros paiement d’un lecteur. Ce lecteur me remerciait, car les idées que je décrivais l’inspiraient pour son projet de site de jeu d’échecs en ligne. 6 années plus tard, un de mes étudiants a choisi, comme logiciel libre à présenter pour son examen, ce logiciel : Lichess. Il m’a décrit le modèle libre de développement de Lichess, la méthode de don et le prix libre. Lichess est l’un des plus importants sites d’échecs dans le monde et est fréquenté par des grands maitres comme Magnus Carlsen.

Outre une immense fierté de savoir que certaines des graines que j’ai semées ont contribué à de magnifiques forêts, cette anecdote illustre surtout un point très important que l’idéologie Randienne tente à tout prix de camoufler : le succès n’est pas la propriété d’un individu. Un individu n’est jamais productif tout seul, il ne peut pas « se faire tout seul » en dépit de l’image que l’on aime donner des milliardaires. Si les parents de Jeff Bezos ne lui avaient pas donné 300.000$ en lui faisant promettre de trouver un vrai travail une fois les 300.000$ dépensés, il n’y aurait pas d’Amazon aujourd’hui. Chacun d’entre nous utilise des routes, des moyens de communication, des hôpitaux, des écoles et a des échanges intellectuels fournis par la communauté. L’idéologie de la propriété intellectuelle et des brevets nous fait croire qu’il y’a un unique inventeur, un génie solitaire qui mérite de récolter le fruit de ses efforts. C’est entièrement complètement faux. Nous sommes dépendants les uns des autres et nos succès sont essentiellement des chances, saisies ou non, que nous offre la communauté.

De plus, les brevets sont une gigantesque arnaque intellectuelle. J’en ai fait l’expérience moi-même dans un article assez ancien qui a eu pas mal de retentissement sans jamais rencontrer de contradiction.

https://ploum.net/working-with-patents/

Brevets qui ne servent d’ailleurs que l’intérêt des riches et puissants. Amazon, par exemple, a développé une technique pour repérer ce qui se vend bien sur son site afin de le copier et d’en faire sa propre version. Même s’il y’a des brevets. Parce que personne n’a les ressources d’attaquer Amazon sur une histoire de brevets.

https://www.currentaffairs.org/2020/12/how-amazon-destroys-the-intellectual-justifications-for-capitalism

Les brevets sont une arnaque construite sur un concept entièrement fictif : celui de l’inventeur solitaire. Une fiction qui nie l’idée même de l’interdépendance sociale.

Une interdépendance sociale dont l’apport essentiel à la productivité individuelle a été illustré par un généticien, William Muir, qui a décidé de sélectionner les poules qui pondaient le plus d’œufs afin de créer un « super poulailler » qui serait hyper productif. Le résultat a été catastrophique. Les poules qui pondaient le plus d’œufs au sein d’un poulailler étaient en fait les plus agressives qui empêchaient les autres de pondre. Le super poulailler est devenu une boucherie d’ou presque aucun œuf ne sortait et dont la majorité des poules mourraient !

La conclusion est simple : même les poules qui pondent peu ont un rôle essentiel dans la productivité globale de la communauté. Le meilleur poulailler n’est pas composé des meilleures pondeuses, bien au contraire.

https://economicsfromthetopdown.com/2021/01/14/the-rise-of-human-capital-theory/

Grâce aux témoignages de mes lecteurs, je peux affirmer que mes billets de blog ont une influence sur la société à laquelle j’appartiens. Influence que j’estime essentiellement positive, voire très positive, selon mes propres critères. Lichess en est un exemple spectaculaire, mais je reçois des mails beaucoup plus intimes qui vont dans le même sens et qui me touchent beaucoup (même si j’ai pris la décision de ne plus y répondre systématiquement). Je peux donc affirmer que je suis utile à mon humble échelle.

Au cours de ma carrière, je ne peux trouver aucun exemple où mon travail salarié ait jamais eu le moindre impact et où mon utilité a été démontrée. Pire : je ne vois pas un seul impact positif des entreprises entières pour lesquelles j’ai travaillé. En étant très optimiste, je peux affirmer qu’on a amélioré la rentabilité de certains de nos clients. Mais ce n’est pas vraiment un impact sociétal positif. Et ce rendement est de toute façon noyé dans une gabegie de projets abscons et de procédures administratives. Pendant dix ans, j’ai été payé dans des super-poulaillers, dans des entreprises qui sont elles-mêmes en compétition. Pour un résultat soit nul, soit nocif pour l’humanité et la planète car augmentant la consommation globale.

À l’opposé, je vois directement l’impact des projets auxquels j’ai contribué sans rétribution, notamment les projets de logiciels libres. Le développeur Mike Williamson est arrivé à la même conclusion.

https://mike.zwobble.org/2021/08/side-projects-vs-industry/

Si vous cherchez mon nom sur Wikipedia, vous arriverez sur la page d’un projet auquel j’ai consacré plusieurs années de sommeil sans toucher le moindre centime.

https://fr.wikipedia.org/wiki/Getting_Things_Gnome

Revenu de base

C’est peut-être pour ça que le revenu de base me semble tellement essentiel. En 2013, je tentais de vous convaincre que le revenu de base était une bonne idée et de signer la pétition pour forcer les instances européennes d’étudier la question. Hélas, le nombre de signatures n’avait pas été atteint.

https://ploum.net/pourquoi-vous-etes-sans-le-savoir-favorable-au-revenu-de-base/

Huit ans plus tard, une nouvelle pétition vient de voir le jour. Si vous êtes citoyen européen, je vous invite vivement à la signer. C’est très facile et très officiel. Il faut mettre vos données personnelles, mais pas votre email. Il est nécessaire d’obtenir un minimum de signatures dans tous les pays d’Europe. N’hésitez pas à partager avec vos contacts internationaux.

https://eci.ec.europa.eu/014/public/#/screen/home

Les observables

Lorsqu’on vous parle de la productivité d’un individu ou du mérite des personnes riches, rappelez-vous l’histoire des poulaillers.

Mais pour les poules, c’est facile. Il suffit de mesurer les œufs pondus. Le problème avec le capitalisme moderne, c’est qu’on se plante tout le temps dans les métriques. Or, si on utilise une mauvaise métrique, on va optimiser tout le système pour avoir des mauvais résultats.

J’ai beaucoup glosé sur ce paradigme des métriques, que j’appelle des « observables ». Je tourne en rond autour du même thème : on mesure la productivité à l’aide des heures de travail (vu que le salarié moyen ne pond pas), donc on crée des heures de travail, donc les jobs servent à remplir le plus d’heures possible. Ce que j’appelle le principe d’inefficacité maximale. Au final, on passe 8h par jour à tenter de brûler la planète afin, une fois sorti du bureau, de pouvoir se payer des légumes bio en ayant l’impression de sauver la même planète.

https://ploum.net/le-principe-dinefficacite-maximale/

Outre les heures de travail, il y’a d’autres métriques absurdes comme les clics, les pages vues et ce genre de choses. Les métriques des gens qui font du marketing : faire le plus de bruit possible ! Le département marketing, c’est un peu un super-poulailler où on a mis tous les coqs les plus bruyants. Et on s’étonne de ne pas avoir un seul œuf. Mais beaucoup de bruit.

https://ploum.net/le-silence-au-milieu-du-bruit/

L’effet des métriques absurdes a un impact direct sur votre vie. Genre si vous utilisez Microsoft Team au travail. Car désormais, votre manager va pouvoir avoir des statistiques sur votre utilisation de Teams. Le programmeur hyper concentré qui a coupé Teams pour coder une super fonctionnalité va bien vite se faire virer à cause de mauvaises statistiques. Et votre vie privée ? Elle ne rentre pas dans les plans du superpoulailler !

https://www.zdnet.com/article/i-looked-at-all-the-ways-microsoft-teams-tracks-users-and-my-head-is-spinning/

Comme plus personne n’a le temps de réfléchir (vu qu’il n’y a pas de métriques sur le sujet et qu’au contraire réfléchir bousille d’autres métriques), l’avenir appartient à ceux qui arrivent à maximiser les métriques. Ou mieux : qui arrive à faire croire qu’ils sont responsables de métriques maximisées. Changer de travail régulièrement permet de ne jamais vraiment exposer son incompétence et de montrer en grade à chaque étape, augmentant ainsi son salaire jusqu’à devenir grand manager hyper bien payé dans un univers où les métriques sont de plus en plus floues. La compétence est remplacée par l’apparence de compétence, qui est essentiellement de la confiance en soi et de l’opportunisme politique. Cela rejoint un peu la thèse de Daniel Drezner développée dans « The Ideas Industry » : les idées simples, prémâchées, faciles à s’approprier (genre TED) prennent le pas sur les analyses profondes et plus subtiles. C’est également un constat fait par Cal Newport dans « A World Without Email » où il dénonce la mentalité de « ruche bourdonnante » de toute entreprise moderne.

Vous êtes entrepreneur ou indépendant ? C’est pareil : vous maximisez les métriques absurdes de vos clients. Si vous avez de la chance d’avoir des clients ! Sinon, vous passez votre temps à optimiser les métriques que vous offrent Facebook, Google Analytics ou Amazon en ayant l’impression de bosser à votre projet. Y’a même un métier entier qui ne fait qu’optimiser une métrique offerte par Google : le SEO.

Il y a quelques années, le simple fait d’avoir émis cette idée m’a valu que des professionnels du secteur s’organisent pour qu’une recherche à mon nom renvoie vers des injures de leur cru. Cette anecdote illustre bien le problème des métriques absurdes : il est impossible de faire comprendre qu’une métrique est absurde à ceux qui payent pour optimiser cette métrique et à ceux qui ont bâti leur carrière sur la même métrique. Une simple remise en question génère une violence complètement disproportionnée, religieuse.

Religion et violence

Le repli identitaire, la religiosité ou la plupart des opinions conservatrices sont générés par l’angoisse et le sentiment de ne pas comprendre. Ce n’est pas une analyse politique, mais bien neurologique. Il suffit de désactiver quelques neurones dans le cerveau pour que, soudainement, l’angoisse ne soit plus liée à ce repli. Comme on ne peut pas désactiver ces neurones chez tout le monde, il reste une solution qui a déjà fait ses preuves : l’éducation, qui permet de comprendre et d’être moins angoissé.

https://www.lemonde.fr/passeurdesciences/article/2015/10/21/moins-croire-en-dieu-avec-la-stimulation-magnetique_6001729_5470970.html

La religion n’est de toute façon qu’un prétexte. Ce ne sont pas les interprétations religieuses qui sont la cause de violences ou de repli, elles en sont au contraire le symptôme, l’excuse.

https://medium.com/incerto/religion-violence-tolerance-progress-nothing-to-do-with-theology-a31f351c729e

Le poulailler sans-tête !

En utilisant religieusement les mauvaises métriques, nous sommes en train de faire de la planète une sorte de super-poulailler où la bêtise et la stupidité sont optimisées. C’est d’ailleurs la définition même de la foi : croire sans poser de question, sans chercher à comprendre. La foi est la bêtise élevée au rang de qualité. L’invasion du capitole par les partisans de Trump en a été l’illustration suprême : des gens pas très malins, ayant la foi que l’un d’entre eux avait un plan et qu’ils allaient le suivre. Sauf qu’il n’y avait pas de plan, que cette invasion était un « meme » comme l’est Q : une simple idée lancée sur les réseaux sociaux qui s’est créé une auto-importance grâce à la rumeur et au bouche-à-oreille virtuel. D’ailleurs, une fois dans le capitole, personne ne savait quoi faire. Ils se sont assis sur les fauteuils pour se sentir importants, ont pris des selfies, ont tenté de trouver des complots croustillants, en quelques secondes, dans les centaines de pages de documents législatifs qui sont probablement disponibles sur le site du gouvernement. Quand votre culture politique est alimentée essentiellement par des séries d’actions sur Netflix, la révolution trouve vite ses limites.

Comme le souligne très bien Cory Doctorrow, les memes et les fake news ne sont pas la réalité, mais ils sont l’expression d’un fantasme. Les memes sur Internet ne sont pas créés pour décrire la réalité, mais pour tenter de faire plier la réalité à nos désirs.

https://locusmag.com/2019/07/cory-doctorow-fake-news-is-an-oracle/

Mais pas besoin d’aller aussi loin. Bien avant Trump, la Belgique avait connu le concept du « politicien-meme » avec le député Laurent Louis. Député tellement absurde que j’avais ironisé sur le fait qu’il n’était qu’une blague à travers un article satirique. Article qui avait d’ailleurs eu pour résultat que Laurent Louis lui-même avait posté son certificat de naissance sur les réseaux sociaux, pour prouver qu’il existait. Cette non-perception de l’ironie m’avait particulièrement frappé.

Comme Trump, Laurent Louis avait fini par trouver un créneau et des partisans. Assez pour foutre un peu le bordel, pas assez pour ne pas disparaitre dans l’oubli comme une parenthèse illustrant les faiblesses d’un système politique bien trop optimisé pour récompenser le marketing et la bêtise. Mais je tombe dans le pléonasme.

https://ploum.net/le-depute-qui-nexistait-pas/

S’évader du poulailler

J’achète un recueil de nouvelles de Valery Bonneau. Je le prête à ma mère avant même de le lire. Elle me dit de lire absolument la première nouvelle,  » Putain de cafetière « . Je me plonge. Je tombe de mon fauteuil de rire. Franchement, le coup du frigo américain avec un code PIN, j’en rigole encore.

Profitez-en ! (en version papier, c’est encore plus délectable !)

https://www.valerybonneau.com/romans/nouvelles-noires-pour-se-rire-du-desespoir/putain-de-cafetiere

Envie d’un roman gonflé à la vitamine ? Besoin de vous évader des confinements et couvre-feux à gogo ? Printeurs de Ploum est fait pour vous!

Ce n’est pas moi qui le dis, c’est une critique que je ne me lasse pas de relire :

https://albdoblog.com/2021/01/20/printeurs-ploum/

D’ailleurs, si vous avez lu Printeurs, n’hésitez pas à donner votre avis sur Senscritique et Babelio. Je déteste Senscritique, mais je n’ai pas encore trouvé d’alternative durable.

https://www.senscritique.com/livre/Printeurs/43808921

https://www.babelio.com/livres/Dricot-Printeurs-Science-fiction/1279338

Un autre plugin Firefox qui me sauve la vie et pour lequel j’ai souscrit un abonnement premium à prix libre :


https://ninja-cookie.com/

Fini de paramétrer les cookies. Le plugin les refuse automatiquement au maximum refus possible. C’est parfait et indispensable.

Ça en dit long sur l’état du web actuel. Quand on voit le nombre de protections qu’il faut avoir pour pouvoir tout simplement « lire » le contenu des pages web sans avoir le cerveau qui frit et sans être espionné de tous les côtés, on comprend mieux l’intérêt d’un protocole comme Gemini qui est conçu à la base pour être le moins extensible possible !

Conseil BD

Après les magnifiques « L’Autre Monde » et « Mary la Noire », je découvre une nouvelle facette de l’univers de Florence Magnin . « L’héritage d’Émilie ».

J’ai découvert Magnin par hasard, dans ma librairie préférée. L’Autre Monde m’a interpellé. Le dessin était magnifique, mais d’une naïveté particulière. Je n’étais pas certain d’aimer. Je n’ai pas aimé, j’ai littéralement été aspiré. Ce mélange de naïveté et d’univers pour adulte, de fantastique à la fois désuet et incroyablement moderne. L’héritage d’Émilie ne fait pas exception. En fait, il transcende même les deux autres en mélangeant le Paris des années folles et les légendes celtiques d’Irlande, le tout dans une œuvre de fantastique champêtre qui glisse brusquement dans le space opera intergalactique. Oui, c’est complètement incroyable. Et oui, j’adore.

Photo by Artem Beliaikin on Unsplash

Si la relation entre les humains et la technologie vous intéresse, abonnez-vous par mail ou par RSS pour ne rater aucun billet (max 2 par semaine). Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, offrez et partagez ses livres.

Ce texte est publié sous la licence CC-By BE.

I'm excited to announce that Acquia has signed a definitive agreement to acquire Widen, a digital asset management (DAM) and product information management (PIM) company.

The Acquia and Widen logos shown next to each other

It's not hard to understand how Widen fits Acquia's strategy. Our goal is to build the best Digital Experience Platform (DXP). Content is at the heart of any digital experience. By adding a DAM and PIM to our platform, our customers will be able to create better content, more easily. That will result in better customer experiences. Plain and simple.

Widen is for organizations with larger marketing teams managing one or more brands. These teams create thousands of "digital assets": images, videos, PDFs and much more. Those digital assets are used on websites, mobile applications, in-store displays, presentations, etc. Managing thousands of files, plus all the workflows to support them, is difficult without the help of a DAM.

For commerce purposes, marketers need to correlate product images with product information like pricing, sizing, or product specifications. To do so, Widen offers a PIM. Widen built their PIM on top of their DAM — an approach that is both clever and unique. Widen's PIM can ingest product content from proprietary systems, master data management (MDM) platforms, data lakes, and more. From there, marketers can aggregate, synthesize, and syndicate product content across digital channels.

In short, organizations need a lot of content to do business. And online commerce can't exist without product information. It's why we are so excited about Widen, and the ability to add a DAM and PIM to our product portfolio.

Because content is at the heart of any digital experience, we will build deep integrations between Widen and Acquia's DXP. So in addition to acquiring Widen, we are making a large investment in growing Widen's engineering team. That investment will go towards extending the existing Widen module for Drupal, and creating integrations with Acquia's products: Acquia Site Studio, Acquia Campaign Studio (Mautic), Acquia Personalization, and more. Digital asset management will be a core building block of our DXP.

Needless to say, we will continue to support and invest in Widen working with other Content Management Systems and Digital Experience Platforms. We are building an open DXP; one of our key principles is that customers should be able to integrate with their preferred technologies, and that might not always be ours. By growing the engineering team, we can focus on building Drupal and Acquia integrations without disrupting the existing roadmap and customer commitments.

A few other facts that might be of interest:

So last but not least, I'd like to welcome all of Widen's customers and employees to the Acquia family. I'm excited to see what we will accomplish together.

September 07, 2021

In this last post on the infrastructure domain, I cover the fifth and final viewpoint that is important for an infrastructure domain representation, and that is the location view. As mentioned in previous posts, the viewpoints I think are most representative of the infrastructure domain are:

Like with the component view, the location view is a layered approach. While I initially wanted to call it the network view, "location" might be a broader term that matches the content better. Still, it's not a perfect name, but the name is less important than the content, not?

September 04, 2021

I had been a happy user of the Nokia 6.1 I bought 3 and a half years ago, but with battery life slowly going down and both OS major updates and security-updates having stopped it was time to find a replacement.

Although the tech reporters and vloggers were underwhelmed by the screen (no Oled or Amoled, only 60Hz refresh rate) and CPU (the SM4350 Snapdragon 480 is considered too slow), I choose the Nokia X20 because:

  • bare-bones Android One experience
  • 3 years OS & security updates
  • 3 years warranty
  • 128GB memory & 8GB RAM
  • 5G

And you know what? This old man has no issues whatsoever with the screen or CPU. The only downside: the eco-friendly backcover is pretty ugly. But all in all pretty good hardware for a very reasonable price (€339 incl. VAT), so if all is well I won’t bother you with boring smartphone-news until 2024? :-)

September 02, 2021

I published the following diary on isc.sans.edu: “Attackers Will Always Abuse Major Events in our Lifes“:

All major events in our daily life are potential sources of revenue for attackers. When elections or major sports events are organized, attackers will surf on these waves and try to make some profit or collect interesting data (credentials). It’s the same with major meteorological phenomena. The hurricane “Ida” was the second most intense hurricane to hit the state of Louisiana on record, only behind “Katrina”… [Read more]

The post [SANS ISC] Attackers Will Always Abuse Major Events in our Lifes appeared first on /dev/random.

September 01, 2021

Blogging sometimes feels like talking to an imaginary friend. It's an interesting comparison because it could help me write more regularly. For example: I can picture myself going to dinner with my imaginary friend. Once we sit down, what would we talk about? What would I share?

I'd share that I've been doing well the past year.

Work is going well. I'm fortunate to help lead at a growing software company. We continue to hit record sales quarter after quarter, and hired more than 250 new employees in 2021 alone. Keeping up with all the work can be challenging but I continue to have fun and learn a lot, which is the most important part.

Most days I work from home. Working from home consists of 8 hours of Zoom meetings, followed by email, presentation and planning work. I finish most work days energized and drained at the same time.

Over the course of two years, I've created a home office setup that is more comfortable, more ergonomic, and more productive than my desk at the office. I invested in an ergonomic chair, standing desk, camera setup, a second screen, and even a third screen. Possibly an interesting topic for a future blog post.

Despite having a great home office setup, I'd like to work more from interesting locations. I'm writing this blog post from an island on Lake Winnipesaukee in New Hampshire where we have a management offsite. Working from an island is as awesome as it sounds. The new hybrid work arrangement provides that extra flexibility.

A chair with a view of Lake Winnipesaukee
Overlooking Lake Winnipesaukee in New Hampshire. Coffee and laptop for morning blogging.

When not working, I've been enjoying the summer in Boston. We moved from the suburbs to the city this year, and have been busy exploring our new neighborhood. We love it!

I've been very happy with our decision to move to the city, except for one thing: tennis. I love playing tennis with a coach, and that has been nearly impossible in the city. As a result I haven't played tennis for months — the lack of workout routine has been really bothering me. Because I love racket sports the most, I started to explore if there are good squash, pickleball or table tennis options in downtown Boston. Recommendations welcome!

Last but not least, we spent some time at Cape Cod this summer, and traveled to Iceland for a weekend. I'll tie off this blog post with a few photos of those trips.

An American flag waving in the light of the moon
A red moon over the water in Cape Cod.
Eating dinner outside overlooking the ocean
Dinner at Cape Cod.
A aarshmallow over a camp fire
S'mores on the beach.
Three people walking alongside dries lava
Hiking alongside the lava from the Gerlingadalur volcano in Iceland. The volcano was active, hence the smoke coming from the lava.

In my previous post, I started with the five different views that would support a good view of what infrastructure would be. I believe these views (component, location, process, service, and zoning) cover the breadth of the domain. The post also described the component view a bit more and linked to previous posts I made (one for services, another for zoning).

The one I want to tackle here is the most elaborate one, also the most enterprise-ish, and one that always is a balance on how much time and effort to put into it (as an architect), as well as hoping that the processes are sufficiently standardized in a flexible manner so that you don't need to cover everything again and again in each project.

So, let's talk about processes...

August 30, 2021

Alors que je déclipsais le pied de mes pédales après ma grande traversée du Massif central en VTT en compagnie de Thierry Crouzet, mon téléphone m’afficha un mail au titre à la fois évident et incompréhensible, inimaginable : « Roudou nous a quittés ».

Avec Internet est apparu une nouvelle forme de relation sociale, une nouvelle forme d’interaction voire, j’ose le terme, d’amitié. Une amitié envers des personnes avec qui on se découvre des affinités intellectuelles, mais qu’on ne verra pas souvent voire jamais. Une amitié tout de même. Une amitié qui peut mener sur une complicité, sur la création de projets communs. Une amitié qui dépasse bien des relations en chair et en os que la proximité nous impose quotidiennement.

Jean-Marc Delforge, Roudou pour les intimes, était pour moi de ces amitiés au long cours. Lecteur de mon blog depuis des années, utilisateur de logiciel libre et illustrateur amateur, il m’a envoyé le tout premier fan-art de Printeur et signera ensuite la couverture du premier EPUB Printeurs.

À force de discussions, nous créerons ensemble le webcomic « Les startupeurs » dont j’ai empilé les scénarios avant que, malheureusement, Roudou ne trouve plus le temps pour les dessiner. Des personnages d’employés un peu désabusés (dont l’un est ma parodie selon Roudou), rêvant de créer leurs startup et addicts de la machine à café (une trouvaille de Roudou !).

https://ploum.net/les-startupeurs-un-nouveau-webcomic/

On s’amusait comme des fous avec ces idées, s’essayant au cartoon politique, partageant, discutant et se découvrant une passion commune pour le VTT.

Car Roudou était plus qu’un passionné de VTT. C’était un meneur, un créateur de trace et le fondateur du forum VTTnet. Dans son sillage, impossible de ne pas pédaler.

En 2015, il m’invita à le rejoindre avec mon filleul Loïc pour 3 jours de VTT intensifs en compagnie des membres du forum.

Roudou, sa fille Noémie, mon filleul Loïc et les autres malades de VTTNet en 2015

Par le plus grand des hasards, Loïc et moi sommes repassés dans la région début juillet pour un trip bikepacking. Lorsque Roudou a découvert cela, il m’a immédiatement envoyé un message pour me dire qu’on s’était raté de peu. Alors que Loïc et moi nous prélassions au bord du lac de l’Eau d’Heure, lui était probablement en train d’y faire du bateau. Il rigolait en lisant l’itinéraire que nous avions pris, me disant qu’il aurait pu nous guider, qu’il habitait tout près.

Je me suis senti triste à l’idée d’avoir manqué une telle opportunité de pédaler ensemble. J’ai promis qu’on referait le trip l’année prochaine. Que ce serait vraiment chouette de se retrouver sur un vélo (même si, pour des raisons de santé qu’il ne voulait pas détailler, le VTT de Roudou était devenu électrique).

À un message un peu accusateur me demandant comment j’osais venir pédaler dans sa région sans le prévenir, je répondis que j’étais persuadé qu’il habitait bien plus à l’ouest.

La réponse de Roudou ne se fit pas attendre : « Ma femme aussi me dit souvent que je suis bien trop à l’ouest. »

Ce fut le dernier message que je reçus de lui. Le 16 juillet, j’embarquais pour 1000km de VTT essentiellement déconnectés, me promettant d’aller rouler avec Roudou l’été prochain.

Mais alors que je pédalais loin de tout, la mort l’a surpris, interrompant à jamais notre fil de discussion, plongeant les startupeurs, les vététistes, sa femme, ses filles et ses amis dans une tristesse infinie.

Roudou va me manquer. Ses crobards et ses photos humoristiques envoyés pour réagir à mes billets de blog et mes livres vont me manquer. Les startupeurs, même s’ils étaient en hibernation, vont me manquer (je n’ai d’ailleurs pas de copie de cette œuvre commune, peut-être perdue). Lorsque je me plongerai dans la suite de Printeurs, je sais que les personnages auront une pensée pour Roudou, ce lecteur qui leur faisait prendre corps sous sa tablette graphique.

Je garderai toujours en moi ce regret d’avoir oublié de le prévenir, d’avoir gâché cette dernière opportunité avant qu’il parte pédaler un peu plus à l’Ouest. Un peu trop à l’Ouest…

Salut l’artiste, salut Roudou ! Nous continuerons à suivre tes traces en pensant à toi.

Si la relation entre les humains et la technologie vous intéresse, abonnez-vous par mail ou par RSS pour ne rater aucun billet (max 2 par semaine). Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, offrez et partagez ses livres.

Ce texte est publié sous la licence CC-By BE.

I published the following diary on isc.sans.edu: “Cryptocurrency Clipboard Swapper Delivered With Love“:

Be careful if you’re a user of cryptocurrencies. My goal is not to re-open a debate about them and their associated financial risks. No, I’m talking here about technical risk. Wallet addresses are long strings of characters that are pretty impossible to use manually. It means that you’ll use your clipboard to copy/paste your wallets to perform payments. But some malware monitors your clipboard for “interesting data” (like wallet addresses) and tries to replace it with another one. If you perform a payment operation, it means that you will transfer some BTC or XMR to the wrong wallet, owned by the attacker… [Read more]

The post [SANS ISC] Cryptocurrency Clipboard Swapper Delivered With Love appeared first on /dev/random.

August 28, 2021

Cover Image

Making code reusable is not an art, it's a job

Extensibility of software is a weird phenomenon, very poorly understood in the software industry. This might seem strange to say, as you are reading this in a web browser, on an operating system, desktop or mobile. They are by all accounts, quite extensible and built out of reusable, shared components, right?

But all these areas are suffering enormous stagnation. Microsoft threw in the towel on browsers. Mozilla fired its engineers. Operating systems have largely calcified around a decades-old feature set, and are just putting up fortifications. The last big shift here was Apple's version of the mobile web and app store, which ended browser plug-ins like Flash or Java in one stroke.

Most users are now silo'd inside an officially approved feature set. Except for Linux, which is still figuring out how audio should work. To be fair, so is the web. There's WebAssembly on the horizon, but the main thing it will have access to is a creaky DOM and an assortment of poorly conceived I/O APIs.

It sure seems like the plan was to have software work much like interchangeable legos. Only it didn't happen at all, not as far as end-users are concerned. Worse, the HTTP-ificiation of everything has largely killed off the cow paths we used to have. Data sits locked behind proprietary APIs. Interchange doesn't really happen unless there is a business case for it. The default of open has been replaced with a default of closed.

This death of widespread extensibility ought to seem profoundly weird, or at least, ungrappled with.

We used to collect file types like Pokémon. What happened? If you dig into this, you work your way through types, but then things quickly get existential: how can a piece of code do anything useful with data it does not understand? And if two programs can interpret and process the same data the same way, aren't they just the same code written twice?

Most importantly: does this actually tell us anything useful about how to design software?

Mallard ducks
The Birds of America, John James Audubon (1827)

Well?

Let's start with a simpler question.

If I want a system to be extensible, I want to replace a part with something more specialized, more suitable to my needs. This should happen via the substitution principle: if it looks like a duck, walks like a duck and quacks like a duck, it's a duck, no matter which kind. You can have any number of sub-species of ducks, and they can do things together, including making weird new little ducks.

So, consider:

If I have a valid piece of code that uses the type Animal, I should be able to replace Animal with the subtype Duck, Pig or Cow and still have valid code.

True or False? I suspect your answer will depend on whether you've mainly written in an object-oriented or functional style. It may seem entirely obvious, or not at all.

This analogy by farm is the usual intro to inheritance: Animal is the supertype. When we call .say(), the duck quacks, but the cow moos. The details are abstracted away and encapsulated. Easy. We teach inheritance and interfaces this way to novices, because knowing what sounds your objects make is very important in day-to-day coding.

But, seriously, this obscures a pretty important distinction. Understanding it is crucial to making extensible software. Because the statement is False.

So, the farmer goes to feed the animals:

type GetAnimal  = () => Animal;
type FeedAnimal = (animal: Animal) => void;

How does substitution apply here? Well, it's fine to get ducks when you were expecting animals. Because anything you can do to an Animal should also work on a Duck. So the function () => Duck can stand-in for an () => Animal.

But what about the actions? If I want to feed the ducks breadcrumbs, I might use a function feedBread which is a Duck => void. But I can't feed that same bread to the cat and I cannot pass feedBread to the farmer who expects an Animal => void. He might try to call it on the wrong Animal.

This means the allowable substitution here is reversed depending on use:

  • A function that provides a Duck also provides an Animal.
  • A function that needs an Animal will also accept a Duck.

But it doesn't work in the other direction. It seems pretty obvious when you put it this way. In terms of types:

  • Any function () => Duck is a valid substitute for () => Animal.
  • Any function Animal => void is a valid substitute for Duck => void.

It's not about using a type T, it's about whether you are providing it or consuming it. The crucial distinction is whether it appears after or before the =>. This is why you can't always replace Animal with Duck in just any code.

This means that if you have a function of a type T => T, then T appears on both sides of =>, which means neither substitution is allowed. You cannot replace the function with an S => S made out of a subtype or supertype S, not in general. It would either fail on unexpected input, or produce unexpected output.

This shouldn't be remarkable at all among people who code in typed languages. It's only worth nothing because intros to OO inheritance don't teach you this, suggesting the answer is True. We use the awkward words covariant and contravariant to describe the two directions, and remembering which is which is hard.

I find this quite strange. How is it people only notice one at first?

let duck: Duck = new Duck();
let animal: Animal = duck;
class Duck extends Animal {
  method() {
    // ...
  } 
}

Here's one explanation. First, you can think of ordinary values as being returned from an implicit getter () => value. This is your default mental model, even if you never really thought about it.

Second, it's OO's fault. When you override a method in a subclass, you are replacing a function (this: Animal, ...) => with a function (this: Duck, ...) => . According to the rules of variance, this is not allowed, because it's supposed to be the other way around. To call it on an Animal, you must invoke animal.say() via dynamic dispatch, which the language has built-in.

Every non-static method of class T will have this: T as a hidden argument, so this constrains the kinds of substitutions you're allowed to describe using class methods. Because when both kinds of variance collide, you are pinned at one level of abstraction and detail, because there, T must be invariant.

This is very important for understanding extensibility, because the common way to say "neither co- nor contravariant" is actually just "vendor lock-in".

birds

The Mirage of Extensibility

The goal of extensibility is generally threefold:

  • Read from arbitrary sources of data
  • Perform arbitrary operations on that data
  • Write to arbitrary sinks of data

Consider something like ImageMagick or ffmpeg. It operates on a very concrete data type: one or more images (± audio). These can be loaded and saved in a variety of different formats. You can apply arbitrary filters as a processing pipeline, configurable from the command line. These tools are swiss army knives which seem to offer real extensibility.

type Input<T> = () => T;
type Process<T> = T => T;
type Output<T> = T => void;
functions of one or two Ts

Formally, you decode your input into some shared representation T. This forms the glue between your processing blocks. Then it can be sent back to any output to be encoded.

It's crucial here that Process has the same input and output type, as it enables composition of operations like lego. If it was Process<A, B> instead, you would only be able to chain certain combinations (A → B, B → C, C → D, ...). We want to have a closed, universal system where any valid T produces a new valid T.

Of course you can also define operators like (T, T) => T. This leads to a closed algebra, where every op always works on any two Ts. For the sake of brevity, operators are implied below. In practice, most blocks are also configurable, which means it's an options => T => T.

This seems perfectly extensible, and a textbook model for all sorts of real systems. But is it really? Reality says otherwise, because it's engineering, not science.

Consider a PNG: it's not just an image, it's a series of data blocks which describe an image, color information, physical size, and so on. To faithfully read a PNG and write it out again requires you to understand and process the file at this level. Therefor any composition of a PNGInput with a PNGOutput where T is just pixels is insufficient: it would throw away all the metadata, producing an incomplete file.

Now add in JPEG: same kind of data, very different compression. There are also multiple competing metadata formats (JFIF, EXIF, ...). So reading and writing a JPEG faithfully requires you to understand a whole new data layout, and store multiple kinds of new fields.

This means a swiss-army-knife's T is really some kind of wrapper in practice. It holds both data and metadata. The expectation is that operations on T will preserve that metadata, so it can be reattached to the output. But how do you do that in practice? Only the actual raw image data is compatible between PNG and JPEG, yet you must be able to input and output either.

meta = {
  png?: {...}
  jpeg?: {...}
}

If you just keep the original metadata in a struct like this, then a Process<T> interested in metadata has to be aware of all the possible image formats that can be read, and try them all. This means it's not really extensible: adding a new format means updating all the affected Process blocks. Otherwise Input<T> and Process<T> don't compose in a useful sense.

meta = {
  color: {...},
  physical: {...},
  geo: {...},
}

If you instead harmonize all the metadata into a single, unified schema, then this means new Input<T> and Output<T> blocks are limited to metadata that's already been anticipated. This is definitely not extensible, because you cannot support any new concepts faithfully.

If you rummage around inside ImageMagick you will in fact encounter this. PNG and JPEG's unique flags and quirks are natively supported.

meta = {
  color: {...},
  physical: {...},
  geo: {...},
  x-png?: {...}
  x-jpeg?: {...}
}

One solution is to do both. You declare a standard schema upfront, with common conventions that can be relied on by anyone. But you also provide the ability to extend it with custom data, so that specific pairs of Input/Process/Output can coordinate. HTTP and e-mail headers are X-able.

meta = {
  img?: {
    physical?: {...},
    color?: {...},
  },
  fmt?: {
    png?: {...},
    jfif?: {...},
    exif?: {...},
  },
},

The problem is that there is no universal reason why something should be standard or not. Standard is the common set of functionality "we" are aware of today. Non-standard is what's unanticipated. This is entirely haphazard. For example, instead of an x-jpeg, it's probably better to define an x-exif because Exif tags are themselves reusable things. But why stop there?

Mistakes stick and best practices change, so the only way to have a contingency plan in place is for it to already exist in the previous version. For example, through judicious use of granular, optional namespaces.

The purpose is to be able to make controlled changes later that won't mess with most people's stuff. Some breakage will still occur. The structure provides a shared convention for anticipated needs, paving the cow paths. Safe extension is the default, but if you do need to restructure, you have to pick a new namespace. Conversion is still an issue, but at least it is clearly legible and interpretable which parts of the schema are being used.

One of the smartest things you can do ahead of time is to not version your entire format as v1 or v2. Rather, remember the version for any namespace you're using, like a manifest. It allows you to define migration not as a transaction on an entire database or dataset at once, but rather as an idempotent process that can be interrupted and resumed. It also provides an opportunity to define a reverse migration that is practically reusable by other people.

This is how you do it if you plan ahead. So naturally this is not how most people do it.

birds

N+1 Standards

X- fields and headers are the norm and have a habit of becoming defacto standards. When they do, you find it's too late to clean it up into a new standard. People try anyway, like with X-Forwarded-For vs Forwarded. Or -webkit-transform vs transform. New software must continue to accept old input. It must also produce output compatible with old software. This means old software never needs to be updated, which means new software can never ditch its legacy code.

Let's look at this story through a typed lens.

What happens is, someone turns an Animal => Animal into a Duck => Duck without telling anyone else, by adding an X- field. This is fine, because Animal ignores unknown metadata, and X- fields default to none. Hence every Animal really is a valid Duck, even though Duck specializes Animal.

Slowly more people replace their Animal => Animal type with Duck => Duck. Which means ducks are becoming the new defacto Animal. But then someone decides it needed to be a Chicken => Chicken instead, and that chickens are the new Animal. Not everyone is on-board with that.

So you need to continue to support the old Duck and the new Chicken on the input side. You also need to output something that passes as both Duck and Chicken, that is, a ChickenDuck. Your signature becomes:

(Duck | Chicken | ChickenDuck) => ChickenDuck

This is not what you wanted at all, because it always lays two eggs at the same time, one with an X and one without. This is also a metaphor for IPv4 vs IPv6.

If you have one standard, and you make a new standard, now you have 3 standards: the old one, the theoretical new one, and the actual new one.

replacing one T=>T in a chain with S=>S

Invariance pops up again. When you have a system built around signature T => T, you cannot simply slot in a S => S of a super or sub S. Most Input and Output in the wild still only produces and consumes T. You have to slot in an actual T => S somewhere, and figure out what S => T means.

Furthermore, for this to do something useful in a real pipeline, T already has to be able to carry all the information that S needs. And S => T cannot strip it out again. The key is to circumvent invariance: neither type is really a subtype or supertype of the other. They are just different views and interpretations of the same underlying data, which must already be extensible enough.

Backwards compatibility is then the art of changing a process Animal => Animal into a Duck => Duck while avoiding a debate about what specifically constitutes quacking. If you remove an X- prefix to make it "standard", this stops being true. The moment you have two different sources of the same information, now you have to decide whose job it is to resolve two into one, and one back into two.

This is particularly sensitive for X-Forwarded-For, because it literally means "the network metadata is wrong, the correct origin IP is ..." This must come from a trusted source like a reverse proxy. It's the last place you want to create compatibility concerns.

sideways birds

If you think about it, this means you can never be sure about any Animal either: how can you know there isn't an essential piece of hidden metadata traveling along that you are ignoring, which changes the meaning significantly?

Consider what happened when mobile phones started producing JPEGs with portrait vs landscape orientation metadata. Pictures were randomly sideways and upside down. You couldn't even rely on something as basic as the image width actually being, you know, the width. How many devs would anticipate that?

The only reason this wasn't a bigger problem is because for 99% of use-cases, you can just apply the rotation once upfront and then forget about it. That is, you can make a function RotatableImage => Image aka a Duck => Animal. This is an S => T that doesn't lose any information anyone cares about. This is the rare exception, only done occasionally, as a treat.

If you instead need to upgrade a whole image and display pipeline to support, say, high-dynamic range or P3 color, that's a different matter entirely. It will never truly be 100% done everywhere, we all know that. But should it be? It's another ChickenDuck scenario, because now some code wants images to stay simple, 8-bit and sRGB, while other code wants something else. Are you going to force each side to deal with the existence of the other, in every situation? Or will you keep the simplest case simple?

A plain old 2D array of pixels is not sufficient for T in the general case, but it is too useful on its own to simply throw it out. So you shouldn't make an AbstractImage which specializes into a SimpleImage and an HDRImage and a P3Image, because that means your SimpleImage isn't simple anymore. You should instead make an ImageView with metadata, which still contains a plain Image with only raw pixels. That is, a SimpleImage is just an ImageView<Image, NoColorProfile>. That way, there is still a regular Image on the inside. Code that provides or needs an Image does not need to change.

It's important to realize these are things you can only figure out if you have a solid idea of how people actually work with images in practice. Like knowing that we can just all agree to "bake in" a rotation instead of rewriting a lot of code. Architecting from the inside is not sufficient, you must position yourself as an external user of what you build, someone who also has a full-time job.

If you want a piece of software to be extensible, that means the software will become somebody else's development dependency. This puts huge constraints on its design and how it can change. You might say there is no such thing as an extensible schema, only an unfinished schema, because every new version is really a fork. But this doesn't quite capture it, and it's not quite so absolute in practice.

Colonel Problem

Interoperability is easy in pairs. You can model this as an Input<T> "A" connecting to an Output<T> "B". This does not need to cover every possible T, it can be a reduced subset R of T. For example, two apps exchange grayscale images (R) as color PNGs (T). Every R is also a T, but not every T is an R. This means:

  • Any function () => grayscaleImage is a valid substitute for () => colorImage.
  • Any function (colorImage) => void is a valid substitute for (grayscaleImage) => void.

This helps A, which is an => R pretending to be a => T. But B still needs to be an actual T =>, even if it only wants to be an R =>. Turning an R => into a T => is doable as long as you have a way to identify the R parts of any T, and ignore the rest. If you know your images are grayscale, just use any of the RGB channels. Therefore, working with R by way of T is easy if both sides are in on it. If only one side is in on it, it's either scraping or SEO.

But neither applies to arbitrary processing blocks T => T that need to mutually interoperate. If A throws away some of the data it received before sending it to B, and then B throws away other parts before sending it back to A, little will be left. For reliable operation, either A → B → A or B → A → B ought to be a clean round-trip. Ideally, both. Just try to tell a user you preserve <100% of their data every time they do something.

photoshop layer panel

Consider interoperating with e.g. Adobe Photoshop. A Photoshop file isn't just an image, it's a collection of image layers, vector shapes, external resources and filters. These are combined into a layer tree, which specifies how the graphics ought to be combined. This can involve arbitrary nesting, with each layer having unique blend modes and overlaid effects. Photoshop's core here acts like a kernel in the OS sense, providing a base data model and surrounding services. It's responsible for maintaining the mixed raster/vector workspace of the layered image. The associated "user space" is the drawing tools and inspectors.

Being mutually compatible with Photoshop means being a PSD => PSD back-end, which is equivalent to re-implementing all the "kernel space" concepts. Changing a single parameter or pixel requires re-composing the entire layer stack, so you must build a kernel or engine that can do all the same things.

Also, let's be honest here. The average contemporary dev eyes legacy desktop software somewhat with suspicion. Sure, it's old and creaky, and their toolbars are incredibly out of fashion. But they get the job done, and come with decades of accumulated, deep customizability. The entrenched competition is stiff.

This reflects what I call the Kernel Problem. If you have a processing kernel revolving around an arbitrary T => T block, then the input T must be more like a program than data. It's not just data and metadata, it's also instructions. This means there is only one correct way to interpret them, aside from differences in fidelity or performance. If you have two such kernels which are fully interoperable in either direction, then they must share the same logic on the inside, at least up to equivalence.

If you are trying to match an existing kernel T => T's features in your S => S, your S must be at least as expressive as their original T. To do more, every T must also be a valid S. You must be the Animal to their Duck, not a Duck to their Animal, which makes this sort of like reverse inheritance: you adopt all their code but can then only add non-conflicting changes, so as to still allow for real substitution. A concrete illustration is what "Linux Subsystem for Windows" actually means in practice: put a kernel inside the kernel, or reimplement it 1-to-1. It's also how browsers evolved over time, by adding, not subtracting.

Therefor, I would argue an "extensible kernel" is in the broad sense an oxymoron, like an "extensible foundation" of a building. The foundation is the part that is supposed to support everything else. Its purpose is to enable vertical extension, not horizontal.

If you expand a foundation without building anything on it, it's generally considered a waste of space. If you try to change a foundation underneath people, they rightly get upset. The work isn't done until the building actually stands. If you keep adding on to the same mega-building, maintenance and renewal become impossible. The proper solution for that is called a town or a city.

Naturally kernels can have plug-ins too, so you can wonder if that's actually a "motte user-land" or not. What's important is to notice the dual function. A kernel should enable and support things, by sticking to the essentials and being solid. At the same time, it needs to also ship with a useful toolset working with a single type T that behaves extensibly: it must support arbitrary programs with access to processes, devices, etc.

If extensibility + time = kitchen sink bloat, how do you counter entropy?

motte and bailey

You must anticipate, by designing even your core T itself to be decomposable and hence opt-in à la carte. A true extensible kernel is therefor really a decomposable kernel, or perhaps a kernel generator, which in the limit becomes a microkernel. This applies whether you are talking about Photoshop or Linux. You must build it so that it revolves around an A & B & C & ..., so that both A => A and B => B can work directly on an ABC and implicitly preserve the ABC-ness of the result. If all you need to care about is A or B, you can use them directly in a reduced version of the system. If you use an AB, only its pertinent aspects should be present.

Entity-Component Systems are a common way to do this. But they too have a kernel problem: opting in to a component means adopting a particular system that operates on that type of component. Such systems also have dependency chains, which have to be set up in the right order for the whole to behave right. It is not really A & B but A<B> or B<A> in practice. So in order for two different implementations of A or B to be mutually compatible, they again have to be equivalent. Otherwise you can't replace a Process<T> without replacing all the associated input, or getting unusably different output.

The main effect of à la carte architecture is that it never seems like a good idea to force anyone else to turn their Duck into a Chicken, by adopting all your components. You should instead try to agree on a shared Animal<T>. Any ChickenDuck that you do invent will have a limited action radius. Because other people can decide for themselves whether they truly need to deal with chickens on their own time.

None of this is new, I'm just recapturing old wisdom. It frankly seems weird to use programming terminology to have described this problem, when the one place it is not a big deal is inside a single, comfy programming environment. We do in fact freely import modules à la carte when we code, because our type T is the single environment of our language run-time.

But it's not so rosy. The cautionary tale of Python 2 vs 3: if you mess with the internals and standard lib, it's a different language, no matter how you do it. You still have a kernel everyone depends on and it can take over a decade to migrate a software ecosystem.

Everyone has also experienced the limits of modularity, in the form of overly wrapped APIs and libraries, which add more problems than they solve. In practice, everyone on a team must still agree on one master, built incrementally, where all the types and behavior is negotiated and agreed upon. This is either a formal spec, or a defacto one. If it is refactored, that's just a fork everyone agrees to run with. Again, it's not so much extensible, just perpetually unfinished.

À la carte architecture is clearly necessary but not sufficient on its own. Because there is one more thing that people tend to overlook when designing a schema for data: how a normal person will actually edit the data inside.

birds

Oil and Water

Engineering trumps theoretical models, hence the description of PSD above actually omits one point deliberately.

It turns out, if all you want to do is display a PSD, you don't need to reimplement Photoshop's semantics. Each .PSD contains a pre-baked version of the image, so that you don't need to interpret it. A .PSD is really two file formats in one, a layered PSD and something like a raw PNG. It is not a ChickenDuck but a DuckAnimal. They planned ahead so that the Photoshop format can still work if all you want to be is MSPaint => MSPaint. For example, if you're a printer.

This might lead you to wonder.

Given that PNG is itself extensible, you can imagine a PNG-PSD that does the same thing as a PSD. It contains an ordinary image, with all the Photoshop specific data embedded in a separate PSD section. Wouldn't that be better? Now any app that can read PNG can read PSD, and can preserve the PSD-ness. Except, no. If anyone blindly edits the PNG part of the PNG-PSD, while preserving the PSD data, they produce a file where both are out of sync. What you see now depends on which app reads it. PNG-PSDs would be landmines in a mixed ecosystem.

It's unavoidable: if some of the data in a schema is derived from other data in it, the whole cannot be correctly edited by a "dumb", domain-agnostic editor, because of the Kernel Problem. This is why "single source of truth" should always be the end-goal.

A fully extensible format is mainly just kicking the can down the road, saving all the problems for later. It suggests a bit of a self-serving lie: "Extensibility is for other people." It is a successful business recipe, but a poor engineering strategy. It results in small plug-ins, which are not first class, and not fundamentally changing any baked in assumptions.

But the question isn't whether plug-ins are good or bad. The question is whether you actually want to lock your users of tomorrow into how your code works today. You really don't, not unless you've got something battle-hardened already.

birds

If you do see an extensible system working in the wild on the Input, Process and Output side, that means it's got at least one defacto standard driving it. Either different Inputs and Outputs have agreed to convert to and from the same intermediate language... or different middle blocks have agreed to harmonize data and instructions the same way.

This must either flatten over format-specific nuances, or be continually forked to support every new concept being used. Likely this is a body of practices that has mostly grown organically around the task at hand. Given enough time, you can draw a boundary around a "kernel" and a "user land" anywhere. To make this easier, a run-time can help do auto-conversion between versions or types. But somebody still has to be doing it.

This describes exactly what happened with web browsers. They cloned each other's new features, while web developers added workarounds for the missing pieces. Not to make it work differently, but to keep it all working exactly the same. Eventually people got fed up and just adopted a React-like.

That is, you never really apply extensibility on all three fronts at the same time. It doesn't make sense: arbitrary code can't work usefully on arbitrary data. The input and output need to have some guarantees about the process, or vice versa.

Putting data inside a free-form key/value map doesn't change things much. It's barely an improvement over having a unknownData byte[] mix-in on each native type. It only pays off if you actually adopt a decomposable model and stick with it. That way the data is not unknown, but always provides a serviceable view on its own. Arguably this is the killer feature of a dynamic language. The benefit of "extensible data" is mainly "fully introspectable without recompilation."

You need a well-defined single type T that sets the ground rules for both data and code, which means T must be a common language. It must be able to work equally well as both an A, B and C, which are needs that must have been anticipated. Yet they should be built such that you can just use a D of your own, without inconvenient dependency. The key quality to aim for is not creativity but discipline.

If you can truly substitute a type with something else everywhere, it can't be arbitrarily extended or altered, it must retain the exact same interface. In the real world, that means it must actually do the same thing, only marginally better or in a different context. A tool like ffmpeg only exists because we invented a bajillion different ways to encode the same video, and the only solution is to make one thing that supports everything. It's the Unicode of video.

If you extend something into a new type, it's not actually a substitute, it's a fork trying to displace the old standard. As soon as it's used, it creates a data set that follows a new standard. Even when you build your own parsers and/or serializers, you are inventing a spec of your own. Somebody else can do the same thing to you, and that somebody might just be you 6 months from now. Being a programmer means being an archivist-general for the data your code generates.

* * *

If you actually think about it, extensibility and substitution are opposites in the design space. You must not extend, you must decompose, if you wish to retain the option of substituting it with something simpler yet equivalent for your needs. Because the other direction is one-way only, only ever adding complexity, which can only be manually refactored out again.

If someone is trying to sell you on something "extensible," look closely. Is it actually à la carte? Does it come with a reasonable set of proven practices on how to use it? If not, they are selling you a fairy tale, and possibly themselves too. They haven't actually made it reusable yet: if two different people started using it to solve the same new problem, they would not end up with compatible solutions. You will have 4 standards: the original, the two new variants, and the attempt to resolve all 3.

Usually it is circumstance, hierarchy and timing that decides who adapts their data and their code to whom, instead of careful consideration and iteration. Conway's law reigns, and most software is shaped like the communication structure of the people who built it. "Patch," "minor" or "major" release is just the difference between "Pretty please?", "Right?" and "I wasn't asking."

We can do a lot better. But the mindset it requires at this point is not extensibility. The job at hand is salvage.

August 27, 2021

IT architects try to use views and viewpoints to convey the target architecture to the various stakeholders. Each stakeholder has their own interests in the architecture and wants to see their requirements fulfilled. A core role of the architect is to understand these requirements and make sure the requirements are met, and to balance all the different requirements.

Architecture languages or meta-models often put significant focus on these views. Archimate has a large annex on Example Viewpoints just for this purpose. However, unless the organization is widely accustomed to enterprise architecture views, it is unlikely that the views themselves are the final product: being able to translate those views into pretty slides and presentations is still an important task for architects when they need to present their findings to non-architecture roles.

August 23, 2021

In the 15 years that I've been blogging, it's never been this quiet on my blog.

Blogging is an important part of me. It's how I crystallize ideas, reflect, and engage with thousands of people around the world. Blogging encourages me to do research; it improves my understanding of different topics. Blogging sometimes forces me to take sides; it helps me find myself.

I miss blogging. Unfortunately, I've lost my blogging routine.

At first, COVID-19 was to blame for that. I'd write many of my blog posts on the train to work. My train ride was one hour each way and that gave me plenty of time to write. Once in the office, there is zero time for blogging. COVID-19 interrupted my blogging routine and took away my protected writing time.

Then earlier this year, we moved from the suburbs of Boston to the city. Renovating our new condo, selling our old condo, and moving homes consumed much of my personal time — blogging time included. And now we live in the city, I no longer commute by train.

Admittedly, I've also felt blocked. I've been waiting to blog until I had something interesting to say, but nothing seemed interesting enough.

So I'm eager to find a new blogging routine. I'm a fan of routines. Routines add productivity and consistency to my life. Without a good blogging routine, I'm worried about the future of this blog.

To get back into a blogging routine, I made two decisions: (1) to target one blog post per week and (2) to balance my inner critic. I will no longer wait for something interesting to come along (as this blog post illustrates).

When you break out of any habit, it can be hard to get back into it. To get back into a routine, it's better to write something regularly than to write nothing at all. These seem achievable goals and I'm hopeful they get me blogging more frequently again.

August 20, 2021

published the following diary on isc.sans.edu: “Waiting for the C2 to Show Up“:

Keep this in mind: “Patience is key”. Sometimes when you are working on a malware sample, you depend on online resources. I’m working on a classic case: a Powershell script decodes then injects a shellcode into a process. There are plenty of tools that help you to have a good idea of a shellcode behavior (like scdbg)… [Read more]

The post [SANS ISC] Waiting for the C2 to Show Up appeared first on /dev/random.

August 19, 2021

Dear IoT manufacturers,

Yes, I admit: I like your products and my Geekness does not help! I like to play with them. If some are “gadgets” that finally land in a drawer amongst others with cables and connectors, some of them are really useful and I use them daily. You can probably imagine that, when I receive a new device, it’s not connected “in the wild” just after unboxing it. Wait… You do?

First, I read the provided documentation (you know the famous “RTFM”), I google for some forum or blog articles to see if other people already played with it. Finally, it is connected to a dedicated network without a full Internet connection (Read: “most of the egress traffic being blocked”). And then, the first problems arise… The initial setup fails, I need to restore to factory settings and try again. Basically, the process is:

Connect > Setup > Fail > Check firewall logs > Open ports > Reset > Try again > Drink more coffee

This discovery process is so annoying! Why your customers should sometimes perform something like “network reverse engineering” to understand how your product is working?

I’ve a message for you:

Please be transparent!

Why don’t you provide all the documentation for a safe setup? By example:

  • Required TCP/UDP port(s)
  • Hardcoded IP addresses/hostnames (yes, not everybody is using 8.8.8.8 as a DNS resolver or pool.ntp.org as NTP server!)
  • Specific services (VPN) and their purpose(s)
  • Avoid obscure protocols and stick to standard ones
  • Sometimes, even the device MAC address is not written on the box. To get the MAC you need to boot it a first timeç

Also avoid advices like:

If it does not work, disable the security feature <x>!

Why don’t you allow some critical settings to be customised like DNS, NTP, etc… or use proxy?

Why don’t you explain why this suspicious VPN connection is required? What are the data exchanged? What do you do with them (Ok, I’m dreaming)

I see you coming… You are right on one point: Such very technical pieces of information can’t be understood by many of your customers and your business model requires addressing the largest customers base. I don’t ask you to include this information in the documentation provided with the device but why not keep technical specs online for those that need to review them? The right balance must be found between usability and security and it’s up to me, your customer, to decide where to slide the cursor. If I disable this option, a sexy feature won’t be available? ok, fair enough but it’s my choice.

No, your FAQ does not contain relevant information for me. “Have you tried turning it off and on again” is not the right answer!

Thank you!

The post Public Message to IoT Manufacturers appeared first on /dev/random.

August 16, 2021

August 08, 2021

pi cluster

Last year I got a raspberry pi 4 to play with and installed Manjaro on it.

The main reason I went with Manjaro was that the ArchLinux Arm image/tgz for the Raspberry Pi 4 was still 32 bits, or you needed to create-your-own kernel.

But started to like Manjaro Linux, it provided a stable base with regular updates. This year I upgraded my setup with 2 additional Raspberry Pi 4 to provide clustering for my k3s (Kubernetes) setup. I used virtual machines on the Raspberry Pi to host the k3s nodes. Also because want to the Pi for other tasks and virtual machines makes it easier to split the resources. It’s also an “abstraction layer” if you want to combine the cluster with other ARM64 systems in the future.

I always (try to) to full disk encryption, when you have multiple nodes it’s important to be able to unlock the encryption remotely.

Install Manjaro on an encrypted filesystem

Manjaro will run an install script after the RPI is booted to complete the installion.

  • We have two options boot the pi from the standard non-encrypted image and extract/move it to an encrypted filesystem.
  • Extract the installation image and move the content to an encrypted filesystem.

You’ll find my journey of the second option below. The setup is mainly the same as I did last year, but with support to unlock the encryption with ssh.

The host system to extract/install the image is an x86_64 system running Archlinux.

Download and copy

Download and verify the Manjaro image from: https://www.manjaro.org/download/#raspberry-pi-4.

Copy the image to keep the original intact.

[root@vicky manjaro]# cp Manjaro-ARM-xfce-rpi4-20.06.img image

Create tarball

Verify the image

Verify the image layout with fdisk -l.

[root@vicky manjaro]# fdisk -l image
Disk image: 4.69 GiB, 5017436160 bytes, 9799680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x090a113e

Device     Boot  Start     End Sectors   Size Id Type
image1           62500  500000  437501 213.6M  c W95 FAT32 (LBA)
image2          500001 9799679 9299679   4.4G 83 Linux
[root@vicky manjaro]# 

We’ll use kpartx to map the partitions in the image so we can mount them. kpartx is part of the multipath-tools.

Map the partitions in the image with kpartx -ax, the “-a” option add the image, “-v” makes it verbose so we can see where the partitions are mapped to.

[root@vicky manjaro]# kpartx -av image
add map loop1p1 (254:10): 0 437501 linear 7:1 62500
add map loop1p2 (254:11): 0 9299679 linear 7:1 500001
[root@vicky manjaro]#

Create the destination directory.

[root@vicky manjaro]# mkdir /mnt/chroot

Mount the partitions.

[root@vicky manjaro]# mount /dev/mapper/loop1p2 /mnt/chroot
[root@vicky manjaro]# mount /dev/mapper/loop1p1 /mnt/chroot/boot
[root@vicky manjaro]#

Create the tarball.

[root@vicky manjaro]# cd /mnt/chroot/
[root@vicky chroot]# tar czvpf /home/staf/Downloads/isos/manjaro/Manjaro-ARM-xfce-rpi4-21.07.tgz .

Umount.

[root@vicky ~]# umount /mnt/chroot/boot 
[root@vicky ~]# umount /mnt/chroot
[root@vicky ~]# cd /home/staf/Downloads/isos/manjaro/
[root@vicky manjaro]# kpartx -d image
loop deleted : /dev/loop1
[root@vicky manjaro]# 

Partition and create filesystems

Partition

Partition your harddisk delete all partitions if there are partition on the harddisk.

I’ll create 3 partitions on my harddisk

  • boot partitions of 500MB (Type c ‘W95 FAT32 (LBA)’
  • root partitions of 50G
  • rest
[root@vicky ~]# fdisk /dev/sdh

Welcome to fdisk (util-linux 2.35.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x49887ce7.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-976773167, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-976773167, default 976773167): +500M

Created a new partition 1 of type 'Linux' and of size 500 MiB.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2): 2
First sector (1026048-976773167, default 1026048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (1026048-976773167, default 976773167): +50G

Created a new partition 2 of type 'Linux' and of size 50 GiB.

Command (m for help): n
Partition type
   p   primary (2 primary, 0 extended, 2 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (3,4, default 3): 
First sector (105883648-976773167, default 105883648): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (105883648-976773167, default 976773167): 

Created a new partition 3 of type 'Linux' and of size 415.3 GiB.

Command (m for help): t
Partition number (1-3, default 3): 1
Hex code (type L to list all codes): c

Changed type of partition 'Linux' to 'W95 FAT32 (LBA)'.

w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Command (m for help):  

Create the boot file system

The raspberry pi uses a FAT filesystem for the boot partition.

[root@vicky tmp]# mkfs.vfat /dev/sdh1
mkfs.fat 4.2 (2021-01-31)
[root@vicky tmp]# 

Create the root filesystem

Overwrite the root partition with random data

[root@vicky tmp]# dd if=/dev/urandom of=/dev/sdg2 bs=4096 status=progress
53644914688 bytes (54 GB, 50 GiB) copied, 682 s, 78.7 MB/s 
dd: error writing '/dev/sdg2': No space left on device
13107201+0 records in
13107200+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 687.409 s, 78.1 MB/s
[root@vicky tmp]# 

Encrypt the root filesystem

Benchmark

I booted the RPI4 from a sdcard to verify the encryption speed by executing the cryptsetup benchmark.

[root@minerva ~]# cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       398395 iterations per second for 256-bit key
PBKDF2-sha256     641723 iterations per second for 256-bit key
PBKDF2-sha512     501231 iterations per second for 256-bit key
PBKDF2-ripemd160  330156 iterations per second for 256-bit key
PBKDF2-whirlpool  124356 iterations per second for 256-bit key
argon2i       4 iterations, 319214 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      4 iterations, 321984 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b        23.8 MiB/s        77.7 MiB/s
    serpent-cbc        128b               N/A               N/A
    twofish-cbc        128b        55.8 MiB/s        56.2 MiB/s
        aes-cbc        256b        17.4 MiB/s        58.9 MiB/s
    serpent-cbc        256b               N/A               N/A
    twofish-cbc        256b        55.8 MiB/s        56.1 MiB/s
        aes-xts        256b        85.0 MiB/s        74.9 MiB/s
    serpent-xts        256b               N/A               N/A
    twofish-xts        256b        61.1 MiB/s        60.4 MiB/s
        aes-xts        512b        65.4 MiB/s        57.4 MiB/s
    serpent-xts        512b               N/A               N/A
    twofish-xts        512b        61.3 MiB/s        60.3 MiB/s
[root@minerva ~]# 
Create the Luks volume

The aes-xts cipher seems to have the best performance on the RPI4.

[root@vicky ~]# cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 256 --hash sha256 --use-random /dev/sdh2

WARNING!
========
This will overwrite data on /dev/sdh2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/sdh2: 
Verify passphrase: 
WARNING: Locking directory /run/cryptsetup is missing!
[root@vicky ~]# 
Open the Luks volume
[root@vicky ~]# cryptsetup luksOpen /dev/sdh2 cryptroot
Enter passphrase for /dev/sdh2: 
[root@vicky ~]# 

Create the root filesystem

[root@vicky tmp]# mkfs.ext4 /dev/mapper/cryptroot
mke2fs 1.46.3 (27-Jul-2021)
Creating filesystem with 13103104 4k blocks and 3276800 inodes
Filesystem UUID: 65c19eb5-d650-4d8a-8335-ef792604009d
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done   

[root@vicky tmp]# 

Mount and extract

Mount the root filesystem.

[root@vicky ~]# mount /dev/mapper/cryptroot /mnt/chroot
[root@vicky ~]# mkdir -p /mnt/chroot/boot
[root@vicky ~]# mount /dev/sdh1 /mnt/chroot/boot
[root@vicky ~]# 

And extract the tarball.

[root@vicky manjaro]# cd /home/staf/Downloads/isos/manjaro/
[root@vicky manjaro]# tar xpzvf Manjaro-ARM-xfce-rpi4-21.07.tgz -C /mnt/chroot/
[root@vicky manjaro]# sync

chroot

To continue the setup we need to boot or chroot into the operating system. It possible to run ARM64 code on a x86_64 system with qemu - qemu will emulate an arm64 CPU -.

Install qemu-arm-static

Install the qemu-arm package. It not in the main Archlinux distribution but it’s available as a AUR.

[staf@vicky ~]$ yay -S qemu-arm-static 

copy qemu-arm-static

Copy the qemu-arm-static into the chroot.

[root@vicky manjaro]# cp /usr/bin/qemu-arm-static /mnt/chroot/usr/bin/
[root@vicky manjaro]# 

mount proc & co

To be able to run programs in the chroot we need the proc, sys and dev filesystems mapped into the chroot.

For dns resolution to work you need to mount /run into the chroot, and start the systemd-resolved.service.

[root@vicky ~]# mount -t proc none /mnt/chroot/proc
[root@vicky ~]# mount -t sysfs none /mnt/chroot/sys
[root@vicky ~]# mount -o bind /dev /mnt/chroot/dev
[root@vicky ~]# mount -o bind /dev/pts /mnt/chroot/dev/pts
[root@vicky ~]# mount -o bind /run /mnt/chroot/run/
[root@vicky ~]# 

dns

Start the systemd-resolved.service on you host system. This is required to have dns available in your chroot during the installation.

Alternativaly you can use a proxy during the installation.

[root@vicky ~]# systemctl start systemd-resolved.service 

chroot

Chroot into ARM64 installation.

LANG=C chroot /mnt/chroot/

Set the PATH.

[root@vicky /]# export PATH=/sbin:/bin:/usr/sbin:/usr/bin

And verify that we are running aarch64.

[root@vicky /]# uname -a 
Linux vicky 5.12.19-hardened1-1-hardened #1 SMP PREEMPT Tue, 20 Jul 2021 17:48:41 +0000 aarch64 GNU/Linux
[root@vicky /]# 

Update and install vi

Update the public keyring

[root@vicky /]# pacman-key --init
gpg: /etc/pacman.d/gnupg/trustdb.gpg: trustdb created
gpg: no ultimately trusted keys found
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/etc/pacman.d/gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
==> Generating pacman master key. This may take some time.
gpg: Generating pacman keyring master key...
gpg: key DC12547C06A24CDD marked as ultimately trusted
gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/etc/pacman.d/gnupg/openpgp-revocs.d/55620B2ED4DE18F5923A2451DC12547C06A24CDD.rev'
gpg: Done
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
[root@vicky /]# 
[root@vicky /]# pacman-key --refresh-keys
[root@vicky /]#
[root@vicky etc]# pacman -Qs keyring
local/archlinux-keyring 20210616-1
    Arch Linux PGP keyring
local/archlinuxarm-keyring 20140119-1
    Arch Linux ARM PGP keyring
local/manjaro-arm-keyring 20200210-1
    Manjaro-Arm PGP keyring
local/manjaro-keyring 20201216-1
    Manjaro PGP keyring
[root@vicky etc]# 
[root@vicky etc]# pacman-key --populate archlinux manjaro archlinuxarm manjaro-arm

Update all packages to the latest version.

[root@vicky etc]# pacman -Syyu
:: Synchronizing package databases...
 core                                   237.4 KiB   485 KiB/s 00:00 [#####################################] 100%
 extra                                    2.4 MiB  2.67 MiB/s 00:01 [#####################################] 100%
 community                                6.0 MiB  2.74 MiB/s 00:02 [#####################################] 100%
:: Some packages should be upgraded first...
resolving dependencies...
looking for conflicting packages...

Packages (4) archlinux-keyring-20210802-1  manjaro-arm-keyring-20210731-1  manjaro-keyring-20210622-1
             manjaro-system-20210716-1

Total Installed Size:  1.52 MiB
Net Upgrade Size:      0.01 MiB

:: Proceed with installation? [Y/n] 

We need an editor.

root@vicky /]# pacman -S vi
resolving dependencies...
looking for conflicting packages...

Packages (1) vi-1:070224-4

Total Download Size:   0.15 MiB
Total Installed Size:  0.37 MiB

:: Proceed with installation? [Y/n] y
:: Retrieving packages...
 vi-1:070224-4-aarch64                         157.4 KiB  2.56 MiB/s 00:00 [##########################################] 100%
(1/1) checking keys in keyring                                             [##########################################] 100%
(1/1) checking package integrity                                           [##########################################] 100%
(1/1) loading package files                                                [##########################################] 100%
(1/1) checking for file conflicts                                          [##########################################] 100%
(1/1) checking available disk space                                        [##########################################] 100%
:: Processing package changes...
(1/1) installing vi                                                        [##########################################] 100%
Optional dependencies for vi
    s-nail: used by the preserve command for notification
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
[root@vicky /]# 

Unlock the encryption remotely

When you have multiple systems it’s handy to be able to unlock the encryption remotely.

Install the required mkinitcpio packages

[root@vicky /]# pacman -S mkinitcpio-utils mkinitcpio-netconf mkinitcpio-dropbear

mkinitcpio

HOOKS

Add netconf dropbear encryptssh to HOOKS before filesystems in /etc/mkinitcpio.conf.

Don’t include the encrypt as this will cause the boot image to try the unlock the encryption twice and will your system will fail to boot.

[root@vicky /]#  vi /etc/mkinitcpio.conf
HOOKS=(base udev plymouth autodetect modconf block netconf dropbear encryptssh filesystems keyboard fsck)

Create the boot image

[root@vicky /]# ls -l /etc/mkinitcpio.d/
total 4
-rw-r--r-- 1 root root 246 Jun 11 11:06 linux-rpi4.preset
[root@vicky /]# 
[root@vicky /]# mkinitcpio -p linux-rpi4
==> Building image from preset: /etc/mkinitcpio.d/linux-rpi4.preset: 'default'
  -> -k 5.10.52-1-MANJARO-ARM -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.10.52-1-MANJARO-ARM
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [plymouth]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [netconf]
  -> Running build hook: [dropbear]
There is no root key in /etc/dropbear/root_key existent; exit
  -> Running build hook: [encryptssh]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
[root@vicky /]# 

Don’t include the encrypt hooks, as this will cause the boot image to try the unlock the encryption twice and will your system will fail to boot. We need to have ssh host created to continue with the configuration. We’ll continue with the unlock encryption configuration after the first boot of the Raspberry Pi.

update boot settings…

Get the UUID for the boot and the root partition.

[root@vicky boot]# ls -l /dev/disk/by-uuid/ | grep -i sdh
lrwxrwxrwx 1 root root 12 Jul  8 11:42 xxxx-xxxx -> ../../sdh1
lrwxrwxrwx 1 root root 12 Jul  8 12:44 xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -> ../../sdh2
[root@vicky boot]# 

The Raspberry PI uses cmdline.txt to specify the boot options.

[root@vicky ~]# cd /boot
[root@vicky boot]# 
[root@vicky boot]# cp cmdline.txt cmdline.txt_org
[root@vicky boot]# 

Remove splash and add plymouth.enable=0. Set console=tty1.

cryptdevice=/dev/disk/by-uuid/43c2d714-9af6-4d60-91d2-49d61e93bf3e:cryptroot root=/dev/mapper/cryptroot rw rootwait console=s
erial0,115200 console=tty1 selinux=0 plymouth.enable=0 quiet plymouth.ignore-serial-consoles smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 
kgdboc=serial0,115200 elevator=noop usbhid.mousepoll=8 snd-bcm2835.enable_compat_alsa=0 audit=0

fstab

[root@vicky etc]# cp fstab fstab_org
[root@vicky etc]# vi fstab
[root@vicky etc]# 
# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
UUID=xxxx-xxxx  /boot   vfat    defaults        0       0

Finish your setup

Set the root password.

[root@vicky etc]# passwd

Set the timezone.

[root@vicky etc]# ln -s /usr/share/zoneinfo/Europe/Brussels /etc/localtime

Generate the required locales.

[root@vicky etc]# vi /etc/locale.gen 
[root@vicky etc]# locale-gen

Set the hostname.

[root@vicky etc]# vi /etc/hostname

clean up

Exit chroot

[root@vicky etc]# 
exit
[root@vicky ~]# uname -a
Linux vicky 5.12.19-hardened1-1-hardened #1 SMP PREEMPT Tue, 20 Jul 2021 17:48:41 +0000 x86_64 GNU/Linux
[root@vicky ~]# 

Make sure that there are no processes still running from the chroot.

[root@vicky ~]# ps aux | grep -i qemu
root       29568  0.0  0.0 151256 15900 pts/4    Sl   08:42   0:00 /usr/bin/qemu-aarch64-static /bin/bash -i
root       46057  0.1  0.0 152940 13544 pts/4    Sl+  08:50   0:05 /usr/bin/qemu-aarch64-static /usr/bin/ping www.google.be
root      151414  0.0  0.0   7072  2336 pts/5    S+   09:52   0:00 grep -i qemu

Kill the processes from the chroot.

root@vicky ~]# kill 29568 46057
[root@vicky ~]# 

Umount the chroot filesystems.

[root@vicky ~]# umount -R /mnt/chroot
[root@vicky ~]# 

Close the luks volume…

[root@vicky ~]# cryptsetup luksClose cryptroot
[root@vicky ~]# sync
[root@vicky ~]# 

Boot

Connect the usb disk to the raspberry pi and power it on. If you are lucky the PI will boot from the USB device and ask you to type the password to decrypt the root filesystem.

Configure unlock with ssh

Configure dropbear

Set the root key

Copy your public ssh key to /etc/dropbear/root_key

[root@staf-pi002 dropbear]# cat > root_key
[root@staf-pi002 dropbear]# chmod 600 root_key 
[root@staf-pi002 dropbear]# 

Convert the ssh rsa host key to pem

Dropbear can only handle the PEM format. We’ll need to convert the host key to the PEM format.

[root@staf-pi002 dropbear]# ssh-keygen -m PEM -p -f /etc/ssh/ssh_host_rsa_key
Key has comment 'root@stafwag-pi002'
Enter new passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved with the new passphrase.
[root@staf-pi002 dropbear]# 

Recreate the boot image.

[root@staf-pi002 dropbear]# mkinitcpio -p linux-rpi4
==> Building image from preset: /etc/mkinitcpio.d/linux-rpi4.preset: 'default'
  -> -k 5.10.52-1-MANJARO-ARM -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.10.52-1-MANJARO-ARM
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [plymouth]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [netconf]
  -> Running build hook: [dropbear]
Key is a ssh-rsa key
Wrote key to '/etc/dropbear/dropbear_rsa_host_key'
Error: Error parsing OpenSSH key
Error reading key from '/etc/ssh/ssh_host_dsa_key'
Error: Unsupported OpenSSH key type
Error reading key from '/etc/ssh/ssh_host_ecdsa_key'
dropbear_rsa_host_key : sha1!! 2a:a4:d5:a0:00:ce:1e:9f:88:84:72:f2:03:ce:ac:4a:27:11:da:09
  -> Running build hook: [encryptssh]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
[root@staf-pi002 dropbear]# 

Configure a static ip address

Update /boot/cmdline with your ip configuration.

cryptdevice=/dev/disk/by-uuid/43c2d714-9af6-4d60-91d2-49d61e93bf3e:cryptroot root=/dev/mapper /cryptroot rw rootwait console=serial0,115200 console=tty1 selinux=0 plymouth.enable=0 quiet plymouth.ignore-serial-consoles smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=serial0,115 200 elevator=noop usbhid.mousepoll=8 snd-bcm2835.enable_compat_alsa=0 audit=0 ip=xxx.xxx.xxx.xxx::yyy.yyy.yyy.254:255.255.255.0:staf-pi002:eth0:none

Reboot and test.

Have fun!

Links

August 06, 2021

I published the following diary on isc.sans.edu: “Malicious Microsoft Word Remains A Key Infection Vector“:

Despite Microsoft’s attempts to make its Office suite more secure and disable many automatic features, despite the fact that users are warned that suspicious documents should not be opened, malicious Word documents remain a key infection vector today. One of our readers (thanks Joel!) shared a sample that he received and, unfortunately, opened on his computer. The document was delivered to him via a spoofed email (sent by a known contact)… [Read more]

The post [SANS ISC] Malicious Microsoft Word Remains A Key Infection Vector appeared first on /dev/random.

August 04, 2021

Occasionally (about 4% of people contacting me) I get a job offer for somewhere in another country.

This is a list of places outside of Belgium where people are apparently interested in having me. 😀

  • India (Hyderabad)
  • Germany (Stuttgart, Wiesbaden)
  • United Kindom (London)
  • France (Paris)
  • Italy (Turin)
  • Spain (Madrid)
  • Netherlands (Amsterdam, Rotterdam, The Hague, Eindhoven, Almere, Arnhem, Deventer, Delft)
  • Sweden (Stockholm)
  • Austria (Graz)
  • Switzerland (Zurich)
  • Norway (Stavanger)
  • Luxembourg (Luxembourg City)

I have never considered moving permanently to another country for work, and I wouldn’t feel comfortable to move to a country where I don’t speak the language. Even if the company language is English, I would still need to communicate with people in everyday life, for example going to the shop. So from the list above, only France and the Netherlands would remain.

Besides the language, there is still the matter of being cut off from the people who matter to me. Yes there is the internet, and during the pandemic there was virtually no other way to stay in touch, but still… it’s not the same. I already have some friends in the Netherlands, so (hypothetically) I would feel less alone there. But there are still plenty of interesting local companies to work for, so no thanks for now.

Have you ever been invited to work abroad? If yes, what was your motivation for doing so? What were your experiences? Feel free to share in the comments!

The post Working abroad? appeared first on amedee.be.

August 03, 2021

After reading a few hundred emails from recruiters, I see a couple of trends popping up. I’m being contacted for job offers that really aren’t relevant or interesting for me. Some of them may be attributed to automatic keyword scanning. But still. If possible, I would kindly ask everyone not to contact me for any of the following:

  • Freelance: I have never done freelance before. Working freelance means that I would first have to start all the paperwork to become self-employed, and at this moment I’m not interested in doing all that. Maybe that could change in the faraway future, but at this point in my life I prefer permanent positions.
  • C/C++ embedded development: At one of my previous jobs, I did testing on the embedded software of a smart printer. Testing. Not development. I have never written a single line of C or C++ in my life. I would probably be able to read and understand other people’s code, but I’m sure that there are plenty of people who are really fluent in C/C++.
  • Drupal development: A long, long time ago, I made and maintained a few small Drupal sites. I have also been to one or two Drupal Dev Days in the early 2000s. I think I still have a T-shirt somewhere. But in all that time, I only did Drupal admin, I never went into the itty-gritty PHP to write custom Drupal code. And I’m pretty sure that my Drupal skills are quite rusty now.
  • Node.js development: Oh dear. I did a few tiny Node.js projects: some “glue code”, some rapid prototyping. Nothing fancy, nothing production quality, never more than 100 lines of code. Let’s not do that.
  • SharePoint development: With the eternal words of William Shakespeare:

Fie on’t! ah fie! ’tis an unweeded garden,
That grows to seed; things rank and gross in nature
Possess it merely. That it should come to this!

Hamlet, Act I, Scene ii

  • Quality Control Operator: This is typically a case of blindly searching for keywords and not verifying the results. I have worked as a Software Quality Engineer, so if you search only for “quality”, you’ll end up with jobs where you do actual physical inspection of physical products. Rule of thumb: if I can’t test it with an Assert-statement in some kind of programming language, then it’s probably not the kind of “quality” that I’m looking for.
  • Production / “blue collar jobs”: Yeah well let’s not do that at all, shall we? With all due respect for the people who do this type of work, and some of it is really essential work, but I don’t think that this would ever make me happy.
  • First line tech support: Been there, done that, got the battle scars. Never again, thank you very much.

Benefits for not contacting me for any of these: you don’t waste time chasing a dead-end lead, and I can spend more time on reading and reacting to job offers that actually are relevant, interesting and even exciting. Everybody happy! 🙂

The post Thanks, but no thanks appeared first on amedee.be.

July 30, 2021

The public cloud is a different beast than an on-premise environment, and that also reflects itself on how we (should) look at the processes that are actively steering infrastructure designs and architecture. One of these is the business continuity, severe incident handling, and the hopefully-never-to-occur disaster recovery. When building up procedures for handling disasters (DRP = Disaster Recovery Procedure or Disaster Recover Planning), it is important to keep in mind what these are about.

I published the following diary on isc.sans.edu: “Infected With a .reg File“:

Yesterday, I reported a piece of malware that uses archive.org to fetch its next stage. Today, I spotted another file that is also interesting: A Windows Registry file (with a “.reg” extension). Such files are text files created by exporting values from the Registry (export) but they can also and can also be used to add or change values in the Registry (import). Being text files, they don’t look suspicious… [Read more]

The post [SANS ISC] Infected With a .reg File appeared first on /dev/random.

July 29, 2021

I published the following diary on isc.sans.edu: “Malicious Content Delivered Through archive.org“:

archive.org, also known as the “way back machine” is a very popular Internet site that allows you to travel back in time and browse old versions of a website (like the ISC website). It works like regular search engines and continuously crawls the internet via bots. But there is another way to store content on archive.org: You may create an account and upload some content by yourself… [Read more]

The post [SANS ISC] Malicious Content Delivered Through archive.org appeared first on /dev/random.

July 27, 2021

Autoptimize 2.9 was released earlier today. It features:

  • New: per page/ post Autoptimize settings so one can disable specific optimizations (needs to be enabled on the main settings page under “Misc Options”).
  • New: “defer inline JS” as sub-option of “do not aggregate but defer” allowing to defer (almost) all JS
  • Improvement: Image optimization now automatically switches between AVIF & WebP & Jpeg even if lazyload is not active (AVIF has to be explicitly enabled).
  • Improvement: re-ordering of “JavaScript optimization” settings
  • Misc. other minor fixes, see the GitHub commit log

This release coincides with my father’s 76th birthday, who continues to be a big inspiration to me. He’s a mechanical engineer who after retirement focused his technical insights, experience and never-ending inquisitiveness on fountain pen design and prototyping, inventing a new bulkfiller mechanism in the process. Search the web for Fountainbel to find out more about him (or read this older blogpost I wrote in Dutch). Love you pops!

July 19, 2021

It’s been a long time since I last looked for a job myself. At job[-1] (7 years) and job[-2] (2 years), the employers contacted me while I was already working somewhere else, and at job[-3] I worked for 5 years, so all added up, that makes more than 14 years since I last did anything like this.

Job sites

I started with creating or updating a profile on a couple of job sites:

There are a couple more job sites that I know of but haven’t done anything with. Please leave a comment if you think any of them offer benefits over those listed above.

  • Viadeo (mostly French, so probably less useful)
  • Xing (I think they are mostly German-based)
  • StepStone
  • Facebook Job Search (I can’t imagine that any employer on Facebook Job Search wouldn’t also be on LinkedIn, but maybe I’ll try it to see if the search works better there)

I have also updated my CV and I’ve put it online: https://amedee.be/cv.

A torrent of messages

But then — I think — I made a mistake. The weather was nice, I wanted to be outdoors, trying to unwind a bit from the unusual times of the past months, and I disconnected.

Meanwhile the messages started pouring in, via email, LinkedIn (messages and connection requests), and occasionally a phone call from an unknown number. First just a few, then dozens, and just a few weeks later, already a couple of hundred. Oops.

The thing is, while I was technically available, I wasn’t yet mentally available. I still had to disconnect from the previous job, where I worked for more than 7 years, and I needed to think about what I really want to do next. Should I do something similar as before, because I already have the experience? Or should I try to find something that truly sparks joy? More on that later.

Strategies

Anyway, I had to come up with some strategies to deal with these high volumes of communication. First of all, not to get completely crazy, I defined a schedule, because otherwise I’d be responding to messages 24/7. There are other important activities too, like actively browsing through the job listings on various sites, or keeping up to date with current technology, or reaching out to my network, or having a social media presence (like this blog), or, you know, being social, having hobbies, and life in general.

One thing I noticed right away in many messages, is that people ask me for a CV — even though my LinkedIn profile is current. But I get it. And a separate document doesn’t confine me to the format of one specific website, and it helps me to emphasize what I think is important. So I made sure that my CV is available on an easy to reach URL: https://amedee.be/cv.

Then I made two short template messages, one in Dutch and one in English, to thank people for contacting me, where they can find my CV, and — for the LinkedIn people — what my email address is. That’s because I find it easier to track conversations in my mailbox. I can also give labels and flags to conversations, to help me in identifying the interesting ones.

LinkedIn

On LinkedIn, it went like this:

  • Read message.
  • Copy contact details to a spreadsheet.
  • Copy/paste the Dutch or English template message, so that they have my CV and email address.
  • If their message was really interesting(*), add an additional message that I’ll get back to them, and close the conversation. That’ll move it to the top of the message queue.
  • If their message wasn’t interesting or unclear, archive the conversation. If they come back after reading my CV, they’ll either end up in my mailbox, or if they use LinkedIn again, they’ll pop back up at the top of the message queue. But I don’t want to worry about the kind of recruiters that are just “fishing”.

This way I reduced my LinkedIn messages from about 150 to about 20. That’s 20 job offers that I want to give a second, more detailed look. Wow. And that’s just LinkedIn.

(*) What makes a message interesting?

  • It’s relevant.
  • The job isn’t too far to commute.
  • They clearly read my LinkedIn profile.
  • There is a detailed job description.
  • My gut feeling.

Email

Email is another huge source of messages. Fortunately Gmail gives me some tools there to help me. One of the first things I had to do, was to clean out my mailbox. Seriously. It was a dumpster fire. My Inbox had thousands (!) of unread emails. I used rules, filters, deleted emails (I think I deleted more than 100 000 emails), archived emails, and unsubscribed from many, many newsletters that had accumulated over the years. I am now at the point where there are currently 3 emails in my Primary Inbox, all 3 of them actionable items that I expect to finish in the next two weeks, and then those emails will be archived too.

Then, for any recent(ish) email about job offers, I labeled them as “jobhunt” and moved them to the Updates Inbox. That’s the Inbox that Gmail already used automatically for most of these emails, so that was convenient. (For those who don’t know: Gmail has 5 inboxes: Primary, Social, Promotions, Updates and Forums.) At this moment, there are 326 emails labeled “jobhunt”. I’m sure that there will be some overlap with LinkedIn, but still. That’s a lot.

I’ll be using Gmail’s stars, “Important” flag, and archive, to classify emails. Again, just like with LinkedIn, if an email isn’t really interesting at first glance, it’ll go to the archive after I’ve send them a short default message.

Phone

I get it. Really, I do. For some of you, talking on the phone comes naturally, you do it all the time, and it’s your preferred way of communication. For you it’s the fastest way to do your job.

But for me it’s a tough one. I wouldn’t say that I have outright phone phobia, but phone really is my least favorite communication channel. I need some time to charge myself up for a planned phone call, and afterwards I need some time to process it. Even if it is just writing down some notes about what was discussed and looking up some stuff.

It also has to do with how I process information. Speech is in one direction, always forward, and always at the same speed. You can’t rewind speech. But that’s not how my brain works. I want to read something again and again, or skip a paragraph, or first jump to a conclusion and then jump back to see how we got to that conclusion. Sometimes my thoughts go faster than how I express them, and putting it in writing helps me to see the gaps.

Calls out of the blue? I prefer to avoid those. Really. Especially the ones where people just want to get to know me. In the time it takes for me to do one such phone call (and I do take them seriously), I’m able to process several emails. So I very much prefer to focus first on contacts who have something concrete and actionable.

As mentioned above, I record contact information in a spreadsheet. I then import that information into Google Contacts, so that when someone calls me, I see their name on the screen of my phone, and not just a number. That also helps me to decide to pick up the phone or let it go to voicemail. I will get back to those that go to voicemail, but it’ll just be at my own pace.

Social media presence

I’m starting to put myself a bit more out there, by engaging in conversations on LinkedIn. I have also picked up blogging again, and I’m sharing links to my posts on LinkedIn, Facebook and Twitter. Besides my Facebook profile, I also have a Facebook page, but I’m not using that fanatically, because for myself at this point I don’t see Facebook as a professional tool.

On Twitter I have two accounts: @amedee and @AmedeeVanGasse. The former is mostly for personal stuff, and is mostly in Dutch. The latter is one that I created to tweet at tech conferences, but we all know how many tech conferences there were in the last 1.5 years… 🙂 Most tweets there will be in English.

Epilogue

I feel like this has become a very long blog post. Maybe too long, I don’t know. Maybe I should have split it up in several parts? But for me it felt like one story I had to tell.

If any of you social media gurus out there have some opinions to share, that’s what the comment box below is for. 🙂

The post So, how is the jobhunt going? appeared first on amedee.be.

In my job as domain architect for "infrastructure", I often come across stakeholders that have no common understanding of what infrastructure means in an enterprise architecture. Since then, I am trying to figure out a way to easily explain it - to find a common, generic view on what infrastructure entails. If successful, I could use this common view to provide context on the many, many IT projects that are going around.

July 15, 2021

Ik schrijf al meer dan twintig jaar over allerlei technische onderwerpen en ik heb daarbuiten nog heel wat andere interesses. Maar het is nog maar enkele jaren geleden dat ik in een epifanie plots de rode draad in al die onderwerpen zag: decentralisatie.

  • Ik geloof in de kracht van opensourcesoftware omdat de gebruikers daardoor niet alleen consument maar ook producent zijn.

  • Ik ben een groot voorstander van self-hosted software zodat je niet afhankelijk bent van clouddiensten van grote bedrijven.

  • Tien jaar geleden al schreef ik over Bitcoin en ik vind blockchains en andere vormen van distributed ledgers fascinerende technologie om zonder centrale controle transacties te kunnen uitvoeren.

  • In mijn strijd voor meer privacy vind ik het vooral belangrijk dat mensen controle over hun eigen data hebben, en daarom ben ik een grote fan van technologieën zoals Nextcloud, Solid en end-to-end encryptie.

  • Ik vind het belangrijk dat je zelf dingen kunt maken en programmeren, en ik ben dan ook blij dat ik deel uitmaak van de DIY/makersbeweging die dit democratiseert met de Raspberry Pi, ESP32, Arduino, 3d-printers, fablabs enzovoort.

  • Ik vind dat huishoudens zoveel mogelijk zelfvoorzienend zouden moeten zijn, niet alleen voor energie, maar ook voor voedsel.

More Equal Animals - The Subtle Art of True Democracy (Bron: Dan Larimer)

Toen ik dan in het begin van dit jaar Dan Larimers (gratis te downloaden) boek More Equal Animals - The Subtle Art of True Democracy ontdekte, las ik dit bijna in één ruk uit. Op bijna elke pagina las ik wel inzichten die verweven waren met de rode draad in mijn interesses.

In essentie is een blockchain een techniek om zonder centrale aansturing tot consensus te komen. Bij Bitcoin is dat consensus over transacties, maar volgens Larimer kun je de principes van blockchains ook op onze samenleving zelf toepassen. Een succesvolle samenleving implementeert volgens Larimer een proces dat zoveel mogelijk tot consensus leidt.

Democratie is in die ogen een proces om geschillen te beslechten met meerdere partijen en dat is iets waarin blockchains zo goed zijn. Echte democratie gaat volgens Larimer dan ook over het coördineren met andere personen terwijl je nog je persoonlijke autonomie en macht behoudt. En de ideale vorm van de samenleving bestaat dan uit gedecentraliseerde, autonome gemeenschappen op alle niveaus.

Voor PC-Active schreef ik uitgebreider over de visie die Larimer in zijn boek beschreef. Als onderwerpen zoals persoonlijke autonomie, gedecentraliseerd bestuur, een antifragiele samenleving en de gevaren van moral hazard je nauw aan het hart liggen, dan is het boek More Equal Animals een aanrader.

July 12, 2021

Quick public service announcement; Autoptimize 2.9 is almost ready to be released but given the planned release of WordPress 5.8 (July 20th) and the risk of support requests mixing up WordPress core update related issues with the Autoptimize update related issues, Autoptimize 2.9 will probably be released one week after WordPress 5.8, so on or around Tuesday 27th.

If you’re eager to use 2.9 (with better image optimization, improved JS optimization and per page/ post Autoptimize settings) you can off course download the beta here immediately.

July 10, 2021

Ik heb de titel van deze blog gewijzigd naar ‘Paepe Thoon‘, omdat dat mijn favoriete Leuvenaar is.

Récit de 3 jours de bikepacking pas toujours entre les gouttes à travers le Hainaut, le nord de la France et le Namurois

Je tiens ma formation initiale et ma philosophie du bikepacking de Thierry Crouzet, auteur du livre « Une initiation au bikepacking » (dans lequel je fais un peu de figuration) : Partir en autonomie, mais le plus léger possible, éviter les routes à tout prix,préférer l’aventure et la découverte à la performance ou à la distance.

Élève appliqué de Thierry, je me transforme en professeur pour initier mon filleul Loïc. Anecdote amusante : la différence d’âge entre Thierry et moi est la même qu’entre moi et Loïc. L’enseignement se propage, de génération en génération.

Après plusieurs virées dans les magasins de camping et une très grosse sortie de préparation de 112km, rendez-vous est pris pour notre premier trip de bikepacking sur une trace que j’ai dessinée pour traverser la province du Hainaut du Nord au sud, couper à travers la France dans la région de Givet avant de remonter le Namurois.

Jour 1 : le Hainaut sauvage, 103km, 1250d+

Nous nous retrouvons le vendredi matin sur le Ravel de Genappe. Je suis en retard : je connais tellement ce parcours que j’étais persuadé qu’il faisait 10km. Mon compteur indique déjà 15km lorsque je trouve Loïc qui piaffe d’impatience.

Le temps de me présenter sa config bikepack (il a notamment troqué le Camelbak sur le dos pour une ceinture porte-gourde) et nous voilà partis. À peine sorti des routes de Genappe et nous sommes confrontés à des chemins qui viennent de vivre deux mois de pluie quasi permanente. Cela signifie d’énormes flaques et une végétation plus qu’abondante. J’avais été témoin, sur mes sentiers habituels, de chemins se refermant complètement en trois ou quatre jours de beau temps après des semaines de pluie.

De tout côté, nous sommes entourés par les ronces, les orties. Mes bras deviennent un véritable dictionnaire des différentes formes de piqures et de lacérations. Il y’a les pointues, les griffues, celles qui se boursouflent, celles qui grattent, celles qui saignent. Loïc se marre en m’entendant hurler. Car je suis de ceux qui hurlent avant d’avoir mal, un cri rauque à mi-chemin entre banzaï et le hurlement de douleur. Loïc, lui, préfère garder son énergie et souffre en silence.

Le contournement des flaques s’avère parfois acrobatique et, moins agile que Loïc, je glisse sur un léger éperon de boue, les deux pieds et les fesses dans une énorme mare de gadoue.

Le soleil nous aide à prendre l’essorage de chaussettes à la rigolade sous la caméra amusée de Loïc qui filme. Je ne le sais pas encore, mais l’eau sera le thème central de notre épopée.

Nous dépassons enfin Fleurus pour traverser la banlieue de Charleroi par Chatelineau et Châtelet. À travers des rues peu engageantes qui serpentent entre des façades borgnes, nous suivons la trace qui s’engouffre sous un pont d’autoroute, nous conduit entre deux maisons pour nous faire déboucher soudainement sur de magnifiques sentiers à travers les champs. Comme si les habitants tenaient à cacher la beauté de leur région aux citadins et aux automobilistes.

Après des kilomètres assez plats, le dénivelé se fait brusquement sentir. Nous atteignons les bois de Loverval pour continuer parmi la région boisée contournant Nalinnes. Si les paysages sont loin d’être époustouflants, la trace est un véritable plaisir, verte, physique et nous fait déboucher dans le chouette village de Thy-le-Château.

Nous nous arrêtons pour un sandwich dans une boucherie. Le boucher nous explique sillonner la région en VTT électrique et est curieux de savoir quelle application nous utilisons pour nos itinéraires. Il note le nom « Komoot » sur un papier avant de s’offusquer lorsque je lui explique que nous nous relayons pour passer les commandes afin d’avoir toujours quelqu’un près des vélos.

« On ne vole pas à Thy-le-Château ! » nous assène-t-il avec conviction. Le sandwich est délicieux et nous continuons à travers des montées et des descentes abruptes, inondées de flaques ou de torrents. Les passages difficiles se succèdent et j’ai le malheur de murmurer que je rêve d’un kilomètre tout plat sur une nationale.

J’ai à peine terminé ma prière que mon mauvais génie m’exauce. Arrivant au pied de Walcourt, étrange village qui flanque une colline abrupte, la trace nous propose de suivre 500m d’une route nationale. Mais celle-ci se révèle incroyablement dangereuse. Une véritable autoroute ! Pour l’éviter, nous devrions remonter toute la pente que nous venons de descendre et faire une boucle de plusieurs kilomètres. Loïc propose de rouler le long de la nationale, derrière le rail de sécurité. « Ça se tente ! » me fait-il.

Nous sommes de cette manière à plusieurs mètres des véhicules et protégés par la barrière. Cependant, ce terre-plein est envahi de ronces, d’orties et des détritus balancés par les automobilistes. Les 500m dans le hurlement des camions et des voitures lancées à vive allure sont très éprouvants. Moi qui suis parfois réveillé par l’autoroute à plus de 3km de mon domicile, je me dis qu’on sous-estime complètement la pollution sonore du transport automobile.

Cette épreuve terminée, nous attaquons la dernière colline avant d’arriver aux Lacs de l’Eau d’Heure, objectif assumé pour notre première pause.

Juste avant le barrage de la Plate Taille, nous bifurquons vers une zone de balade autour du lac. Nous nous planquons dans un petit bosquet où, malgré les panneaux d’interdiction, j’enfile un maillot pour profiter d’une eau délicieuse à 19°C. Sur la rive d’en face, je pointe l’endroit où Loïc a fait son baptême de plongée en ma compagnie.

Le cuissard renfilé, je remonte sur ma selle et nous repartons. La trace nous conduit dans des petits sentiers qui longent la route du barrage. Nous arrivons sur le parking du spot de plongée où nous sommes censés retrouver la route, séparée de nous par une barrière fermée. Nous continuons un peu au hasard dans les bois avant de tomber sur le village de Cerfontaine.

Nous quittons désormais la civilisation. Plusieurs kilomètres de sentiers escarpés nous attendent. Loïc voit passer un sanglier. Je vois plusieurs biches. La région est sauvage. Deux choses inquiètent Loïc. Le risque d’orage et la question de trouver à manger. Hein chef ?

Heureusement, nous débouchons sur Mariembourg où une terrasse accueillante nous tend les bras au centre du village. Nous mangeons bercés par les cris de quelques villageois se préparant pour le match de foot du soir à grand renfort de canettes de bière.

Nous étudions la trace, occupation principale d’un bikepacker en terrasse. J’avais prévu un zigzag à proximité de Couvin pour aller découvrir le canyon « Fondry des Chiens ». Étant donné l’heure avancée, je suggère de couper à travers la réserve naturelle de Dourbes.

Nous sommes à peine sortis de Mariembourg que Loïc reconnait la gare. Nous sommes sur les terres où Roudou nous avait emmenés lors d’un mémorable week-end VTTnet en 2015.

La réserve naturelle de Dourbes est tout sauf plate. Un régal de vététiste. Un peu moins avec près de 100bornes dans les pattes. Ça fait partie du bikepacking : parler de régal pour ce qui te fait pester au moment même.

Nous arrivons sur les berges du Viroin. La trace nous fait monter vers le château de Haute-Roche, véritable nid d’aigle qui semble inaccessible. La pente est tellement abrupte qu’il faut escalader d’une main en tirant les vélos de l’autre. Loïc vient m’aider pour les derniers mètres.

Les ruines de la tour moyenâgeuse se dressent devant nous. Après cet effort, Loïc décide qu’il a bien mérité de contempler la vue. Il contourne la tour par un étroit sentier qui nécessite même un mètre d’escalade sur le mur médiéval. J’hésite à le suivre puis me laisse gagner par son enthousiasme.

Loïc a découvert une terrasse qui surplombe la vallée de manière majestueuse. Derrière nous, la tour, devant le vide et la vue. C’est magnifique.

Loïc a soudain une idée : » Et si on plantait la tente ici ? »

J’hésite. Nous sommes sur une propriété privée. L’à-pic n’est pas loin. Les sardines ne se planteront peut-être pas dans la terre fine de la terrasse. Mais je vois les yeux de Loïc pétiller. Je propose de tester de planter une sardine pour voir si c’est faisable. Loïc propose une manière de disposer les deux tentes sur la terrasse de manière à être le plus éloigné possible du trou. Nous finissons par retourner aux vélos, décrocher tous les sacs pour les amener sur notre terrasse. Il reste à faire passer les vélos eux-mêmes par le même chemin. C’est acrobatique, mais nous y arrivons et bénéficions d’un coucher de soleil sublime alors que nous montons nos tentes.

J’utilise un peu d’eau de mon Camelbak pour improviser une douche rapide. Je tends mes fesses à toute la vallée. Vue pour vue, paysage pour paysage.

De la vallée, les faibles cris nous informent que les Belges perdent le match de foot. Nous nous couchons à l’heure où les multiples camps scouts qui parsèment la vallée décident de se lancer dans des chants qui relèvent plus du cri permanent. Au bruit du matelas pneumatique, je devine que Loïc se retourne et ne trouve pas le sommeil.

Jour 2 : la brousse française, 80km, 1500d+

Les supporters et les scouts ont à peine achevé leur tintamarre que les coqs de la vallée prennent le relais. Il n’est pas encore 7h que j’émerge de ma tente. Loïc a très mal dormi et est abasourdi par l’humidité qui dégouline dans sa tente. J’espérais que l’altitude nous protégerait de l’humidité du Viroin, il n’empêche que tout est trempé. Mon Camelbak, mal fermé, s’est vidé dans mon sac de cadre qui, parfaitement étanche, m’offre le premier vélo avec piscine intérieure, comble du luxe.

Heureusement, il fait relativement beau. J’avais prévenu Loïc de compter une grosse heure pour le remballage des affaires, surtout la première fois. Le fait de devoir repasser les vélos en sens inverse le long de la tour complique encore un peu plus la tâche. Nous pratiquons la philosophie « no trace » et Loïc en profite même pour ramasser des vieilles canettes. Au final, il nous faut plus d’1h30 pour être enfin prêts à pédaler. Nous traversons les bois, descendons le long d’une route où nous aidons un scout flamand un peu perdu à s’orienter avant d’accomplir la courte, mais superbe escalade des canons de Vierves. Escalade que nous avions accomplie en 2015 avec Roudou et sa bande sans que j’en aie le moindre souvenir. En pensée, Loïc et moi envoyons nos amitiés et nos souvenirs aux copains de VTTnet.

La trace nous fait ensuite longer la route par un single escarpé avant de nous conduire à Treignes où nous déjeunons sur le parking d’un Louis Delhaize. Je constate que la trace fait un gros détour pour éviter 3km de route et nous fais escalader un énorme mamelon pour en redescendre un peu plus loin en France. La route étant peu fréquentée, je propose d’avancer par la route pour gagner du temps. L’avenir devait révéler ce choix fort judicieux.

Une fois en France, je m’arrange pour repiquer vers la trace. Nous faisons une belle escalade en direction du fort romain du Mont Vireux. Comme le fort en lui-même est au bout d’un long cul-de-sac, nous décidons de ne pas le visiter et de descendre immédiatement sur Vireux où nous traversons la Meuse.

Nous escaladons la ville. Je m’arrête à la dernière maison avant la forêt pour me ravitailler en eau auprès d’habitants absolument charmants et un peu déçus de ne pas pouvoir faire plus pour moi que de me donner simplement de l’eau.

Nous quittons désormais la civilisation pour nous enfoncer dans les plateaux au sud de Givet. Les chemins forestiers sont magnifiques, en montée permanente. Quelques panneaux indiquent une propriété privée. Nous croisons cependant un 4×4 dont le conducteur nous fait un signe amical qui me rassure sur le fait que le chemin soit public. Mais, au détour d’un sentier, une grande maison se dresse, absurde en un endroit aussi reculé. La trace la contourne et nous fait arriver devant une barrière un peu bringuebalante. Je me dis que nous sommes sur le terrain de la maison, qu’il faut en sortir. Nous passons donc la barrière, prenant soin de la refermer, et continuons une escalade splendide et très physique.

Au détour d’un tournant, je tombe sur une harde de sangliers. Plusieurs adultes protègent une quinzaine de marcassins. Les adultes hésitent en me voyant arriver. L’un me fait face avant de changer d’avis et emmener toute la troupe dans la forêt où je les vois détaler. Loïc arrive un peu après et nous continuons pour tomber sur une harde d’un autre type : des humains. Un patriarche semble faire découvrir le domaine à quelques adultes et une flopée d’enfants autour d’un pick-up. Il nous arrête d’un air autoritaire et nous demande ce que nous faisons sur cette propriété privée.

Je lui explique ma méprise à la barrière et la trace GPS en toute sincérité. Il accepte avec bonne grâce mes explications et tente de nous indiquer un chemin qui nous conviendrait. Je promets de tenter de marquer le chemin comme privé sur Komoot (sans réfléchir au fait que c’est en fait sur OpenStreetMap qu’il faut le marquer et que je n’ai pas encore réussi à le faire). Finalement, il nous indique la barrière la plus proche pour sortir du domaine qui se révèle être exactement le chemin indiqué par notre trace. Nous recroisons la harde de sangliers et de marcassins.

Nous escaladons la barrière en remarquant l’immensité de la propriété privée que nous avons traversée et sommes enfin sur un chemin public qui continue sur un plateau avant de foncer vers le creux qui nous sépare de la Pointe de Givet, Pointe que nous devons escalader à travers un single beaucoup trop humide et trop gras pour mes pneus. J’en suis réduit à pousser mon vélo en regardant Loïc escalader comme un chamois. Au cours du périple, les descentes et les montées trop grasses seront souvent à la limite du petit torrent de montagne. Une nouvelle discipline est née : le bikepack-canyoning.

Le sommet nous accueille sous forme de vastes plaines de hautes graminées où le chemin semble se perdre. La trace descend dans une gorge sensée déboucher sur la banlieue est de Givet. Mais la zone a été récemment déboisée. Nous descendons au milieu des cadavres de troncs et de branches dans un paysage d’apocalypse sylvestre. La zone déboisée s’arrête nette face à un mur infranchissable de ronces et de buissons. La route n’est qu’à 200m d’après le GPS, mais ces 200m semblent infranchissables. Nous remontons péniblement à travers les bois pour tenter de trouver un contournement.

Loïc fait remarquer que le paysage ressemble à une savane africaine. Nous roulons à l’aveuglette. Parfois, un souvenir de chemin semble nous indiquer une direction. Nous regagnons l’abri de quelques arbres avant de déboucher sur une vaste prairie de très hautes graminées, herbes et fleurs. Comme nous sommes beaucoup trop à l’ouest, je propose de piquer vers l’est. Une légère éclaircie dans un taillis nous permet de nous faufiler dans une pente boisée que je dévale sur les fesses, Loïc sur les pédales. Le pied de cette raide descente nous fait déboucher sur un champ de blé gigantesque. Du blé à perte de vue et aucun chemin, aucun dégagement. Nous nous résignons à la traverser en suivant des traces de tracteur afin de ne pas saccager les cultures. Les traces nous permettent de traverser le champ en largeur avant de s’éloigner vers l’ouest où la limite du champ n’est même pas visible.

À travers une haie d’aubépines particulièrement touffue, nous apercevons une seconde prairie. Avec force hurlements de douleur et de rage, nous faisons traverser la haie à nos vélos avant de suivre le même passage. De la prairie de pâturage, il devient facile de regagner un chemin desservant l’arrière des jardins de quelques maisons.

Après plusieurs heures de galère et très peu de kilomètres parcourus, nous regagnons enfin la civilisation. Loïc vient de faire son baptême de cet élément essentiel du bikepacking : l’azimut improvisé (autrement connu sous le nom de « On est complètement paumé ! »).

On pourrait croire qu’avec les GPS et la cartographie moderne, se perdre est devenu impossible. Mais la réalité changeante et vivante de la nature s’accommode mal avec la fixité d’une carte. L’état d’esprit du bikepacker passera rapidement du « Trouver le chemin le plus engageant pour arriver à destination » à « Trouver un chemin pour arriver à destination » à « Trouver un chemin praticable » pour finir par un « Mon royaume pour trouver n’importe quoi qui me permet tout simplement de passer ». Après des passages ardus dans les ronces ou les aubépines, après avoir dévalé des pentes particulièrement raides, l’idée de faire demi-tour n’est même plus envisageable. Il faut lutter pour avancer, pour survivre.

Un aphorisme me vient spontanément aux lèvres : « L’aventure commence lorsque tu as envie qu’elle s’arrête ».

Nous pénétrons alors dans Givet par l’ouest alors que j’avais prévu d’éviter la ville. Nous avons faim, nous sommes fatigués et nous n’avons fait qu’une vingtaine de kilomètres. Loïc a du mal de se rendre compte du temps perdu.

Sur une placette un peu glauque où se montent quelques maigres attractions foraines, nous enfilons un sandwich. Pour ma part, un sandwich que je viens d’acheter, mais pour Loïc, un sandwich particulièrement savoureux, car acheté le matin en Belgique et qui a fait toute l’aventure accroché au vélo. Se décrochant même avant une violente descente, emportant la veste de Loïc au passage et nécessitant une réescalade de la pente pour récupérer ses biens.

Hors de Givet, la nature reprend ses droits. Les montées boueuses succèdent aux singles envahis de flaques. Nous retrouvons la Belgique au détour d’un champ. Après quelques patelins typiquement namurois (les différences architecturales entre les bleds hennuyers, français et namurois me sautent aux yeux), nous enchainons de véritables montagnes russes jouant sur les berges de la Lesse.

Alors que j’ai un excellent rythme, une pause impromptue s’impose, le lieu me subjuguant par la beauté un peu irréelle d’une petite cascade. Je m’arrête et m’offre un bain de pieds tandis que Loïc prend des photos. Une fois sortis des gorges de la Lesse, nous nous arrêtons pour étudier la situation.

J’avais prévu un itinéraire initial de 330km, mais, Loïc devant être absolument rentré le 4 au soir, j’ai également concocté un itinéraire de secours de 270km pour le cas où nous aurions du retard. Les itinéraires divergeaient un peu après le retour en Belgique, l’un faisant une boucle par Rochefort, l’autre revenant en droite ligne vers Waterloo.

Par le plus grand des hasards, je constate que je me suis arrêté littéralement au point de divergence. Étant donné le temps perdu le matin, il me semble beaucoup plus sage de prendre l’itinéraire court, au grand dam de Loïc, très motivé, mais très conscient de la deadline.

Le seul problème est que mon itinéraire court ne passe par aucune ville digne de ce nom avant le lendemain, que je n’ai repéré aucun camping. Loïc me demande d’une petite voix inquiète si on va devoir se coucher le ventre vide. Parce qu’il y’aura aussi la question de trouver à manger. Hein chef ?

Je propose d’aviser un peu plus loin. Sur le chemin, quelques moutons échappés de leur enclos me regardent méchamment. Le mâle dominant commence même à gratter du sabot. Je leur crie dessus en fonçant, ils s’écartent.

Arrivés à un croisement, nous consultons les restaurants disponibles dans les quelques villages aux alentours. Un détour par Ciney me semble la seule solution pour s’assurer un restaurant ouvert. Nous sommes au milieu de nos hésitations lorsqu’un vététiste en plein effort s’arrête à notre hauteur. Tout en épongeant la sueur qui l’inonde, il nous propose son aide. Sa connaissance du lieu est bienvenue : il nous conseille d’aller à Spontin pour être sûrs d’avoir à manger puis d’aller dans un super camping au bord du Bocq. Par le plus grand des hasards, il est justement en train de flécher un parcours VTT qui passe tout prêt.

Nous le remercions et nous mettons à suivre ses instructions et ses flèches. Un petit détour assez pittoresque qui nous fait passer dans des singles relativement techniques par moment. C’est vallonné et la journée commence à se faire sentir. Psychologiquement, l’idée d’être presque arrivés rend ces 15km particulièrement éprouvants. Après une grande descente nous débouchons sur un carrefour au milieu de Spontin, carrefour orné, ô miracle, d’une terrasse de restaurant. Nous nous installons sans hésiter. Je commande une panna cotta en entrée.

Loïc est très inquiet à l’idée de ne pas avoir de place au camping recommandé par notre confrère. Le téléphone ne répond pas. De plus, ce camping est à une dizaine de kilomètres, dans un creux qu’il faudra escalader au matin. Alors que nous mangeons, j’aperçois derrière Loïc un panneau au carrefour qui indique un camping à seulement 2km. Un coup d’oeil sur la carte m’apprend que ce camping est à quelques centaines de mètres de notre trace. Je téléphone et le gérant me répond qu’il n’y a aucun souci de place.

Après le repas, nous sautons sur nos montures pour gravir ces 2 derniers kilomètres, le camping étant sur une hauteur. Rasséréné par la certitude d’avoir un logement et le ventre plein, Loïc me lâche complètement dans la côte. Son enthousiasme est multiplié, car il reconnait le camping. C’est une constante de ce tour : alors que je cherche à lui faire découvrir des choses, il reconnait sans cesse les lieux et les paysages pour y être venu à l’une ou l’autre occasion. Parfois même avec moi.

L’emplacement de camping est magnifique, aéré, calme avec une vue superbe. Par contre, les douches sont bouillantes sans possibilité de régler la température, les toilettes sont « à terrasse », sans planche ni papier. Je préfère encore chier dans les bois, mais la douche fait du bien.

La nuit est ponctuée d’épisode de pluie. Je croise les doigts pour qu’il fasse sec au moment de remballer la tente. Je n’ai encore jamais remballé le matériel sous la pluie.

Jour 3 : l’aquanamurois, 97km, 1200d+ 

À 7h30, je commence à secouer la tente de Loïc. Je l’appelle. Pas un bruit. Je recommence, plus fort. Je secoue son auvent. J’espère qu’il est toujours vivant. À ma cinquième tentative, un léger grognement me répond : « Gnnn… »

Loïc a dormi comme un bébé. Il émerge. Nous remballons paisiblement sous un grand soleil et faisons sécher les tentes.

La grande inquiétude de la journée, ce sont les menaces d’orage. Jusqu’à présent, nous sommes littéralement passés entre les gouttes. Nous précédons les gros orages de quelques heures, roulant toujours dans des éclaircies.

Sous un soleil très vite violent, nous nous échappons dans une série de petits singles envahis de végétation avant de commencer l’escalade pour sortir de Crupet. Nous escaladons un magnifique chemin à plus de 15%. Sur la gauche, la vue vers la vallée est absolument à couper le souffle avec des myriades de fleurs bleues au premier plan. À moins que ce ne soit la pente qui coupe le souffle. Des randonneurs nous encouragent, je suis incapable de répondre. Le sommet se profile au bord d’un camp scout. Après quelques centaines de mètres sur la route, un panneau indiquant une église médiévale attire mon attention. Cette fois-ci, c’est moi qui reconnais l’endroit ! Nous sommes à 1km du lieu de mon mariage. J’entraine Loïc dans un bref aller-retour pour envoyer une photo souvenir à mon épouse.

À partir de là, je connais l’endroit pour y être venu de multiples fois à vélo. Après des traversées de champs, nous nous enfonçons dans les forêts du Namurois, forêts aux chemins dévastés par les orages et les torrents de boue. Au village de Sart-Bernard, j’interpelle un habitant pour savoir s’il y’a un magasin ou une boulangerie dans les environs. À sa réponse, je comprends que j’aurais pu tout aussi bien lui demander un complexe cinéma 15 salles, un parc d’attractions et un centre d’affaires.

Nous nous enfonçons donc dans la forêt, zigzaguant entre les chemins privés, pour déboucher finalement sur Dave. Un kilomètre de nationale malheureusement incontournable nous permet d’aller traverser la Meuse sur une écluse juste au moment où celle-ci commence à se remplir pour laisser passer un bateau. Nous continuons le long du fleuve pour aller déguster une crêpe à Wépion. Le temps se couvre, mais reste sec.

La crêpe engloutie, il est temps de sortir du lit de la Meuse. Ma trace passe par une côte que j’ai déjà eu le plaisir d’apprécier : le Fonds des Chênes. Jamais trop pentue ni technique, la côte est cependant très longue et se durcit vers la fin, alors même qu’on a l’impression de sortir du bois et d’arriver dans un quartier résidentiel.

J’arrive au sommet lorsque les premières gouttes commencent à tomber. J’ai à peine le temps d’enfiler ma veste que le déluge est sur nous. Abrité sous un arbre, j’attends Loïc qui, je l’apprendrai après, a perdu beaucoup de temps en continuant tout droit dans une propriété privée.

À partir de ce moment-là, nous allons rouler sous des trombes d’eau incessantes. À travers les bois, nous descendons sur Malonne dont nous escaladons le cimetière à travers des lacets dignes d’un col alpin. La trace traverse littéralement le cimetière au milieu des tombes. Loïc s’étonne. Je réponds que, au moins, on ne dérange personne. C’est ensuite la descente sur Seneffe avant de longer la Sambre.

Lors de notre journée de préparation, nous sommes passés par là dans l’autre sens. Nous sommes en terrain connu, le côté exploration du bikepacking s’estompe pour laisser la place à la douleur psychologique du retour. Étant donné la pluie, je suis heureux de rentrer. Je n’ose imaginer installer une tente sous la pluie, renfiler des vêtements trempés le lendemain.

Nous n’essayons même plus de contourner les flaques qui se sont, de toute façon, transformées en inévitables marigots. Nous roulons des mètres et des mètres avec de l’eau jusqu’aux moyeux, chaque coup de pédale remplissant les chaussures d’eau comme une noria.

Loïc m’a plusieurs fois expliqué être motivé par la pluie. Sous la pluie, il pédale mieux. J’ai en effet observé qu’il supporte assez mal la chaleur alors que, pour moi, rien n’est aussi délectable que d’escalader un col en plein cagnard.

Ses explications se confirment. Loïc fonce, escalade. J’ai de plus en plus de mal à le suivre. L’eau me mine, ma nouvelle selle me torture les fesses. Nous traversons Spy, les plaines de Ligny, probablement tout aussi inondées qu’en 1815 et le golf de Rigenée. La trace traverse le bois Pigeolet, mais je me souviens avoir été bloqué au château de Cocriamont lors d’une de mes aventures antérieures. J’impose un demi-tour et nous gagnons Sart-Dames-Avelines par la route.

Alors que nous arrivons à Genappe, la pluie qui s’était déjà un peu calmée s’arrête tout à fait. Nous en profitons pour prendre un dernier verre en terrasse avant de nous dire au revoir. Nous avons le sentiment d’être à la maison.

Il me reste néanmoins encore 15km à faire. 15km essentiellement de Ravel. Mes chaussures sont presque sèches, l’optimisme est de mise.

C’est sans compter que le Ravel est inondé par endroit, traversé de coulées de boue. Certaines maisons se sont barricadées avec des sacs de sable. Des arbres arrachés rendent le passage compliqué. Alors que je traverse une flaque que je croyais étendue, mais peu profonde, le Ravel étant en théorie essentiellement plat, je m’enfonce jusqu’au moyeu. Je suis recouvert, ainsi que mon vélo et mes sacs, d’une boue jaune, grasse, épaisse et collante.

Il était dit que je ne pouvais pas arriver sec à Louvain-la-Neuve…

280km, près de 4000m de d+ et une expérience mémorable. Je suis enchanté d’avoir pu condenser en 3 jours toutes les expériences d’un trip de bikepacking : camping sauvage, heures perdues à pousser le vélo dans une brousse sans chemin, découragements suivis d’espoirs, pauses imprévues et terrasses délectables.

Maintenant que Loïc a gouté aux joies du bikepacking « extreme », je n’ai qu’une envie : qu’on reparte pour explorer d’autres régions. J’ai une attirance toute spéciale pour les Fagnes… Par contre, cette expérience de la pluie me fait renoncer au rêve de parcourir l’Écosse en bikepacking.

Alors qu’une Grande Traversée du Massif Central (GTMC pour les intimes) se profile avec Thierry, deux inquiétudes restent vives : mes fesses me font toujours autant souffrir (peut-être devrais-je passer le cap du tout suspendu) et je ne me sens pas psychologiquement armé pour affronter un bivouac sous la pluie.

Mais, après tout, l’aventure ne commence-t-elle pas au moment où tu as envie qu’elle s’arrête ?

Si la relation entre les humains et la technologie vous intéresse, abonnez-vous par mail ou par RSS pour ne rater aucun billet (max 2 par semaine). Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, offrez et partagez ses livres.

Ce texte est publié sous la licence CC-By BE.

July 08, 2021

I published the following diary on isc.sans.edu: “Using Sudo with Python For More Security Controls“:

I’m a big fan of the Sudo command. This tool, available on every UNIX flavor, allows system administrators to provide access to certain users/groups to certain commands as root or another user. This is performed with a lot of granularity in the access rights and logging/reporting features. I’m using it for many years and I’m still learning great stuff about it. Yesterday, at the Pass-The-Salt conference, Peter Czanik presented a great feature of Sudo (available since version 1.9): the ability to extend features using Python modules… [Read more]

The post [SANS ISC] Using Sudo with Python For More Security Controls appeared first on /dev/random.

As I mentioned in An IT services overview I try to keep track of the architecture and designs of the IT services and solutions in a way that I feel helps me keep in touch with all the various services and solutions out there. Similar to how system administrators try to find a balance while working on documentation (which is often considered a chore) and using a structure that is sufficiently simple and standard for the organization to benefit from, architects should try to keep track of architecturally relevant information as well.

So in this post, I'm going to explain a bit more on how I approach documenting service and solution insights for architectural relevance.

July 07, 2021

I did not write any wrap-up for a while because we are all stuck at home and most conference organizers still decided to cancel live events (even if it seems to change by the end of 2021 where some nice events are already scheduled). For the second time, Pass-The-Salt was converted to a virtual event. I like this small event with a great atmosphere. This edition in a few numbers: they received 27 proposals and selected 16 amongst them. They were presented spread across three half-days. Let’s review what was presented.

The first day started with Eloi Benoist-Vanderbeken who talked about jailbreak detection mechanisms in iOS and how to bypass them. iOS is a closed operating system and Apple really takes this seriously. Indeed, when a device has been “jailbroken”, all security measures are bypassed and the user gains full control of the device. Some apps really take this as a major problem and try to detect this. Think about banking applications but also games. To detect how the jailbreak detection is implemented, it’s mandatory to reverse the app to read the code. Also, Apple restricts a lot of operations: only signed code can be executed, side loading is not allowed (you remember the Epic Game problem?) and apps may only use public API’s. One of the best tools to help is debugging an app is Frida. How to debug an iOS app? Without a jailbreak in place, Frida is injected using ptrace but the app must be repackaged, with a lot of side effects. With a jailbreak already in place, Frida can just attach to the process. As a case study, a banking app was taken as an example. When the app is launched on a jailbroken device, it just crashes. A first set of checks did not reveal anything special. Check deeper, the app tried to trigger an invalid entry error. Some Frida usage examples were reviewed, based on hooks. Frida can hook syscalls to get information about the parameters and/or return values or completely replace the syscall with another function. It can also execute a function before any syscall. This is very interesting to follow what’s happening (example: to keep an eye on decrypted data). Another technique is the classic usage of breakpoints. Some techniques to detect jailbreaks: access specific files with open(), utime(), stats(), …), try to detect the presence of a debugger, check the parent PID, is the root filesystem writable?

The second presentation was the one of Esther Onfroy. She’s an Android expert, hacktivist and speaker. The title of Esther’s presentation was “Pithus: let’s open the Android pandora’s box”. Pithus is a project that focus on analyzing Android applications (APK). Why a new project? According to Esther, threat intelligence should not be the proprietary of a private company. For some, the price is a subscription is crazy high so the idea to develop an open-source tool based on free software and self-hostable. Pithus is able to check domains, certificates, the application time line, behavioral analysis and is linked to 3rd party services like MalwareBazaar. A specific function was presented deeper: The CFG (“Control Flow Graph”) dissection using AndroGuard. This is an interesting feature to search for share code:

Benoit Forgette presented “Hook as you want it“. Android is very popular in the mobile landscape. It provides a standard bootloader, an API for hardware, an ARM trustzone, High-level language (easy to write apps) and IPC. Review of the Android permission enforcement mechanism. AST is a tree and a hook is a programming trick that helps to intercept system calls. Once again here Frida will be used.

The next talk was “PatrowlHears and Survival tips for prioritizing threats” by Nicolas Mattiocco. It starts with a quick overview of Patrowl with themotto “demotivate attackers”. In vulnerability management, process is key: “How to manage vulnerabilities?”. It’s not only running a tool at regular interval:

One of the problems is the changing landscape, a lack of skilled people, budget pressure, … Automation is also a key point. And when running the tool more often, you get more findings and more alerts How to improve this? Then Nicolas explained how to prioritise findings. Some ideas: check the CVSS score, are we vulnerable? Is it exposed to the Internet? Is it a critical asset? Functional exploit? Patch or work around? etc… Most important: does it have a nice logo? 🙂 To help in this process, PatrolHears was introduced. It provides scalable, free and open-source solutions for orchestrating Security Operations and providing Threat Intelligence feeds. PatrowlHears is an advanced and real-time Vulnerability Intelligence platform, including CVE, exploits and threats news. Its also a feed aggregator (Packetstorm, Tenable Nessus DB, Metasploit, Exploitdb, ZDI, … Great tool to check if you’re dealing with vulnerability management.

To wrap-up the first day, we had a pretty funny presentation by Michael Hamm from circl.lu. The Luxembourg CERT produces always nice content and it was, once again, the case. Michael presented three live demos (yes, live) of USB devices alterations from a forensic point of view. The first one was about changing the content of a USB disk mounted in “read-only” mode. Yes, it’s possible. That’s why using a physical write-blocker is always highly recommended. The second one, was a kind of DoS attack: By modifying only a few bytes on the filesystem, a Linux system will, once the modified disk is inserted, mount it up to 250 times 😉 Finally, the last demo was even more impressive: by altering the MDR (Master Boot Record) and creating a “Polyglot boot sector”, Michael demonstrated how different files can be seen by different operating systems. Mounting the drive on Linux, some files will be listed, on Windows, other files will be listed.

Day two started with “Fedora CoreOS, a container focused OS to securely deploy and run applications” by Timothée Ravier. Intro to CoreOS: automatic updates, automatic provisioning (Ignition is used to provision nodes on 1st boot), immutable infrastructure and finally huge use of containers. What about security? “Softwares have bugs” so reduce the OS footprint (-> reduce the attack surface). Next, use safer languages: Go & Rust (memory safe languages). They use rpm-ostree to update the system (think about a “git for operating system”. Uses read-only filesystems and a clear split across important directories (/usr, /etc, /var). Everything runs in containers (podman, docker). Confinement with SELinux. Demos! The first one was to run a Matrix server More about CoreOS here.

The second talk was provided by Clément Oudot, a regular PTS speaker: “Hosting Identity in the Cloud with free softwares“. After a nice live song, Clément did a quick introduction/recap about IAM (“Identity and Access Management”) and a review of the market: big players are expensive – pay per user – and complex to install/manage.

So what about FOSS? A lot of tools exist but don’t cover the complete set of IAM features. FusionIAM is a project to propose a unified platform based on different tools: OpenLDAP, LDAP ToolBox, Fusion Directory, LDAP Synchronization Connector, and LemonLDAP:NG. From a features point of view, it provides whites pages, access manager, sync connector, service desk, directory server, directory manager. A great solution to investigate if you need to deploy IAM in a cloud environment.

The next presentation was “Biscuit: pubkey signed token with offline attenuation and Datalog authz policies” by Geoffroy Couprie. Geoffroy works for a hosting company and is often facing authentication issues (example for API’s). A classic way to authenticate users is a JWT (JSON Web Token). They are usually signed using a public key and contain a lot of data. They are not perfect therefore Geoffroy developed a new authentication and authorization token called Biscuit. You can find more information about this project here (code) and here.

Ange Albertini presented “Generating Weird Files“. Ange is known to play a lot with file formats. He already presented several pieces of research where he demonstrated how files can be built in a way that, depending on the program used to open them, they will present different types of content. Today, Ange presented his tool called Mitra. The open-source tool takes 2 files, checks their types, and generates possible polyglots.

What are the strategies used by Mitra?

  • Concatenation: easy because some file format do not need to start at offset 0
  • Cavities (filling empty spaces)
  • Parasite (abusing comments in metadata)
  • Zipper

Ange performed some demos. It was funny to see a reference to 1 file that identified up to 190 different file formats! The conclusion to this? These techniques could be prevented by creating new file formats that starts at offset 0 (magic bytes).

The next time slot was assigned to … me! I presented “Home-Made Distributed Blocklist“. More to come soon on this blog!

We wrapped the afternoon with a very interesting talk by Peter Czanik: “Security alerting made easy using Python“. Peter is a also a regular contributor to the conference and is leading open source products like syslog-ng or Sudo. Even if I’m an old Sudo user (probably for more than 15 years), I’m always learning new stuff about this wonderful tool and it was again the case. Peter explained how to interconnect Sudo and Python. Sudo is a modular tool that accepts plugins in C but also in Python to extend features and controls. You can filter data, alert, sanitize them etc… Peter reviewed some examples of integration.

Day 3 started with some Kunernetes security stuff: “ATT&CKing Kubernetes: A technical deep dive into the new ATT&CK for Containers” by Magno Login. Magno is the maintainer of the following github project: awesome-k8s-security. Then, he talked about the MITRE ATT&CK and more precisely, the matrix dedicated to containers (link). This matrix was created for a while but was updated recently with plenty of new content (thanks to the contribution of volunteers)

The second part of the talk was a suite of attacks scenarios based on a vulnerable Drupal CMS instance. Magno presented many techniques from the MITRE matrix. Scripts described in demos were collected from honeypots or found on C2 servers.

Then we switched to the wonderful world of shellcodes with Yashdeep Saini and Harpreet Singh who presented “Revisiting the Art of Encoder-Fu for novel shellcode obfuscation techniques“. Back to the roots with a review of the differences between x86, x64 and ARM registers and instructions (comparisons, shifting, control flows, stack, data movement and arithmetic operations.

They reviewed how encoders affect the shell code (XOR, Shikata_ga_nai, etc). Another comparison was to show comparative graphs based on the distributions of instructions:

Other observations: encoder types adding laters and branches(mode control, call changes). Other add transformation (more data movements). Why encode? The shell code can be transformed as per transport supported by the target applications. Bad characters can be replaced and more obfuscation.

In Search of Lost Time: A Review of JavaScript Timers in Browsers” by Thomas Rokicki. Timing attacks exploit timing differences to infer secrets from the sanbox.Attacks can be classified in 4 classes: Harare-contention-base attack (Ex: rawhammer.js), Transcient execution attacks (Spectre), Attacks based on system resources (Keystroke attacks, memory deduplication attacks ) and attacks based on browser resources(History sniffing or fingerprinting). This is a topic that remains always difficult to follow for me… I just kept this in mind as a recap to the talk: “If you are concerned by timing attacks, use the TOR browser, they are the strictest on timers countermesures

Then Nils Amiet and Tommaso Gagliardoni presented “ORAMFS: Achieving Storage-Agnostic Privacy“. The important word here is “privacy” or how to store data on a 3rd party service in a safe way. The classic control is to encrypt the complete set of data. But even if data can’t be read more info can be leaked: The size of data, some fields of database will change, etc… Think about an employees db. Example with the good old Truecrypt software: Create a TC volume, encrypted then mounted by the user. A cool feature was the “hidden volume” but can be detected even without the passphrase:

Keep in mind that “Encryption alone does not hide access patterns“. That’s why oramfs (Oblivious Random Access Machines) has been developed. Storage agnostic, written in Rust, resizing supported, multiple encryption cyphers, the tool was demonstrated live. Nice demo!

Finally, to wrap-up day 3, Damien Cauquil presented “Meet Piotr, a firmware emulation tool for trainers and researchers“. Damien performs a lot of IoT trainings and was looking for a nice solution to setup labs. Indeed, even more with the pandemic, it has been challenging to prepare labs. For example, some IoT device might be discontinued or their firmware upgraded to a non-vulnerable version. Another requirement was the size of the lab (to not download gigabytes of data). That’s why Piotr has been developed. It is based on Qemu, Firmadyne and ARM-X. After describing the architecture (how those components are interconnected), Damien performed some demos and booted/exploited some IoT devices. It looks to be a great solution also to researchers. If interested, the code is available here.

And that’s over for this virtual edition of Pass-The-Salt! Let’s hope that the next one will be in-person. Thanks to the organisers for keeping the event up’n’running! All slides are available online as well as videos.

The post Pass-The-Salt 2021 Virtual Wrap-Up appeared first on /dev/random.

I published the following diary on isc.sans.edu: “Python DLL Injection Check“:

They are many security tools that inject DLL into processes running on a Windows system. The classic examples are anti-virus products. They like to inject plenty of code that, combined with API hooking, implements security checks. If DLLs are injected into processes, they can be detected and it’s a common anti-debugging or evasion technique implemented by many malware samples. If you’re interested in such techniques, they are covered in the FOR610 training. The detection relies on a specific API call GetModuleFileName()… [Read more]

The post [SANS ISC] Python DLL Injection Check appeared first on /dev/random.

July 05, 2021

Axl staring at the Boston skyline, waiting for the Fourth of July fireworks to start.
Axl staring at the Boston skyline, waiting for the Fourth of July fireworks to start. It had been years since he saw the fireworks so it was extra special, despite the fog and rain.
Cover Image

Live

It is actually pretty easy to build a mediocre headless React today, i.e. an implementation of React that isn't hooked directly into anything else.

react-reconciler is an official package that lets you hook up React to anything already. That's how both React-DOM and React-Native share a run-time.

Most third-party libraries that use it (like react-three-fiber) follow the same approach. They are basically fully wrapped affairs: each notable Three.js object (mesh, geometry, material, light, ...) will tend to have a matching node in the React tree. Three.js has its own scene tree, like the browser has a DOM, so react-reconciler will sync up the two trees one-to-one.

The libraries need to do this, because the target is a retained in-memory model. It must be mutated in-place, and then re-drawn. But what would it look like to target an imperative API directly, like say 2D Canvas?

You can't just call an imperative API directly in a React component, because the idea of React is to enable minimal updates. There is no guarantee every component that uses your imperative API will actually be re-run as part of an update. So you still need a light-weight reconciler.

Implementing your own back-end to the reconciler is a bit of work, but entirely doable. You build a simple JS DOM, and hook React into that. It doesn't even need to support any of the fancy React features, or legacy web cruft: you can stub it out with no-ops. Then you can make up any <native /> tags you like, with any JS value as a property, and have React reconcile them.

Then if you want to turn something imperative into something declarative, you can render elements with an ordinary render prop like this:

<element render={(context) => {
  context.fillStyle = "blue";
  context.drawRect(/*...*/);
}} />

This code doesn't run immediately, it just captures all the necessary information from the surrounding scope, allowing somebody else to call it. The reconciler will gather these multiple "native" elements into a shallow tree. They can then be traversed and run, to form a little ad-hoc program. In other words, it's an Effect-like model again, just with all the effects neatly arranged and reconciled ahead of time. Compared to a traditional retained library, it's a lot more lightweight. It can re-paint without having to re-render any Components in React.

You can also add synthetic events like in React-DOM. These can be forwarded with conveniences like event.stopPropagation() replicated.

I've used this with great success before. Unfortunately I can't show the results here—maybe in the future—but I do have something else that should demonstrate the same value proposition.

React works hard to synchronize its own tree with a DOM-like tree, but it's just a subset of the tree it already has. If you remove that second tree, what's left? Does that one tree still do something useful by itself?

I wagered that it would and built a version of it. It's pretty much just a straight up re-implementation of React's core pattern, from the ground up. It has some minor tweaks and a lot of omissions, but all the basics of hook-driven React are there. More importantly, it has one extra superpower: it's designed to let you easily collect lambdas. It's still an experiment, but the parts that are there seem to work fine already. It also has tests.

Yeet Reduce

As we saw, a reconciler derives all its interesting properties from its one-way data flow. It makes it so that the tree of mounted components is also the full data dependency graph.

So it seems like a supremely bad idea to break it by introducing arbitrary flow the other way. Nevertheless, it seems clear that we have two very interesting flavors just asking to be combined: expanding a tree downstream to produce nodes in a resumable way, and yielding values back upstream in order to aggregate them.

Previously I observed that trying to use a lambda in a live DFG is equivalent to potentially creating new outputs out of thin air. Changing part of a graph means it may end up having different outputs than before. The trick is then to put the data sinks higher up in the tree, instead of at the leaves. This can be done by overlaying a memoized map-reducer which is only allowed to pass things back in a stateless way.

yeet reduce

The resulting data flow graph is not in fact a two-way tree, which would be a no-no: it would have a cycle between every parent and child. Instead it is a DFG consisting of two independent copies of the same tree, one forwards, one backwards, glued together. Though in reality, the second half is incomplete, as it only needs to include edges and nodes leading back to a reducer.

chain of fibers in the forwards direction turns down and back to yield values in the backwards direction

Thus we can memoize both the normal forward pass of generating nodes and their sinks, as well as the reverse pass of yielding values back to them. It's two passes of DFG, one expanding, one contracting. It amplifies input in the first half by generating more and more nodes. But it will simultaneously install reducers as the second half to gather and compress it back into a collection or a summary.

When we memoize a call in the forward direction, we will also memoize the yield in the other direction. Similarly, when we bust a cache on the near side, we also bust the paired cache on the far side, and keep busting all the way to the end. That's why it's called Yeet Reduce. Well that and yield is a reserved keyword.

when 'yeeting' a value, you throw it into a bin on the far side and knock everything down after it

What's also not obvious is that this process can be repeated: after a reduction pass is complete, we can mount a new fiber that receives the result as input. As such, the data flow graph is not a single expansion and contraction, but rather, many of them, separated by a so-called data fence.

This style of coding is mainly suited for use near the top of an application's data dependency graph, or in a shallow sub-tree, where the number of nodes in play is typically a few dozen. When you have tons of tiny objects instead, you want to rely on data-level parallelism rather than mounting each item individually.

Horse JS

I used to think a generalized solution for memoized data flow would be something crazy and mathematical. The papers I read certainly suggested so, pushing towards the equivalent of automatic differentiation of any code. It would just work. It would not require me to explicitly call memo on and in every single Component. It should not impose weird rules banning control flow. It would certainly not work well with non-reactive code. And so on.

There seemed to be an unbridgeable gap between a DFG and a stack machine. This meant that visual, graph-based coding tools would always be inferior in their ability to elegantly capture Turing-complete programs.

Neither seems to be the case. For one, having to memoize things by hand doesn't feel wrong in the long run. A minimal recomputation doesn't necessarily mean a recomputation that is actually small and fast. It feels correct to make it legible exactly how often things will change in your code, as a substitute for the horrible state transitions of old. Caching isn't always a net plus either, so fully memoized code would just be glacial for real use cases. That's just how the memory vs CPU trade-off falls these days.

That said, declaring dependencies by hand is annoying. You need linter rules for it because even experienced engineers occasionally miss a dep. Making a transpiler do it or adding it into the language seems like a good idea, at least if you could still override it. I also find <JSX> syntax is only convenient for quickly nesting static <Components> inside other <Components>. Normal JS {object} syntax is often more concise, at least when the keys match the names. Once you put a render prop in there, JSX quickly starts looking like Lisp with a hangover.

When your Components are just resources and effects instead of widgets, it feels entirely wrong that you can't just write something like:

live (arg) => {
  let [service, store] = mount [
     Service(...),
     Store(...),
  ];
}

Without any JSX or effect-like wrappers. Here, mount would act somewhat like a reactive version of the classic new operator, with a built-in yield, except for fiber-mounted Components instead of classes.

I also have to admit to being sloppy here. The reason you can think of a React component as an Effect is because its ultimate goal is to create e.g. an HTML DOM. Whatever code you run exists, in theory, mostly to generate that DOM. If you take away that purpose, suddenly you have to be a lot more conscious of whether a piece of code can actually be skipped or not, even if it has all the same inputs as last time.

This isn't actually as simple as merely checking if a piece of code is side-effect free: when you use declarative patterns to interact with stateful code, like a transaction, it is still entirely contextual whether that transaction needs to be repeated, or would be idempotent and can be skipped. That's the downside of trying to graft statelessness onto legacy tech, which also requires some mutable water in your immutable wine.

I did look into writing a Babel parser for a JS/TS dialect, but it turns out the insides are crazy and it takes three days just to make it correctly parse live / mount with the exact same rules as async / await. That's because it's a chain of 8 classes, each monkey patching the previous one's methods, creating a flow that's impractical to trace step by step. Tower of Babel indeed. It's the perfect example to underscore this entire article series with.

It also bothers me that each React hook is actually pretty bad from a garbage collection point of view:

const memoized = useMemo(() => slow(foo), [foo]);

This will allocate both a new dependency array [foo] and a new closure () => slow(foo). Even if nothing has changed and the closure is not called. This is unavoidable if you want this to remain a one-liner JS API. An impractical workaround would be to split up and inline useMemo into into its parts which avoid all GC:

// One useMemo() call
let memoized;
{
  useMemoNext();
  useMemoPushDependency(foo);
  memoized = useMemoSameDependencies() ? useMemoValue() : slow(foo);
}

But a language with a built-in reconciler could actually be quite efficient on the assembly level. Dependencies could e.g. be stored and checked in a double buffered arrangement, alternating the read and write side.

I will say this: React has done an amazing job. It got popular because its Virtual DOM finally made HTML sane to work with again. But what it actually was in the long run, was a Trojan horse for Lisp-like thinking and a move towards Effects.

No-API

So, headless React works pretty much exactly as described. Except, without the generators, because JS generators are stateful and not rewindable/resumable. So for now I have to write my code in the promise.then(…) style instead of using a proper yield.

I tried to validate it by using WebGPU as a test case, building out a basic set of composable components. First I hid the uglier parts of the WebGPU API inside some pure wrappers (the makeFoo(...) calls below) for conciseness. Then I implemented a blinking cube like this:

export const Cube: LiveComponent<CubeProps> = memo((fiber) => (props) => {
  const {
    device, colorStates, depthStencilState,
    defs, uniforms, compileGLSL
  } = props;

  // Blink state, flips every second
  const [blink, setBlink] = useState(0);
  useResource((dispose) => {
    const timer = setInterval(() => {
      setBlink(b => 1 - b);
    }, 1000);
    dispose(() => clearInterval(timer));
  });

  // Cube vertex data
  const cube = useOne(makeCube);
  const vertexBuffers = useMemo(() =>
    makeVertexBuffers(device, cube.vertices), [device]);

  // Rendering pipeline
  const pipeline = useMemo(() => {
    const pipelineDesc: GPURenderPipelineDescriptor = {
      primitive: {
        topology: "triangle-list",
        cullMode: "back",
      },
      vertex: makeShaderStage(
        device,
        makeShader(compileGLSL(vertexShader, 'vertex')),
        {buffers: cube.attributes}
      ),
      fragment: makeShaderStage(
        device,
        makeShader(compileGLSL(fragmentShader, 'fragment')),
        {targets: colorStates}
      ),
      depthStencil: depthStencilState,
    };
    return device.createRenderPipeline(pipelineDesc);
  }, [device, colorStates, depthStencilState]);

  // Uniforms
  const [uniformBuffer, uniformPipe, uniformBindGroup] = useMemo(() => {
    const uniformPipe = makeUniforms(defs);
    const uniformBuffer = makeUniformBuffer(device, uniformPipe.data);
    const entries = makeUniformBindings([{resource: {buffer: uniformBuffer}}]);
    const uniformBindGroup = device.createBindGroup({
      layout: pipeline.getBindGroupLayout(0),
      entries,
    });
    return ([uniformBuffer, uniformPipe, uniformBindGroup]
         as [GPUBuffer, UniformDefinition, GPUBindGroup]);
  }, [device, defs, pipeline]);

  // Return a lambda back to parent(s)
  return yeet((passEncoder: GPURenderPassEncoder) => {
    // Draw call
    uniformPipe.fill(uniforms);
    uploadBuffer(device, uniformBuffer, uniformPipe.data);

    passEncoder.setPipeline(pipeline);
    passEncoder.setBindGroup(0, uniformBindGroup);
    passEncoder.setVertexBuffer(0, vertexBuffers[0]);
    passEncoder.draw(cube.count, 1, 0, 0);
  });
}

This is 1 top-level function, with zero control flow, and a few hooks. The cube has a state (blink), that it decides to change on a timer. Here, useResource is like a sync useEffect which the runtime will manage for us. It's not pure, but very convenient.

All the external dependencies are hooked up, using the react-like useMemo hook and its mutant little brother useOne (for 0 or 1 dependency). This means if the WebGPU device were to change, every variable that depends on it will be re-created on the next render. The parts that do not (e.g. the raw cube data) will be reused.

This by itself is remarkable to me: to be able to granularly bust caches like this deep inside a program, written in purely imperative JS, that nevertheless is almost a pure declaration of intent. When you write code like this, you focus purely on construction, not on mutation. It also lets you use an imperative API directly, which is why I refer to this as "No API": the only wrappers are those which you want to add yourself.

Notice the part at the end: I'm not actually yeeting a real draw command. I'm just yeeting a lambda that will insert a draw command into a vanilla passEncoder from the WebGPU API. It's these lambdas which are reduced together in this sub-tree. These can then just be run in tree order to produce the associated render pass.

What's more, the only part of the entire draw call that actually changes regularly is the GPU uniform values. This is why uniforms is not an immutable object, but rather an immutable reference with mutable registers inside. In react-speak it's a ref, aka a pointer. This means if only the camera moves, the Cube component does not need to be re-evaluated. No lambda is re-yeeted, and nothing is re-reduced. The same code from before would keep working.

Therefor the entirety of Cube() is wrapped in a memo(...). It memoizes the entire Component in one go using all the values in props as the dependencies. If none of them changed, no need to do anything, because it cannot have any effect by construction. The run-time takes advantage of this by not re-evaluating any children of a successfully memoized node, unless its internal state changed.

The very top of the (reactive) part is:

export const App: LiveComponent<AppProps> = () => (props) => {
  const {canvas, device, adapter, compileGLSL} = props;

  return use(AutoCanvas)({
    canvas, device, adapter,
    render: (renderContext: CanvasRenderingContextGPU) => {

      const {
        width, height, gpuContext,
        colorStates, colorAttachments,
        depthStencilState, depthStencilAttachment,
      } = renderContext;

      return use(OrbitControls)({
        canvas,
        render: (radius: number, phi: number, theta: number) =>

          use(OrbitCamera)({
            canvas, width, height,
            radius, phi, theta,
            render: (defs: UniformAttribute[], uniforms: ViewUniforms) =>

              use(Draw)({
                device, gpuContext, colorAttachments,
                children: [

                  use(Pass)({
                    device, colorAttachments, depthStencilAttachment,
                    children: [

                      use(Cube)({device, colorStates, depthStencilState, compileGLSL, defs, uniforms}),

                    ]
                  })

                ],
              })
          })
      });
    }
  });
};

This is a poor man's JSX, but also not actually terrible. It may not look like much, but, pretty much everyone who's coded any GL, Vulkan, etc. has written a variation of this.

This tree composes things that are completely heterogeneous: a canvas auto-sizer, interactive controls, camera uniforms, frame buffer attachments, and more, into one neat, declarative structure. This is quite normal in React-land these days. The example above is static to keep things simple, but it doesn't need to be, that's the point.

The nicest part is that unlike in a traditional GPU renderer, it is trivial for it to know exactly when to re-paint the image or not. Even those mutable uniforms come from a Live component, the effects of which are tracked and reconciled: OrbitCamera takes mutable values and produces an immutable container ViewUniforms.

You get perfect battery-efficient sparse updates for free. It's actually more work to get it to render at a constant 60 fps, because for that you need the ability to independently re-evaluate a subtree during a requestAnimationFrame(). I had to explicitly add that to the run-time. It's around 1100 lines now, which I'm happy with.

Save The Environment

If it still seems annoying to have to pass variables like device into everything, there's the usual solution: context providers, aka environments, which act as invisible skip links across the tree:

export const GPUDeviceContext = makeContext();
export const App: LiveComponent<AppProps> = () => (props) => {
  const {canvas, device, adapter, compileGLSL} = props;

  return provide(GPUDeviceContext, device,
    use(AutoCanvas)({ /*...*/ })
  );
}

export const Cube: LiveComponent<CubeProps> = memo((fiber) => (props) => {
  const device = useContext(GPUDeviceContext);
  /* ... */
}

You also don't need to pass one variable at a time, you can pass arbitrary structs.

In this situation it is trickier for the run-time to track changes, because you may need to skip past a memo(…) parent that didn't change. But doable.

Yeet-reduce is also a generalization of the chunking and clustering processes of a modern compute-driven renderer. That's where I got it from anyway. Once you move that out, and make it a native op on the run-time, magic seems to happen.

This is remarkable to me because it shows you how you can wrap, componentize and memoize a completely foreign, non-reactive API, while making it sing and dance. You don't actually have to wrap and mount a <WebGPUThingComponent> for every WebGPUThing that exists, which is the popular thing to do. You don't need to do O(N) work to control the behavior of N foreign concepts. You just wrap the things that make your code more readable. The main thing something like React provides is a universal power tool for turning things off and on again: expansion, memoization and reconciliation of effects. Now you no longer need to import React and pretend to be playing DOM-jot either.

The only parts of the WebGPU API that I needed to build components for to pull this off, were the parts I actually wanted to compose things with. This glue is so minimal it may as well not be there: each of AutoSize, Canvas, Cube, Draw, OrbitCamera, OrbitControls and Pass is 1 reactive function with some hooks inside, most of them half a screen.

I do make use of some non-reactive WebGPU glue, e.g. to define and fill binary arrays with structured attributes. Those parts are unremarkable, but you gotta do it.

If I now generalize my Cube to a generic Mesh, I have the basic foundation of a fully declarative and incremental WebGPU toolkit, without any OO. The core components look the same as the ones you'd actually build for yourself on the outside. Its only selling point is a supernatural ability to get out of your way, which it learnt mainly from React. It doesn't do anything else. It's great when used to construct the outside of your program, i.e. the part that loads resources, creates workloads, spawns kernels, and so on. You can use yeet-reduce on the inside to collect lambdas for the more finicky stuff, and then hand the rest of the work off to traditional optimized code or a GPU. It doesn't need to solve all your problems, or even know what they are.

I should probably reiterate: this is not a substitute for typical data-level parallelism, where all your data is of the exact same type. Instead it's meant for composing highly heterogeneous things. You will still want to call out to more optimized code inside to do the heavy lifting. It's just a lot more straightforward to route.

For some reason, it is incredibly difficult to get this across. Yet algorithmically there is nothing here that hasn't been tried before. The main trick is just engineering these things from the point of view of the person who actually has to use it: give them the same tools you'd use on the inside. Don't force them to go through an interface if there doesn't need to be one.

The same can be said for React and Live, naturally. If you want to get nerdy about it, the reconciler can itself be modeled as a live effect. Its actions can themselves become regular nodes in the tree. If there were an actual dialect with a real live keyword, and WeakMaps on steroids, that would probably be doable. In the current implementation, it would just slow things down.

Throughout this series, I've used Javascript syntax as a lingua franca. Some might think it's insane to stretch the language to this point, when more powerful languages exist where effects fit more natively into the syntax and the runtime. I think it's better to provide stepping stones to actually get from here to there first.

I know that once you have gone through the trouble of building O(N2) lines of code for something, and they work, the prospect of rewriting all of them can seem totally insane. It probably won't be as optimized on the micro-level, which in some domains does actually still matter, even in this day and age. But how big is that N? It may actually be completely worth it, and it may not take remotely as long as you think.

As for me, all I had to do was completely change the way I structure all my code, and now I can finally start making proper diagrams.

Source code on GitLab.

July 04, 2021

Mijn vrouw, die bijna vijf jaar geleden de Boekenjagers oprichtte op Facebook, heeft zich naar aanleiding van verjaardag van die grootste FB boekengroep in Vlaanderen (meer dan 72.000 leden) nu ook in de wereld van het uitgeverij-vak gestort om het eerste Boekenjagers-boek uit te geven. Geloof me, als Veerle zich iets voorneemt, gaat ze er volledig voor en dat blijkt uit de indrukwekkende lijst van bijdrages van onder andere Lucinda Riley (!), Diane Broeckhoven, Toon Tellegen, Lize Spit, Patrick De Bruyn, Lectrr, Herr Seele, Boerken …

De pre-order fase loopt straks af, maar als je je haast kun je het boek nu nog aan €20 bestellen i.p.v. de €25 waarvoor het in augustus in de winkel zal liggen. Rep je dus naar https://boeken-jagers.be/boekenjagers-boek-pre-order/ en bestel er een paar, de opbrengst gaat naar 3 leesbevorderende goede doelen! :-)

July 03, 2021

Cover Image

Memoization

If you have a slow function slow(x) in your code, one way to speed it up is to memoize it: you cache the last result inside, so that if it's called again with the same x, you get the same result. If the function is static, this is equivalent to just storing the last input and output in a global variable. If it's dynamic, you can use e.g. a closure as the storage:

let memo = (func) => {
  let input = undefined;
  let output = undefined;
  return (x) => {
    // Return same output for same input
    if (x === input) return output;

    // Store new output for new input
    input = x;
    output = func(x);

    return output;
  }
};

let slow = memo((x) => { /*...*/ });

This can only buy you time in small code bases with very limited use cases. As soon as you alternate between e.g. slow(a) and slow(b), your cache is useless. The easy solution is to upgrade to a multi-valued cache, where you can retain the outputs for both a and b. Problem is, now you need to come up with an eviction policy to keep your cache from growing indefinitely. This is also assuming that a and b are immutable values that can be compared, and not e.g. mutable trees, or URLs whose remote content you don't even know.

In garbage collected languages, there is a built-in solution to this, in the form of WeakMap. This is a key/value map that can hold data, but which does not own its keys. This means any record inside will be garbage collected unless another part of the program is also holding on to the same key. For this to work, keys must be objects, not primitive values.

let memo = (func) => {
  let cache = new WeakMap();
  return (x) => {
    // Return same output for cached input
    let output = cache.get(x);
    if (output !== undefined) return output;

    // Store new output for new input
    output = func(x);
    cache.set(x, output);

    return output;
  }
};

let slow = memo((x) => { /*...*/ });

If a source object is removed elsewhere in the program, the associated cache disappears. It allows you to blindly hold on to an entire collection of derived data, while writing zero lines of code to invalidate any of it. Actual O(0) work. That is, assuming your xs are immutable. This is similar to what you get with reference counting, except WeakMap can also collect cycles.

Unfortunately, this only works for functions of one argument, because each WeakMap key must be one object. If you wish to memoize (x, y) => {...}, you'd need a WeakMap whose keys are xs, and whose values are WeakMaps whose keys are ys. This would only work well if y changes frequently but x does not.

I think this points to a very useful insight about caching: the reason it's so hard is because caches are normally one of the least informed parts of the entire system. A cache has zero contextual information to make any decisions with: it's the code that's accessing the cache that knows what's going on. If that code decides to drop a key or reference, the cache should always follow. Aside from that, how is it supposed to know?

If we are looking for a better solution for caching results, and resumability of code in general, then that's a pretty big clue. We are not looking for a better cache. We are looking for better ways to create context-specific storage. Then we use the simplest possible memoizer. This is 100% reliable. Stale data problems just go away even though you have hidden caches everywhere, it's wild.

Data Flow Graphs

At this point I should address the elephant in the room. If we're talking about declared computations with input dependencies and cached results, isn't this what data flow graphs are for? Today DFGs are still only used in certain niches. They are well suited for processing media like video or audio, or for designing shaders and other data processing pipelines. But as a general coding tool, they are barely used. Why?

We do have a zoo of various reactive, pure, stream-based, observable things... but almost all the successful ones work through code. Even with visual tools, much of their power comes from the ability to wire up custom scriptable nodes, which contain non-reactive, non-pure, non-observable code.

(x) => {
  a = A();
  b = B(x);
  return C(a, b);
}
data flow graph for the previous program

At first it seems like you can trivially take any piece of imperative code and represent it as a DFG. Each function call becomes a node in the graph, and local variables become the edges between function inputs and outputs.

You can also think of a function like B as a sub-graph in a box. It has 1 input and 1 output exposed to the outside in this case.

But there's an important difference between a node in a DFG and a function in a piece of code. In a DFG, each node is a call, not a definition. If you want to call B twice, then you need two unique B nodes, with unique instances of everything inside. You can't just point all the edges to and from the same node.

(x, y) => {
  a  = A();
  b1 = B(x);
  b2 = B(y);
  return C(a, b1, b2);
}
data flow graph for the previous program

That's because a DFG has no control flow (if/for/while) and no recursion. It represents a stateless computation, without a traditional stack to push and pop. At most you will have e.g. if and match nodes, which select only one of their inputs, as an optimization.

How a DFG is used depends primarily on whether you are editing a DFG or running a DFG. An example of editing a DFG is shader programming: the goal is to write code using a graphical tool. This code will be run on a GPU, applied to each pixel in parallel.

This means the DFG will be transformed into an abstract syntax tree (AST), and compiled down into regular shader code. It's never run directly in its DFG form. If you wish to inspect intermediate results, you need to make a truncated copy of the shader and then run only that.

It is worth noting that this sort of DFG tends to describe computation at a very granular level, i.e. one pixel or audio sample at time, at the very bottom of our program.

live data flow graph with values on the edges

This is all very different from running a live DFG. Here the nodes represent an actual data flow that is being computed. Each edge is a register containing a data value. The values are pushed or pulled through from top to bottom, with intermediate results cached and directly inspectable. This is so the graph can be minimally recomputed in response to changes.

These sorts of graphs tend to operate at the opposite end of the scale, connecting a small number of resources and nodes at the top. An individual edge often carries an entire image or even data set at once, to contain the complexity.

Even then, they get notoriously messy. One reason is that these models often don't have a way to represent a lambda. This causes spaghetti. Think about a piece of code such as:

square = (x) => x * x;
squared = call(square, number);
representing a lambda in a DFG

Here, square is not a definition, but a value, which we assign to a variable. We then pass it to the call function, to actually run the lambda. It is possible to represent this in a DFG, and some tools do support this.

We box in a sub-graph like before. However, we don't connect any of the inputs or outputs. Instead there is a special output â–¸, which represents a value of type Function, which you can connect to another function-valued input.

hooking up one lambda to two separate nodes

The question is what would happen if you connected the same lambda to two different calls.

This represents a stateless computation, so the two calls happen simultaneously, and need to pass along unique values. This means you need two copies of everything inside the lambda too. So in a live DFG, lambdas don't really provide a Function type but rather a FunctionOnce: a function you may only call once. This is a concept that exists in languages with a notion of ownership, like Rust. If you wish to call it more than once, you need to copy it.

This also means that this does not generalize to N elements. Take for example:

square = (x) => x * x;
numbers = [1, 2, 3, 4]
squared = map(square, numbers);

What would the inside of map look like?

implementing map in a DFG

It would need to split the items into individual variables, call the function square on each, and then join them back into a new array. If we add a number to numbers, we would need to add another column, so that we can call the lambda an additional time. If numbers shrinks, the reverse. The DFG must contain N multiply nodes, where N is data-driven and varies at run-time. This is an operation that simply does not exist in a typical DFG environment, at least not as a generic reusable op.

"Scriptable nodes" let you produce any output you like. But they don't let you write code to produce new nodes on the fly. That would mean returning a new piece of DFG instead of a value. You would likely want to attach something to the end so data can flow back to you. If that were possible, it would mean your graph's topology could change freely in response to the data that is flowing through it.

This would have all sorts of implications for maintaining the integrity of data flow. For example, you wouldn't be able to pull data from the bottom of a graph at all if some data at the top changed: you don't even know for sure what the eventual shape will be until you start re-evaluating it in the middle.

Managing a static DFG in memory is pretty easy, as you can analyze its topology once. But evaluating and re-constructing a DFG recursively on the fly is a very different job. If the graph can contain cycles, that's extra hard because now there isn't even a consistent direction of higher or lower anymore.

All this might sound like science fiction but actually it already exists, sort of, almost.

Reconciliation

Here's a simple mock program. We run it twice, once with argument true and once with false. I've lined up the two execution traces to show the matching calls:

let main = (x) => {
  A(x);
}

let A = (x) => {
  B(x);
  D(1);
}

let B = (x) => {
  let foo = x ? 3 : 2;
  if (x) B(false);
  C();
  if (x) D(0);
  D(foo);
}

let C = () => {};
let D = (x) => {};
comparing main(true) with main(false), call by call

None of these functions return anything, but let's just ignore that for now.

The tree shape shows the evolution of the stack over time. When entering a function, it reserves some space for local variables like foo. When exiting, that space is reclaimed. Any two sibling calls share the same position on the stack, overwriting each other's temporary data. A stack frame can only be identified by its position in the trace's tree, and only exists while a function is running.

Suppose then, that you do record such a trace at run-time. Then when the program is run the second time, you compare at every step, to see if it's making the same calls as before. That would help you with caching.

You can start by trivially matching the initial calls to A and B 1-to-1: only their argument differs. But once you enter B, things change. On the left you have the siblings B(false), C(), D(0), D(3) and on the right you have C(), D(2).

The actual changes are:

  • Remove B(false) and its sub-calls
  • Remove D(0)
  • Replace D(3) with D(2)

Figuring this out requires you to do a proper minimal diff. Even then, there is ambiguity, because the following would also work:

  • Remove B(false) and its sub-calls
  • Replace D(0) with D(2)
  • Remove D(3)

From the code you can tell that it should be the former, not the latter. But while this is easy to describe after the fact, it's difficult to imagine how such a thing might actually work at run-time. The only way to know what calls will be made is to run the code. Once you do, it's too late to try and use previous results to save time. Plus, this code only has a few trivial ifs. This becomes even harder if you allow e.g. for loops because now there's a variable number of elements.

In case it's not clear, this is the exact same problem as the live DFG with lambdas in disguise.

We need to introduce the operation of reconciliation: rather than doing work, functions like B must return some kind of data structure that describes the work to be done. Sort of like an effect. Then we can reconcile it with what it returned previously, and map 1-to-1 the calls that are identical. Then we can run it, while reusing cached results that are still valid.

It would also be useful to match calls where only arguments changed, because e.g. B(true) and B(false) share some of the same calls, which can be reused. At least in theory.

Granted, there is a huge constraint here, which the mock scenario obfuscates. Memoizing the calls only makes sense if they return values. But if we passed any returned value into another call, this would introduce a data dependency.

That is, in order to reconcile the following:

let F = (x) => {
  let foo = C(x);
  D(foo);
  if (foo) E();
}

We would need to somehow yield in the middle:

let F = function* (x) {
  let foo = C(x);
  yield;
  D(foo);
  if (foo) E();
}

That way we can first reconcile foo = C(x), so we can know whether to reconcile D(false) or D(true), E().

Unfortunately our language does not actually have a trace reconciler built into it, so this can't work. We have two options.

First, we can transpile the code to a deferred form:

let F = function* (x) {
  foo = yield call(C)(x)
  yield [
    call(D)(foo),
    foo ? call(E)() : null,
  ]);
}

Here, call(C)(x) is a value that says we want to call C with the argument x. A deferred function like F returns one or more wrapped calls via a yield. This allows C(x) to be reconciled, obtaining either a cached or fresh value for foo. Then we can reconcile the calls to D(foo) and E().

To make this work would require functions C, D and E to receive the exact same treatment, which we have to be honest, is not a compelling prospect.

Alternatively, we could recognize that C(x) is called first and unconditionally. It doesn't actually need to be reconciled: its presence is always guaranteed if F is called. Let's call such a function a hook.

let F = (x) => {
  let foo = C(x);
  return defer([
    call(D)(foo),
    foo ? call(E)() : null
  ]);
}

If hooks like C(x) aren't reconciled, they're regular function calls, so is all the code inside. Like a scriptable node in a DFG, it's an escape hatch inside the run-time.

But we're also still missing something: actual memoization. While we have the necessary information to reconcile calls across two different executions, we still don't have anywhere to store the memoized state.

So we'll need to reserve some state when we first call F. We'll put all the state inside something called a fiber. We can pass it in as a bound argument to F:

let F = (fiber) => (x) => {
  let foo = C(fiber)(x);
  return defer([
    call(D)(foo),
    foo ? call(E)() : null
  ]);
}

We also pass the fiber to hooks like C: this provides the perfect place for C to store a memoized value and its dependencies. If we run the program a second time and call this exact same F again, in the same place, it will receive the same fiber as before.

As long as the execution flow remains the same between two runs, the fiber and the memoized values inside will remain. Because functions like C are likely to receive exactly the same argument next time, memoization works very well here.

Apropos of nothing, here's how you build a UI component on the web these days:

const Component: React.FC<Props> = (props) => {
  const {foo, bar} = props;

  // These are hooks, which can only be called unconditionally
  const [state, setState] = useState('hello world');
  const memoized = useMemo(() => slow(foo), [foo]);

  // And there's also something called useEffect
  useEffect(() => {
    doThing(foo)
    return () => undoThing(foo);
  }, [foo]);

  // Regular JS code goes here
  // ...

  // This schedules a call to D({foo}) and E()
  // They are mounted in the tree inside <Component> recursively
  return <>
    <D foo={foo} />
    {foo ? <E /> : null}
  </>
}

It's all there. Though useEffect is a side-show: the real Effects are actually the <JSX> tags, which seem to have all the relevant qualities:

  • JSX doesn't run any actual code when we evaluate it
  • We can compose multiple JSX elements together in series (nesting) or in parallel (<> aka "fragments")
  • The run-time takes care of all the scheduling, mounting and clean up, in the right order

It will reconcile the before and after calls, and preserve the matching fibers, so functions like useMemo can work.

You can also reconcile variable size sets, by returning an array where every call has a key property. This allows minimal diffing in O(N) time.

You may eventually realize that the JSX at the bottom is really just an obscure dialect of JavaScript which lacks a real return statement: what this is returning is not a value at all. It is also not passed back to the parent component but to the run-time. The syntax is optimized for named rather than positional arguments, but that's about it.

What's more, if you do accept these constraints and manage to shoehorn your code into this form, it's strangely not terrible in practice. Often the opposite. Suddenly a lot of complex things that should be hard seem to just fall out naturally. You can actually begin to become 10x. Compared to O(N2) anyway.

The difference with our hypothetical trace reconciler is that there is no way to yield back to a parent during a deferred render. A common work-around in React land is a so-called render prop, whose value is a lambda. The lambda is called by the child during rendering, so it must be entirely side-effect free.

The code:

x = A();
y = B();
z = C(x, y);

must be turned into:

<A>{
  (x) => <B>{
    (y) => <C x={x} y={y}>{
      (z) => {}
    }</C>
  }</B>
}</A>

This is hideous. Because there is no ability for sibling calls to pass data, B must go inside A, or the other way around. This introduces a false data dependency. But, the data flow graph does match the normal execution trace: in code we also have to decide whether to call A or B first, even if it doesn't matter, unless you explicitly parallelize.

It's interesting that a render prop is an injectable lambda which returns new nodes to mount in the tree. Unlike a "scriptable node", this allows the tree to extend itself on the fly in a Turing-complete yet data-driven way.

So don't think of a reconciler as a tree differ. Think of it as a generalized factory which maintains a perfectly shaped tree of caches called fibers for you. You never need to manually init() or dispose() them... and if you re-run the code in a slightly different way, the fibers that can be re-used will be reused. The value proposition should be pretty clear.

When we made the fiber visible as an argument, much of the magic went away: a React Component is merely a function (fiber) => (props) => DeferredCall. The first argument is implicit, binding it to a unique, persistent fiber for its entire lifetime. The fiber is keyed off the actual call site and the depth in the stack. The hooks can work because they just reserve the next spot in the fiber as each one is called. This is why hooks must be called unconditionally in React: it's only way to keep the previous and next states in sync.

Where we go next is hopefully clear: what if you could use these patterns for things that aren't UI widgets in a tree? Could you retain not just the Effect-like nature, but also the memoization properties? Also, without hideous code? That would be pretty nice.

July 02, 2021

Cover Image

Declarative Code and Effects

This is a series about incrementalism in code and coding. On the one hand, I mean code that is rewindable and resumable. On the other, I mean incremental changes in how we code.

This is not an easy topic, because understanding some of the best solutions requires you to see the deep commonalities in code across different use cases. So I will deliberately start from basic principles and jump a few domains. Sorry, it's unavoidable.

Hopefully by the end, you will look differently at the code you write. If you don't, I hope it's because you already knew all this.

Hello World

If an abstraction is good enough to be adopted, it tends to make people forget why it was necessary to invent it in the first place. Declarative code is such an abstraction.

The meaning of "declarative code" is often defined through contrast: code that is not imperative or OO (aka "retained mode"). This is not very useful, because you can use declarative patterns all over imperative code, and get the exact same benefits. It can also be a great way to tame wild code you don't own, provided you can build the right glue.

OO however is a different story, and this applies just the same to OO-without-classes. Mind you, this has nothing to do with the typical endless debate between FP vs OO, which is the one about extensibility vs abstract data types. That's unrelated.

While seeing wide adoption in certain niches (e.g. UI), the foundational practices are often poorly understood. As a result, coders tend to go by example rather than principle: do whatever the existing code does. If they need to stray from the simple paths, they easily fall into legacy habits, resulting in poorly performing or broken code.

So far the best solution is to use linters to chide them for using things as they were originally intended. The better solution is to learn exactly when to say No yourself. The most important thing to understand is the anti-patterns that declarative code is supposed to remedy. Headache first, then aspirin.

a modernized version of dwarf fortress

For a perfect example, see the patch notes for almost any complex video game sandbox. They often contain curious, highly specific bugs:

  • Fix crouch staying on if used while a bear attacks
  • Fix player getting control-locked if loading an autosave created while lockpicking
  • Fix a bug which lets you immediately cancel a knockdown by swapping ammo types
  • Fixed an issue where leaving and returning after the briefing would block the quest's progress

Some combination of events or interactions corrupts the game's state in some way. Something changed when it shouldn't have, or the other way around. In the best case this results in a hilarious glitch, in the worst case a permanently unfinishable game. Developers can spend months chasing down weird bugs like this. It turns into whack-a-mole, as each new fix risks breaking other things.

This is very different from issues that occur when the game systems do exactly what they should do. Like that time cats were getting alcohol poisoning in Dwarf Fortress, because they had walked through puddles of spilled beer in the pub, and licked themselves clean afterwards. Such a complex chain of causality may be entirely unanticipated, but it is readily apparent why it happens. It is the result of code working too well.

Part of me dies every time I see a game get stuck in the mud instead, when I really want to see them succeed: I have a pretty good idea of the mess they've created for themselves, and why it's probably only going to get worse. Unless they dramatically refactor. Which usually isn't an option.

It's Imperative

So why does this happen?

Imagine an App with 5 views in tabs. Only one view is visible at a time, so there are 5 possible states. Because you can switch from any tab to any other, there are 5 x 4 = 20 possible state changes.

declarative vs imperative - for N states, there are N x (N - 1) possible state transitions

So you might imagine some code that goes:

constructor() {
  this.currentView = HomeView;
}

onSelect(newView) {
  // Hide selected view (exit old state)
  this.currentView.visible = false;
  // Show new view (enter new state)
  newView.visible = true;
  this.currentView = newView;
}

This will realize the arrows in the diagram for each pair of currentView and newView.

But wait. Design has decided that when you switch to the Promotions tab, it should start auto-playing a video. And it needs to stop playing if you switch away. Otherwise it will keep blaring in the background, which is terrible:

onSelect(newView) {
  if (this.currentView == newView) return;

  // Exit old state
  this.currentView.visible = false;
  if (this.currentView == PromoView) {
    PromoView.video.stop();
  }

  // Enter new state
  newView.visible = true;

  if (newView == PromoView) {
    PromoView.video.play();
  }

  this.currentView = newView;
}

No wait, they want it to keep playing in the background if you switch to the Social tab and back.

onSelect(newView) {
  if (this.currentView == newView) return;

  // Exit old state
  this.currentView.visible = false;
  if ((this.currentView == PromoView && newView != SocialView) ||
      (this.currentView == SocialView && newView != PromoView)) {
    PromoView.video.pause();
  }

  // Enter new state
  newView.visible = true;

  if (newView == PromoView) {
    PromoView.video.play();
  }

  currentView = newView;
}

Ok. Now they want to add a Radio tab with a podcast player, and they want it to auto play too. But only if the promo video isn't already playing.

Just look at how the if statements are popping out of the ground like mushrooms. This is the point at which things start to go terminally wrong. As features are piled on, a simple click handler explodes into a bloated mess. It is difficult to tell or test if it's even right, and there will likely be bugs and fixes. Replace the nouns and verbs, and you will get something very similar to what caused the curious bugs above.

The problem is that this code is written as delta-code (*). It describes state changes, rather than the states themselves. Because there are O(N2) arrows, there will potentially be O(N2) lines of code for dealing with N states. Each new feature or quirk you add will require more work than before, because it can interact with all the existing ones.

So imperative code isn't a problem by itself. The issue is code that becomes less maintainable over time. Ironically, a software industry notorious for snooty whiteboard interviews spent decades writing code like an O(N2) algorithm.

The real reason to write declarative code is to produce code that is O(N) lines instead. This work does not get harder over time. This is generally not taught in school because neither the students nor their teachers have spent enough time in one continuous codebase. You need to understand the O(N2) > O(N) part of the curve to know why it's so terrible, especially in the real world.

It probably seemed attractive in the first place, because it seemed economical to only touch the state that was changing. But this aversion to wasted CPU cycles has to be seriously traded off against the number of wasted human cycles.

Better to write code that makes it impossible to reach a bad state in the first place. Easier said than done, of course. But you should start with something like:

let tabs = [
  {
    view: homeView,
  }
  {
    view: promoView,
    video: 'promo-video.mp4',
    autoplayVideo: true,
  },
  {
    view: socialView, 
    backgroundVideo: true,
  },
  {
    view: newsView, 
    video: 'news-video.mp4',
    autoplayVideo: true,
  },
  // ...
]  

If you need to add a new kind of tab quirk, you define 1 new flag. You set it on a constant number of tabs, and then write the code to handle that quirk in 1 place. O(1) work. This code also tells you exactly where and what the special exceptions are, in one place, which means another engineer can actually understand it at a glance.

The declarative approach to visibility is just to constantly overwrite the state of every tab in a plain old loop. When somebody inevitably wants the tabs to animate in and out, you will discover that is actually what you have to do in the general case anyway.

Same with the video: you declare which one is currently supposed to be playing. Then you compare new with old, to see if you need to replace, pause, or adopt the current video.

The goal is to have most code only declare intent. Then you use other code to reach that target, no matter what the prior state was.

In some circles this is all old hat. But I suspect the people writing the hot frameworks don't quite realize how alien these problems feel from the outside. If the run-time is invisible, how are you ever supposed to figure out how it works, and learn to apply those same tricks? And how do you reach declarative bliss when there isn't a nice, prepackaged solution for your particular domain yet?

Or maybe there is a very capable library, but it's written in a retained style. If you try to fit this into declarative code without a proper adapter, you will end up writing delta-code all over again.

Strange Effects

The underlying challenge here is as banal as it is important and universal: "Have you tried turning it off and on again?"

Code is Turing-complete, so you don't know what it's going to do until you run it. It has the ability to amplify the complexity of its inputs. So if you change some of them, which parts are going to switch on and which will switch off?

Even on the back-end, similar things crop up behind the scenes:

  • connecting and disconnecting to network peers
  • starting and stopping jobs in response to events
  • building indices of data being edited
  • watching file systems for changes
  • allocating and freeing memory for data

These are all state transitions of some kind, which are usually orchestrated by hand.

Take the job of creating and managing a dependent resource inside a class. I swear I have seen this code written in dozens of codebases, each with dozens of files all doing the same thing, including one of my own:

constructor(size) {
  this.size = size;
  this.thing = new Thing(this.size);
}

onUpdate() {
  if (this.thing.size != this.size) {
    this.thing.dispose();
    this.thing = new Thing(this.size);
  }
}

dispose() {
  if (this.thing) this.thing.dispose();
}

This general pattern is:

constructor() {
  // Enter state - on
}

onUpdate() {
  // Exit old state - off
  // Enter new state - on
}

dispose() {
  // Exit state - off
}

You have to write code in 3 separate places in order to create, maintain and dispose of 1 thing. This code must also have access to this, in order to mutate it. Look around and you will see numerous variations of this pattern. Whenever someone has to associate data with a class instance whose lifecycle they do not fully control, you will likely spot something like this.

The trick to fix it is mainly just slicing the code differently, e.g. using a generator:

let effect1 = function* () {
  // Enter state - on
  yield // Wait
  // Exit state - off
}

let effect2 = function* () {
  // Enter state - on
  yield // Wait
  // Exit state - off
}

There are now only 2 identically shaped functions, each of which only refers to 1 state, not the previous nor next. A yield allows the function to be interrupted mid-call. The idea here is to simplify the lifecycle:

From

  • constructor()
  • onUpdate()
  • onUpdate()
  • onUpdate()
  • dispose()

To

  • Call effect
  • Cleanup effect
  • Call effect
  • Cleanup effect
  • Call effect
  • Cleanup effect
  • Call effect
  • Cleanup effect
  • Call effect
  • Cleanup effect

It doesn't have fewer calls, just fewer unique parts. It might seem dramatically less useful, but it's actually mostly the opposite. Though this is not obvious at first.

If you don't have a good mental model of a generator, you can pretend that instead it says:

let effect = () => {
  // Enter state - on
  return {
    // I'm not done yet
    done: false,
    // I don't have a value to return
    value: undefined,
    // Call me 🤙
    next: () => {
      // Exit state - off
      return {done: true, value: undefined}
    }
  };
}

This could actually be described as a self-rewinding effect. We can make a function that produces Thing resources this way:

// This creates an effect that describes the lifecycle of a Thing of size
let makeThingEffect = (size) => function* () {
  // Make thing
  thing = new Thing(size);

  yield thing;

  // Dispose thing
  thing.dispose();
};

So let's first talk about just effects as a formal type. They are similar to async/await and promises. But effects and promises are subtly different. A promise represents ongoing work. An effect is merely a description of work.

The difference is:

  • In a promise-based API, fetch(url) => Promise will run a new HTTP request
  • In an effect-based API, fetch(url) => Effect will describe an HTTP request

As a first approximation, you can think of an effect-based API as:

fetch(url) => () => Promise

It won't actually start until you call it a second time. Why would you need this?

Suppose you want to implement an auto-retry mechanism for failed requests. If a fetch promise fails, there is no way for you to retry using the promise itself. Each is one-use-only. You have to call fetch(url) again to get a fresh one. You need to know the specific type of promise and its arguments.

But if a fetch effect fails, then you can retry the same effect just by passing it back to your effect run-time. You don't need to know what effect it is. So in an Effect-based API, you can make a universal attemptRetry(effect, N) in a few lines. To emulate this with Promises, you need to use a () => Promise instead.

However, real Effects are supposed to be chained together, passing data from start to end. This is either a value or an error, to another effect, or back to the calling context. e.g.:

let makeAnEffect = (nextEffect) => function* (input, error) {
  if (!error) {
    output = f(input);
    return [nextEffect, output];
  }
  else return [null, null, error];
}

let combinedEffect = makeAnEffect( makeAnEffect( makeAnEffect() ) );

Here we return a static nextEffect that was decided on effect construction, along with a successful output. combinedEffect means running it 3 times in a row.

You could also return arbitrary effects on the fly. This is a retry combinator:

let attemptRetry = (effect, n) => function* (i, e) {
  // Make request
  const [value, error] = yield [effect, i, e];
  // Success
  if (value) return [null, value];
  // Retry n times
  if (n > 0) return retryEffect(effect, n - 1);
  // Abort
  return [null, null, error];
}

This is 1 function in 1 place you can use anywhere.

You can focus purely on defining and declaring intended effects, while letting a run-time figure out how to schedule and execute them. In essence, an Effect is a formalization of 🤙 aka (...) =>.

Whether effects are run serially or parallel depends on the use case, just like promises. Whether an effect should actually be disposed of or undone is also contextual: if it represents an active resource, then disposal is necessary to ensure a clean shutdown. But if an effect is part of a transaction, then you should only be rolling it back if the effect chain failed.

Other people have different definitions, and some of their Effects do different or fancier things. So buyer beware. But know that even just () => Promise, () => Generator and () => void can solve a whole bunch of issues elegantly.

Disposable

Going back to the makeThingEffect above, it might still seem overkill to wrap a "classic OO" class like new Thing this way. What's wrong with just having an interface Disposable that you implement/inherit from? Why should you want to box in OO code with effects even if you don't need to compose them? The difference is subtle.

For reliable operation, it is desirable that completion of one effect or disposal of a resource happens before the next one starts (i.e. to avoid double-allocating memory). But you can't call new on a class without immediately running a constructor. So you can't hold both an old and a new instance in your hands, so to speak, without having already initialized and run the new one. A common workaround is to have an empty constructor and delegate to an init() instead. This means some (or all) of your classes are non-standard to use, and you've reinvented the native new and delete.

Often you wish to retain some resources across a change, for efficiency, which means you need a hand-coded onChange() to orchestrate that. The main way you will take care of a series of possible creations and disposals is to just write new Thing and if (this.thing) and this.thing.foo != this.foo code repeatedly. That's why I've seen this code a thousand times.

While you can easily create a central mechanism for tracking disposals with classic OO, it is much harder to create a mechanism that handles both creation and updates. Somebody has to invoke specific constructors or reallocators, on specific classes, with specific arguments. This is called a Factory, and they pretty much suck.

It might seem like effects don't address this at all, as the pure form forbids any retention of state from one effect to the next, with each thing created in isolation. But it totally works: if every resource is wrapped in its own effect, you can declare the data dependencies per resource too. You can say "only re-allocate this resource if ____ or ____ changed since last time."

Instead of having an init() and an onChange() that you write by hand, you just have a render() method which spawns N resources and their dependencies. The run-time will figure out which ones are stale, clean them up, and create replacements.

How to actually do this depends on your usage. If the resource needs to be available immediately, that is different from whether it only needs to be returned to other parts of the code. In the latter case, you can just return effects which haven't run yet. In the former case, well, maybe we can fix that later.

"Effect run-time" sounds intimidating but 90% of the benefit here comes from giving yourself the ability to put your "exit" code below your "enter" code, where it belongs. Which ever way you can. Then there only needs to be 1 place in your code where "exit" appears above "enter" (i.e. an onUpdate), which you only have to write once. O(1).

At least that's the dream. I know I'm still handwaving a lot. It's entirely on purpose. In practice, unless you have a fixed structure to hang your effects and resources off of, it is actually quite hard to track all the associated state. You simply don't have anywhere specific to put it. Where does yield thing go, actually?

The potential benefits here are so enormous, that it's absolutely worth to figure this out.

More tomorrow.

(*) hat-tip James Crook for this term

June 30, 2021

Elf jaar geleden blogde ik over mijn Tide bestek. Bij deze een update.

Ik weet totaal niet waarom dit mij nu weer boeit, misschien omdat dit in de jaren 70 ons thuis bestek was? Toch zou ik graag de ontbrekende vleesvork, minivork en vismes hebben, wie wil ruilen?

Hier twee foto's van wat ik nu heb: (De inleg van de vleesvork, het kleinste vorkje en het vismes zijn copyright S.R.)

De geschiedenis van dit bestek: Het was te krijgen via premiebons die bij Tide waspoeder zaten in de jaren 1950 (en jaren 60?). Er bestaat een FB-groep voor Tide bestek, maar ik zit niet op FB. Veel kans hebben ze daar meer foto's en ruilmogelijkheden.

 

UPDATE 2021-05-20: Joepie 6 dessertvorkjes

UPDATE 2021-06-21: Danku Laura C. voor de vleesvork in uitstekende staat. Foto hieronder van deze grote Tide vork naast het grote mes (in mindere staat).


 

Nu enkel de vismesjes nog...

TOSCA is an OASIS open standard, and is an abbreviation for Topology and Orchestration Specification for Cloud Applications. It provides a domain-specific language to describe how an application should be deployed in the cloud (the topology), which and how many resources it needs, as well as tasks to run when certain events occur (the orchestration). When I initially came across this standard, I was (and still am) interested in how far this goes. The promise of declaring an application (and even bundling the necessary application artefacts) within a single asset and then using this asset to deploy on whatever cloud is very appealing to an architect. Especially in organizations that have a multi-cloud strategy.

June 25, 2021

Tentative d’analyse rationnelle et d’élargissement d’un débat émotionnel

Le sujet du port du voile par les femmes musulmanes est un sujet récurrent et politicomédiatique un peu trop vendeur. Forcément, car tout le monde à son avis sur le sujet. Mon avis personnel ne me semble pas avoir plus de valeur que n’importe quel autre.

Aussi ai-je envie d’aborder le sujet d’une manière que je n’ai que trop rarement vue : rationnellement et avec autant de rigueur logique que possible bien que n’étant, sensibilité humaine oblige, pas objectif. Le voile est en effet un exemple parfait pour illustrer le point de friction entre la liberté religieuse et la neutralité législative garante de cette liberté.

Le dress code professionnel

Il semble important de préciser que le débat ne porte pas sur l’interdiction du voile. Dans notre pays, le citoyen peut se vêtir comme bon il semble tant qu’une certaine pudeur est respectée. Chaque citoyen peut porter sur la tête le couvre-chef de son choix, le voile ne pose aucun problème sur la voie publique (je m’abstiendrai de parler du voile intégral, hors propos ici).

La question réelle peut donc être formulée en ces termes : « Si un employeur refuse que ses employés portent un voile, est-ce de la discrimination ? ».

Mise en contexte : les employeurs ont toujours disposé d’un droit de regard sur l’habillement de leurs employés. Depuis l’uniforme au port obligatoire de la cravate. Un voisin m’a un jour affirmé que jamais il n’accepterait un travail où on le forcerait de mettre une cravate. J’ai personnellement été une fois renvoyé chez moi pour me changer, car j’avais mis un bermuda, ce qui était interdit dans le règlement de travail (que je n’avais pas lu). Je fais pourtant partie de la sainte église du bermuda (2 membres en Belgique) pour laquelle le port du pantalon long entre mars et octobre est un blasphème.

Pour l’anecdote, dans cette même entreprise de plusieurs centaines de personnes est un jour arrivée une nouvelle employée voilée. Personne n’a émis le moindre commentaire (le voile n’était pas interdit dans le règlement) excepté une collègue qui m’a expliqué être musulmane et opposée au voile, voile qui n’était selon elle ni musulman ni cité dans le coran.

De même, dans le collège où j’ai fait mes études, le règlement stipulait strictement que les élèves étaient tenus d’être nu-tête à l’intérieur des bâtiments. J’ai été témoin de bon nombre de casquettes et chapeaux confisqués par les éducateurs.

Lors d’une embauche ou de l’accès à un établissement public, ces obligations vestimentaires clairement stipulées n’ont jamais été assimilées à une discrimination. Un postulant est libre de refuser un emploi si les conditions de travail ne lui conviennent pas.

Pourtant, le fait de devoir retirer le voile pour accéder à un travail ou à une école (chose que des milliers de musulmanes font quotidiennement, il faut le préciser, et qui serait acceptable selon les préceptes de l’Islam) est perçu par certaines comme une discrimination.

Je ne vois que deux alternatives logiques.

Soit le voile est un simple morceau de tissu vestimentaire, comme l’affirment certains, et il peut être retiré si nécessaire. J’abhorre la cravate, que j’associe à une laisse, mais je la passerai au cou si j’estime que les circonstances l’exigent.

Soit le port du voile est l’expression d’une religiosité profonde ne permettant pas à une croyante de le retirer en public. Auquel cas, il est manifeste que la personne ne peut accéder à un poste où une certaine forme de neutralité est nécessaire. Les postes de représentation publique, par exemple, impliquent une neutralité jusque dans les détails de l’habillement. Un député ne peut pas porter un t-shirt avec un slogan.

Il est important de souligner que les deux situations sont mutuellement exclusives. Soit le voile est un accessoire vestimentaire, auquel cas il peut-être retiré, soit il est le symbole d’une religiosité forte qui peut être considérée comme incompatible avec certaines fonctions, et cela à la discrétion de l’employeur.

Dans les deux cas, remarquons qu’il apparait raisonnable pour un employeur de ne pas engager une personne qui refuse de retirer son voile sur son lieu de travail alors que l’employeur l’estime nécessaire. Il n’y a pas de discrimination sur la couleur de peau, l’origine sociale ou la religion de l’employé, mais simplement un choix de ce dernier de se conformer ou non au règlement de travail. Ce que les plaignantes appellent discrimination dans les affaires où le port du voile leur a été refusé n’est donc que le refus de l’octroi d’un privilège spécifique qui n’est disponible pour personne d’autre.

L’exception religieuse

À cette constatation, la réponse la plus courante est celle de « l’exception religieuse ». L’employeur peut obliger la cravate et interdire le bermuda, car ceux-ci ne sont pas religieux. Le voile bien.

Cette exception est une véritable boîte de Pandore. Il est important de rappeler que la liberté de religion et de conscience garantie par l’article 18 des droits de l’homme porte sur toutes les religions, y compris les religions personnelles. Certains ont peut-être cru que je me moquais en parlant de l’église du bermuda, mais mon coreligionnaire (qui se reconnaitra), confirmera que notre foi est sincère et assortie de rituels (comme la photo du premier bermuda de l’année et l’expiation du port du pantalon long). Si l’état belge propose une liste de religions reconnues, c’est uniquement pour des motifs de financement. Chaque citoyen est libre de suivre les préceptes de la religion de son choix.

L’exception religieuse, par sa simple existence, sépare le corpus législatif en deux types de lois.

Les premières sont celles qui ne souffriront aucune exception religieuse. Si ma religion recommande la consommation de nouveau-nés au petit-déjeuner, l’exception religieuse sera difficilement recevable.

Par contre, mutiler le sexe du même nouveau-né sans raison médicale est couvert par l’exception religieuse.

Cette contradiction est visible dans tous les services communaux chargés d’établir nos cartes d’identité : il est en effet stipulé que la photo doit représenter le demandeur tête nue (point 8 du règlement), mais qu’en cas de motif religieux s’opposant à apparaitre tête nue sur nue sur la photo, ce point ne s’applique pas (point 10 du règlement).

Extrait du règlement téléchargé sur le site du gouvernement en juin 2021

Paradoxalement, la discrimination porte donc sur les personnes non religieuses. Celles-là ne peuvent pas porter de couvre-chef (et personne ne peut sourire, pourtant, ce serait joli une religion du sourire) sur leur photo d’identité ! La discrimination est bénigne et anecdotique, mais ouvre la porte à de dangereux précédents.

C’est pour illustrer ce paradoxe que plusieurs pastafariens à travers le monde ont tenté, avec plus ou moins de succès, de figurer sur leur document d’identité avec une passoire sur la tête ou avec un chapeau de pirate.

En Suède, face à la répression de la copie illicite de films et de musiques, le Kopimisme s’est mis en place. Selon cette religion, la vie nait de la copie d’information et tout acte de copie de l’information est sacré. En ce sens, les kopimistes ont exigé (et obtenu) une exemption religieuse leur permettant de copier tout document informatique sans être poursuivis. La loi « anti-piratage » fait donc partie, en Suède, de la deuxième catégorie des lois.

Ici encore, la discrimination envers les non-religieux est flagrante. Doit-on créer une religion pour tester et contourner chaque loi ?

Preuve est faite que l’exception religieuse est non seulement dangereuse, mais complètement inutile. Soit une loi ne peut accepter aucune exception religieuse (le meurtre par exemple), soit la loi n’a pas lieu d’être ou doit être retravaillée (vu qu’il est acceptable pour une frange arbitraire de citoyens de ne pas la respecter).

Soit il est essentiel d’être nu-tête sur un document d’identité, soit ce n’est pas nécessaire.

Soit un employeur peut imposer certaines règles vestimentaires à ses employés, soit il ne peut en imposer aucune. Toute tentative de compromis est, par essence, arbitraire et entrainera des débats émotionnels sans fin voire des violences.

Comme le disait déjà Thomas Hobbes, cité par le philosophe et député François De Smet dans son livre « Deus Casino », il est impossible à un humain d’obéir simultanément à deux autorités, deux ensembles distincts de lois (sauf si, par miracle, elles sont temporairement compatibles). L’un des ensembles de loi doit donc être supérieur à l’autre. Si c’est la loi religieuse qui est supérieure, on est tout simplement dans le cas d’une théocratie, chose qui n’est possible que pour une seule religion.

On peut tourner le problème dans tous les sens : la coexistence de plusieurs religions implique nécessairement la primauté des lois séculières sans distinction ni exception. Si une personne religieuse accomplit un rite qui enfreint une loi civile, elle doit être poursuivie comme après n’importe quelle infraction !

Cette conclusion, bien que contre-intuitive, est primordiale : le seul garant d’une réelle liberté de pensée et de culte passe par un état qui affirme que tous les citoyens sont égaux, que la loi est identique pour tous et ne tolère aucune exception religieuse !

L’expression de la croyance en une divinité et en la vérité d’un seul livre relève pour l’athée que je suis du blasphème et du non-respect envers les millions d’intellectuels qui ont fait progresser l’étendue du savoir humain. La religiosité m’offense profondément. Pour certaines femmes, le port du voile est une insulte aux féministes des décennies précédentes, un rappel permanent de la fragilité des récents et encore incomplets droits de la femme.

La liberté de culte passe donc nécessairement par la liberté de blasphème et le droit à l’offense.

Un combat politique à peine voilé

Rationnellement et logiquement parlant, la question du port du voile au travail est donc très simple à trancher : soit un employeur n’a aucun droit sur l’habillement de ses employés, soit il peut en avoir. Fin du débat.

Malheureusement, comme le souligne François De Smet, les religions instituées bénéficient d’une immunité contre l’irrationnel qui ne peut être justifiée par des arguments logiques. La religion est par essence irrationnelle, mais elle peut s’appuyer sur des arguments psychologiques, historiques, culturels ou politiques. Argument psychologique que j’ai intitulé « Le coût de la conviction ».

https://ploum.net/le-cout-de-la-conviction/

Le port du voile est souvent défendu par les traditionalistes comme un élément culturel et historique. Pourtant, en 1953 Nasser faisait exploser de rire son auditoire en ironisant sur l’impossibilité, n’en déplaise à une minorité, de forcer 10 millions d’Égyptiennes à porter le voile. Les photos de Téhéran dans les années 70 montrent également des femmes libérées, vêtues selon des normes modernes et se promenant en bikini au bord de la plage. Dans les années 90, je crois me souvenir qu’il était relativement rare de voir une femme voilée en Belgique, malgré plus de 30 ans d’immigration marocaine intense.

S’il y’a bien une certitude, c’est que l’interdiction pour une musulmane de se départir de son voile ne faisait, jusqu’à un passé relativement récent, peu ou plus partie du patrimoine culturel et qu’il n’est donc clairement pas « historique ». À l’opposé, la fameuse danse du ventre, tradition égyptienne millénaire, est en passe de disparaitre du pays, illustrant l’hypocrisie de l’argument « défense de la culture et de la tradition ».

La question qui est certainement la plus intéressante à se poser est donc : « Si ce n’est culturel, ni historique, ni rationnel, d’où vient cet engouement soudain pour le voile ? Qu’est-ce qui a fait apparaitre cette intransigeance récente qui pousse des femmes à refuser un emploi voire à attaquer l’employeur en justice plutôt que de retirer temporairement leur voile ou trouver un autre emploi ? ».

Les réponses à cette question pourraient aller de « Les immigrées n’osaient pas ne pas le retirer et osent enfin s’affirmer » à « Il s’agit d’un effet de mode, une pression sociale ». Ne pouvant répondre par une analyse logique, je laisserai le sujet aux sociologues.

À noter que dans un long billet très documenté, Marcel Sel propose une réponse essentiellement politique. Le port du voile ne serait pas une lutte pour les libertés individuelles, mais une volonté d’ingérence politique à peine voilée, si j’ose dire.

http://blog.marcelsel.com/2021/06/20/oh-bro-2-en-nommant-ihsane-haouach-la-belgique-met-la-charia-avant-la-meuf/

Le droit des femmes

Pour les millions de femmes dans le monde victime d’une théocratie et qui mettent en péril leur intégrité pour avoir le droit… de ne pas porter le voile, le combat de quelques-unes pour avoir le droit de le porter doit sembler incroyablement absurde. Mais, dans la vie quotidienne, il me semble important de laisser de côté les considérations politiques. Quand bien même le voile serait, à large échelle, un instrument politique, la femme en face de vous est avant tout un être humain le plus souvent sincère dans ses convictions.

Au risque d’enfoncer des portes ouvertes, j’aimerais rappeler le droit inaliénable au respect. Dans certains quartiers, des femmes se font insulter et se sentent en insécurité, car elles ne sont pas voilées. Autre part, ce sont les femmes voilées qui sont victimes d’insultes racistes voire d’agressions. Dans les deux cas, les femmes et la dignité humaine sont perdantes.

Quelles que soient leur religion et leur origine, beaucoup de femmes connaissent une insécurité permanente pour une raison unique : une culture machiste tolérée.

https://ploum.net/la-moitie-du-monde-qui-vit-dans-la-peur/

C’est là où, en tant qu’homme je peux agir. En surveillant mon comportement et celui de mes condisciples, en ne riant pas aux blagues sexistes, en sermonnant un camarade aux mains un peu trop baladeuses ou aux remarques trop sonores. En respectant totalement l’humaine qui est en face de moi, qu’elle se promène nue ou voilée de pied en cap.

Et en me gardant de faire étalage de manière inappropriée de mes convictions philosophiques. On n’a pas besoin d’être d’accord pour s’entendre et se respecter.

Si la relation entre les humains et la technologie vous intéresse, abonnez-vous par mail ou par RSS pour ne rater aucun billet (max 2 par semaine). Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, offrez et partagez ses livres.

Ce texte est publié sous la licence CC-By BE.

June 23, 2021

Software as a Service (SaaS) solutions are often a quick way to get new capabilities into an organization’s portfolio. Smaller SaaS solutions are simple, web-based solutions which barely integrate with the organization’s other solutions, besides the identity and access management (which is often handled by federated authentication).

More complex or intermediate solutions require more integration focus, and a whole new market of Integration Platform as a Service (iPaaS) solutions came up to facilitate cross-cloud integrations. But even without the iPaaS offerings, integrations are often a mandatory part to leverage the benefits of the newly activated SaaS solution.

In this post I want to bring some thoughts on the integrations that might be needed to support customizing a SaaS solution.

June 17, 2021

Bluetooth Low Energy (BLE) devices have two ways to transfer data:

  • Connectionless: This is called "broadcasting" or "advertising": the device advertises its existence, its capabilities and some data, and every BLE device in the neighbourhood can pick this up.

  • With a connection: This requires the client device to connect to the server device and ask for the data.

Until recently, ESPHome only supported reading BLE data without a connection. In my book, I gave some examples about how to use this functionality. I also referred to pull request #1177 by Ben Buxton, which was only merged in ESPHome 1.18.0, after the book was published.

In this blog post, I'll show you how to read the battery level and button presses from a Gigaset keeper Bluetooth tracker, as well as the heart rate of a heart rate sensor, for instance in your fitness tracker.

Note

These examples only work for ESP32 boards. The ESP8266 doesn't have Bluetooth Low Energy, and external BLE modules aren't supported.

Setting up a BLE client

If you want your ESPHome device to connect to another device using BLE, you first need to add a ble_client component, which requires an esp32_ble_tracker component. For instance:

esp32_ble_tracker:

ble_client:
  - mac_address: FF:EE:DD:CC:BB:AA
    id: gigaset_keeper

Just specify the BLE MAC address of the device you want to connect to, and give it an ID.

Tip

If you don't know the device's MAC address, just add the esp32_ble_tracker component and upload the firmware to your ESPHome device. It will start scanning for BLE devices and will show the devices it finds, with their name and MAC address. You can also scan with the mobile app nRF Connect for Android or iOS.

Reading a one-byte characteristic

After connecting to the device, you can create a BLE Client sensor to read one-byte characteristics such as the battery level from your device.

Each BLE server (the device you connect to) has various services and characteristics. The Bluetooth Special Interest Group has published a lot of standard services and their characteristics in their Specifications List. So if you want to read a device's battery level, consult the Battery Service 1.0 document. You see that it defines one characteristic, Battery Level, which "returns the current battery level as a percentage from 0% to 100%; 0% represents a battery that is fully discharged, 100% represents a battery that is fully charged."

Each service and characteristic has a UUID. The standard services and characteristics are defined in the Bluetooth SIG's Assigned Numbers specifications. These are 16-bit numbers. For instance, the Battery service has UUID 0x180f and the Battery Level characteristic has UUID 0x2a19. To read this in ESPHome, you add a sensor of the ble_client platform:

sensor:
  - platform: ble_client
    ble_client_id: gigaset_keeper
    name: "Gigaset keeper battery level"
    service_uuid: '180f'
    characteristic_uuid: '2a19'
    notify: true
    icon: 'mdi:battery'
    accuracy_decimals: 0
    unit_of_measurement: '%'

Make sure to refer in ble_client_id to the ID of the BLE client you defined before.

If you compile this, your ESPHome device connects to your Bluetooth tracker and subscribes to notifications for the battery level. 1

The Gigaset keeper has another one-byte characteristic that's easy to read, but it's not a standard one: the number of clicks on the button. By exploring the characteristics with nRF Connect, clicking the button and reading their values, you'll find the right one. You can then read this in ESPHome with:

sensor:
  - platform: ble_client
    ble_client_id: gigaset_keeper
    name: "Gigaset keeper button clicks"
    service_uuid: '6d696368-616c-206f-6c65-737a637a796b'
    characteristic_uuid: '66696c69-7020-726f-6d61-6e6f77736b69'
    notify: true
    accuracy_decimals: 0

Every time you click the button on the Gigaset keeper, you'll get an update with the click count. Of course you're probably not interested in the number of clicks, but just in the event that a click happens. You can act on the click with an on_value automation in the sensor, which can for instance toggle a switch in Home Assistant. This way you can use your Bluetooth tracker as a wireless button for anything you want.

Reading arbitrary characteristic values

In ESPHome 1.18 you could only read the first byte of a characteristic, and it would be converted to a float number. In ESPHome 1.19, David Kiliani contributed a nice pull request (#1851) that allows you to add a lambda function to parse the raw data. All received bytes of the characteristic are passed to the lambda as a variable x of type std::vector<uint8_t>. The function has to return a single float value.

You can use this for example to read the value of a heart rate sensor. If you read the specification of the Heart Rate Service, you'll see that the Heart Rate Measurement characteristic is more complex than just a number. There's a byte with some flags, the next one or two bytes is the heart rate, and then come some other bytes. So if you have access to the full raw bytes of the characteristic, you can read the heart rate like this:

sensor:
  - platform: ble_client
    ble_client_id: heart_rate_monitor
    id: heart_rate_measurement
    name: "${node_name} Heart rate measurement"
    service_uuid: '180d'  # Heart Rate Service
    characteristic_uuid: '2a37'  # Heart Rate Measurement
    notify: true
    lambda: |-
      uint16_t heart_rate_measurement = x[1];
      if (x[0] & 1) {
          heart_rate_measurement += (x[2] << 8);
      }
      return (float)heart_rate_measurement;
    icon: 'mdi:heart'
    unit_of_measurement: 'bpm'

Note how the heart rate is in the second byte (x[1]), and if the rightmost bit of x[0] is set, the third byte holds the most significant byte of the 16-bit heart rate value.

However, this way you can still only return numbers. What if you want to read a string? For instance, the device name is accessible in characteristic 0x2a00. Luckily, there's a trick. First define a template text sensor:

text_sensor:
  - platform: template
    name: "${node_name} heart rate sensor name"
    id: heart_rate_sensor_name

And then define a BLE client sensor that accesses the raw bytes of the Device Name characteristic, converts it to a string and publishes it to the template text sensor:

sensor:
  - platform: ble_client
    ble_client_id: heart_rate_monitor
    id: device_name
    service_uuid: '1800'  # Generic Access Profile
    characteristic_uuid: '2a00'  # Device Name
    lambda: |-
      std::string data_string(x.begin(), x.end());
      id(heart_rate_sensor_name).publish_state(data_string.c_str());
      return (float)x.size();

The only weirdness is that you need to return a float in the lambda, because it's a sensor that's expected to return a float value.

After someone on the Home Assistant forum asked how he could read the heart rate of his BLE heart rate monitor, I implemented all this in a project that displays the heart rate and device name of a BLE heart rate sensor on an M5Stack Core or LilyGO TTGO T-Display ESP32, my two go-to ESP32 boards. I published the source code on GitHub as koenvervloesem/ESPHome-Heart-Rate-Display.

This looks like this:

/images/esphome-heart-rate-display-m5stack-core.jpg

This is just one example of the many new possibilities with Bluetooth Low Energy in ESPHome 1.19.

1

Subscribing for notifications (with notify: true) uses less energy than continuously polling for a new value. With a notification, the server only pushes an update when the characteristic value changes.

June 15, 2021

PlatformIO supports the Digispark USB development board, a compact board with the ATtiny85 AVR microcontroller. The canonical example code that lets the built-in LED blink looks like this:

digispark_blink_platformio/main.c (Source)

/* Atml AVR native blink example for the Digispark
 *
 * Copyright (C) 2021 Koen Vervloesem (koen@vervloesem.eu)
 *
 * SPDX-License-Identifier: MIT
 */
#include <avr/io.h>
#include <util/delay.h>

// Digispark built-in LED
// Note: on some models the LED is connected to PB0
#define PIN_LED PB1
#define DELAY_MS 1000

int main(void) {
  // Initalize LED pin as output
  DDRB |= (1 << PIN_LED);

  while (1) {
    PORTB ^= (1 << PIN_LED);
    _delay_ms(DELAY_MS);
  }

  return 0;
}

Note

This doesn't use the Arduino framework, but directly uses the native AVR registers.

If you've bought your Digispark recently, the Micronucleus bootloader is a recent version that isn't supported by PlatformIO's older micronucleus command.

If you've already upgraded the bootloader on your Digispark, you also have the newest version of the micronucleus command. So the only thing you need is make PlatformIO use this version when uploading your code. You can do this with the following platformio.ini:

[env:digispark-tiny]
platform = atmelavr
board = digispark-tiny
upload_protocol = custom
upload_command = micronucleus --run .pio/build/digispark-tiny/firmware.hex

The platform and board options are just the configuration options the PlatformIO documentation lists for the Digispark USB. By setting the upload_protocol to custom, you can supply your own upload command, and the micronucleus in this command refers to the one you've installed globally with sudo make install in /usr/local/bin/micronucleus.

After this, you can just build the code and upload it:

koan@tux:~/digispark_blink_platformio$ pio run -t upload
Processing digispark-tiny (platform: atmelavr; board: digispark-tiny)
---------------------------------------------------------------------------------------------------------
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/atmelavr/digispark-tiny.html
PLATFORM: Atmel AVR (3.1.0) > Digispark USB
HARDWARE: ATTINY85 16MHz, 512B RAM, 5.87KB Flash
DEBUG: Current (simavr) On-board (simavr)
PACKAGES:
 - tool-avrdude 1.60300.200527 (6.3.0)
 - toolchain-atmelavr 1.50400.190710 (5.4.0)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 0 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Checking size .pio/build/digispark-tiny/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [          ]   0.0% (used 0 bytes from 512 bytes)
Flash: [          ]   1.4% (used 82 bytes from 6012 bytes)
Configuring upload protocol...
AVAILABLE: custom
CURRENT: upload_protocol = custom
Uploading .pio/build/digispark-tiny/firmware.hex
> Please plug in the device ...
> Device is found!
connecting: 33% complete
> Device has firmware version 2.5
> Device signature: 0x1e930b
> Available space for user applications: 6650 bytes
> Suggested sleep time between sending pages: 7ms
> Whole page count: 104  page size: 64
> Erase function sleep duration: 728ms
parsing: 50% complete
> Erasing the memory ...
erasing: 66% complete
> Starting to upload ...
writing: 83% complete
> Starting the user app ...
running: 100% complete
>> Micronucleus done. Thank you!
====================================== [SUCCESS] Took 3.99 seconds ======================================

I've created a GitHub project with this configuration, the example code, a Makefile and a GitHub Action to automatically check and build the code: koenvervloesem/digispark_blink_platformio. This can be used as a template for your own AVR code for the Digispark with PlatformIO.

June 14, 2021

The Digispark USB development board has an ATtiny85 microcontroller. (source: Digistump)

Recently I have been playing with the Digispark development board with the ATtiny85 AVR microcontroller. The cool thing is that it's only 2 by 2 cm big and it plugs right into a USB port.

But when I flashed the canonical blink example to the microcontroller, I noticed that the program didn't blink continuously with the fixed interval I had set up. The first five seconds after plugging in the Digispark the LED blinked in a fast heartbeat pattern (which seems to be normal, this is the bootloader staying in the programming mode for five seconds), then my program slowly blinked for a few seconds, then it was five seconds in heartbeat pattern again, then slower blinks again, and so on. It seemed the microcontroller got stuck in a continuous boot loop.

Thinking about the previous time I solved a microcontroller problem by upgrading the bootloader, I decided to try the same approach here. The Digispark is sold with the Micronucleus bootloader, but the boards I bought 1 apparently had an older version. Upgrading the bootloader is easy on Ubuntu.

First build the micronucleus command-line tool:

git clone https://github.com/micronucleus/micronucleus.git
cd micronucleus/commandline
sudo apt install libusb-dev
make
sudo make install

The Micronucleus project offers firmware files to upgrade older versions, so then upgrading the existing firmware was as easy as:

cd ..
micronucleus --run firmware/upgrades/upgrade-t85_default.hex

Then insert the Digispark to write the firmware.

After flashing the blink example, I didn't see the boot loop this time. And the Micronucleus developers apparently got rid of the annoying heartbeat pattern in the newer version.

1

Only after I received the 5-pack I realized that I had bought a Chinese clone. So it's possible that the original product sold by Digistump doesn't have the issue I'm describing here.

My current role within the company I work for is “domain architect”, part of the enterprise architects teams. The domain I am accountable for is “infrastructure”, which can be seen as a very broad one. Now, I’ve been maintaining an overview of our IT services before I reached that role, mainly from an elaborate interest in the subject, as well as to optimize my efficiency further.

Becoming a domain architect allows me to use the insights I’ve since gathered to try and give appropriate advice, but also now requires me to maintain a domain architecture. This structure is going to be the starting point of it, although it is not the true all and end all of what I would consider a domain architecture.

June 11, 2021

I just upped Autoptimize 2.9 beta to version 4, which is likely to be the last version before the official 2.9 release (eta end June/ early July).

Main new features;

You can download the beta from Github (do disable 2.8.x before activating 2.9-beta-4) and you can log any issues/ bugs over at https://github.com/futtta/autoptimize/issues

Looking forward to your feedback!

I published the following diary on isc.sans.edu: “Keeping an Eye on Dangerous Python Modules“:

With Python getting more and more popular, especially on Microsoft Operating systems, it’s common to find malicious Python scripts today. I already covered some of them in previous diaries. I like this language because it is very powerful: You can automate boring tasks in a few lines. It can be used for offensive as well as defensive purposes, and… it has a lot of 3rd party “modules” or libraries that extend its capabilities. For example, if you would like to use Python for forensics purposes, you can easily access the registry and extract data… [Read more]

The post [SANS ISC] Keeping an Eye on Dangerous Python Modules appeared first on /dev/random.

June 09, 2021

At my workplace, I jokingly refer to the three extra layers on top of the OSI network model as a way to describe the difficulties of discussions or cases. These three additional layers are Financial Layer, Politics Layer and Religion Layer, and the idea is that the higher up you go, the more challenging discussions will be.

June 08, 2021

Fastly had some issues today.

Here is a top 10 of the number of complainers on downdetector of some random internet services...

10. github 199
9. spotify 264
8. stackoverflow 351
7. twitter 368
6. discord 906
5. nytimes 2069
4. hbomax 2245
3. cnn 3432
2. twitch 5438
1. reddit 18628

Does this measure popularity? probably not.

Does it measure anything useful? Nope.