Planet Grep

Planet'ing Belgian FLOSS people

Planet Grep is maintained by Wouter Verhelst. All times are in UTC.

November 29, 2022

More and more people are requesting how they could connect to MySQL without using a password but using a SSL certificate. Known as X509.

CA Certificate

A CA certificate is a digital certificate issued by a certificate authority (CA). It’s used by clients to verify the SSL certificates sign by this CA.

Such certificates is usually paid and needs to be manually installed with MySQL Server. But by default, MySQL generates a self-signed certificate and provides its own CA.

For obvious reason, I will use the certificates that have been auto-generated by MySQL on my system. However, for production, I encourage you to have a real CA signed certificate.

The CA certificate is called ca.pem and is located in MySQL’s datadir (/var/lib/mysql/ca.pem on Oracle Linux, RHEL, Fedora, CentOS, …).

In case you don’t know where your ca.pem is located, you can check in the global variables of your MySQL Server:

SQL> select * from performance_schema.global_variables
     where variable_name like 'ssl_ca';
+---------------+----------------+
| VARIABLE_NAME | VARIABLE_VALUE |
+---------------+----------------+
| ssl_ca        | ca.pem         |
+---------------+----------------+

The server and the client must use the same CA.

Client Key

MySQL also generates a client key (client-key.pem) but we will generate one per client.

We need of course to use openSSL to generates and verify our certificates.

We start by creating a client certificate, remove the passphrase and sign it:

$ openssl req -newkey rsa:2048 -days 365 -nodes -keyout user1-key.pem -out user1-req.pem
Ignoring -days without -x509; not generating a certificate
..........+...+..... and plenty others ;-)....++++++...
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:BE
State or Province Name (full name) []:Hainaut
Locality Name (eg, city) [Default City]:Maurage
Organization Name (eg, company) [Default Company Ltd]:MySQL
Organizational Unit Name (eg, section) []:Community
Common Name (eg, your name or your server's hostname) []:user1
Email Address []:user1@oracle.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

You have now to new files:

  • user1-key.pem – the user’s private key
  • user1-req.pem – the user’s PEM certificate request

And now we need to generate the public key (x509 certificate) that will also use the same CA certificates of MySQL. I will generate a certificate valid for 1 year (365 days):

$ sudo openssl x509 -req -in user1-req.pem -days 365 -CA /var/lib/mysql/ca.pem \
     -CAkey /var/lib/mysql/ca-key.pem -set_serial 01 -out user1-cert.pem
Certificate request self-signature ok
subject=C = BE, ST = Hainaut, L = Maurage, O = MySQL, OU = Community,
        CN = user1, emailAddress = user1@oracle.com

The “subject” output is very important as this is what we will use in MySQL for the user credentials.

Verifying the Certificate

We can now already verify the certificate we generated:

$ openssl verify -CAfile /var/lib/mysql/ca.pem /var/lib/mysql/server-cert.pem \
  user1-cert.pem 
/var/lib/mysql/server-cert.pem: OK
user1-cert.pem: OK

MySQL User Creation

We need to create a MySQL user that will use the certificate. By default, with the loaded password policy, we also need to provide a password:

SQL> CREATE USER user1 IDENTIFIED BY 'Passw0rd!' REQUIRE 
     SUBJECT '/C=BE/ST=Hainaut/L=Maurage/O=MySQL/OU=Community/CN=user1/emailAddress=user1@oracle.com';
Query OK, 0 rows affected (0.0114 sec)

If we don’t want to use a password but only the certificate, it’s possible the remove “identified by ‘Passw0rd!’, but you need to uninstall the component validate_password and re-install it just after.

UNINSTALL COMPONENT 'file://component_validate_password';

Even with the privilege PASSWORDLESS_USER_ADMIN, if the component is installed, the password must comply with the policy.

Pay attention that it’s recommended to also specify the “issuer” of the certificate like CREATE USER user1 REQUIRE ISSUER '/C=BE/ST=Bruxelles/L=Bruxelles/ O=MySQL/CN=CA/emailAddress=ca@oracle.com' SUBJECT '/C=BE/ST=Hainaut/L=Maurage/O=MySQL/OU=Community/CN=user1/emailAddress=user1@oracle.com';

Connecting

We can now connect using the certificate and the key:

The same certificate and key can be used in MySQL Shell for Visual Studio Code:

Conclusion

It’s possible to use X509 certificates (self-signed or not) to connect to MySQL. With or without a password. This method is working with the old mysql client, with MySQL Shell for classic and X protocol and also with MySQL Shell for Visual Studio Code.

You can find more information related to MySQL and Encrypted Connections in the manual.

Enjoy MySQL !

November 28, 2022

Couverture de « Stagiaire au spatioport Omega 3000… et autres joyeusetés que nous réserve le futur »

Pourriez-vous devenir le premier Madame pipi mâle de la station spatiale Omega 3000 ? Ou optimiser le rendement des mines de chocolat de la Lune ? La vie privée étant abolie, percerez-vous l’identité secrète de l’homme le plus riche du monde ? Comment lutter contre les monopoles informatiques si, lassée de vous voir taper à la machine, votre famille vous inscrit à une initiation aux ordinateurs ? Jouerez-vous un rôle majeur dans le destin de la galaxie ou resterez-vous un figurant ?

Toutes les réponses à ces questions (et à bien d’autres) sont désormais disponibles dans « Stagiaire au spatioport Omega3000 et autres joyeusetés que nous réserve le futur », un recueil de nouvelles désormais disponibles en ligne et dans toutes les librairies de Suisse. Il arrivera dans celles de France et de Belgique en février 2023.

Ce qui est un peu tard pour les cadeaux de Noël/Newtonmass, raison pour laquelle vous pouvez directement commander ce recueil chez l’éditeur.

=> Commander « Stagiaire au spatioport Omega 3000 et autres joyeusetés que nous réserve le futur »

« Stagiaire au spatioport Omega 3000 » est une idée cadeau idéale, car, contrairement à un roman, qu’on aime ou qu’on n’aime pas, le livre offre ici 15 histoires très différentes. Certaines plus farfelues, d’autres sérieuses. Des rigolotes, des absurdes ou des interpelantes voire tout simplement poétiques. L’une évoque la problématique du genre dans le cadre du space opera ( « Stagiaire au spatioport Omega 3000 »). D’autres alertent sur l’emprise des monopoles informatiques ( « Le dernier espoir »), la disparition de la vie privée en ligne ( « Le jour où la transparence se fit » ) ou l’impact à très long terme de nos choix technologiques ( « Les successeurs » ).

En (vous) offrant ce recueil, vous offrez donc des moments de plaisir, de rire et de poésie, mais également, sans en avoir l’air, des pistes de réflexion et des introductions à des sujets potentiellement difficiles que vous, lecteurs de mon blog, vous connaissez probablement déjà.

Autour de la bûche de Noël, rien que le titre et la couleur de la couverture devraient occuper une bonne partie de la soirée et détourner un bon moment les conversations de la coupe du monde au Qatar, de la crise économique et de la guerre en Ukraine. Avouez que, à ce prix là, c’est donné !

Alors, plutôt que de parcourir les centres commerciaux surchauffés, offrir 15 nouvelles est une idée de cadeau rapide, chic et pas cher !

Pour ceux dont la liste de lecture peut attendre février, commandez le livre dès maintenant chez votre libraire. On ne se rend compte de l’importance des librairies que lorsqu’on les perd, soutenez-les ! L’ISBN est 978-2-940609-29-1.

=> Fiche du livre sur Place des libraires

Plus qu’un simple recueil…

Depuis ma plus tendre enfance, je dévore les recueils de nouvelles. J’adore quand les nouvelles sont entrecoupées d’anecdotes par l’auteur. Ce que fait Isaac Asimov avec un talent incroyable.

Lorsque Lionel, mon éditeur homonyme, m’a proposé de publier un recueil de nouvelles, j’ai d’abord pensé à les rassembler de manière traditionnelle, n’osant même pas tenter d’imiter le grand Asimov. Mon épouse m’a convaincu d’écouter mon intuition et de faire ce recueil avant tout pour moi, comme je voudrais le lire.

Donc acte. Chaque nouvelle est désormais accompagnée d’une petite note où j’explique l’inspiration et le processus d’écriture derrière le texte. Parfois, je digresse un peu sur les thèmes qui me sont chers. Vous me connaissez, on ne se refait pas…

Le résultat est que loin d’être juste un assemblage de texte, ce recueil est devenu une forme de mise à nu, un partage très intime entre l’écrivain et chaque lect·eur·rice. Avec mon éditeur, nous avons pris la décision d’inclure également quelques « erreurs de jeunesse ». Ce ne sont pas mes meilleurs textes, mais rendre transparente mon évolution personnelle est une manière d’illustrer mon travail et, je l’espère, d’inspirer d’autres à apprécier leurs propres progrès. Pour tout avouer, je n’ose pas me relire, je suis un peu gêné de ce que vous allez découvrir de moi. Tout en étant très fier d’offrir un recueil qui est bien plus que la somme des textes qui le composent.

Si vous lisez ce blog, ce recueil est ce qui s’en rapproche le plus au format papier. Tout en étant bien plus amusant et gai à lire. Le partager et le recommander est la plus belle manière de soutenir mon travail.

=> Commander « Stagiaire au spatioport Omega 3000 et autres joyeusetés que nous réserve le futur » (format papier)
=> Le recueil au format epub

Je serai très heureux d’avoir vos avis, vos réactions, de lire vos critiques sur vos blogs ou vos espaces en ligne respectifs. N’hésitez pas à m’envoyer vos retours. Sur Mastodon, je vous propose d’utiliser le hashtag #omega3000.

Et si vous avez découvert la surprise (qui est, si Wikipédia est exact, une première mondiale), chut ! Ne la spoilez pas pour les autres…

Recevez les billets par mail ou par RSS. Max 2 billets par semaine, rien d’autre. Adresse email jamais partagée et définitivement effacée lors du désabonnement. Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, lisez, offrez et partagez des livres.

Ce texte est publié sous la licence CC-By BE.

November 20, 2022

Today I upgraded my home computers to Fedora 37.

I’m using some software that don’t have any rpm available for Fedora 37.

Some of these applications, I used them every day and I couldn’t work on my machine without them. Hamster is one of them, I use it for time tracking. However the project seems to not be maintained for some time. I use it with the Gnome-Shell extension too.

I also regularly use sysbench compiled with libmysql from MySQL.

The last package is the one of an application I blogged recently, Projecteur.

If you are also using those applications or if you want to try them on Fedora 37, here are the packages I made for my computers:

Enjoy !

November 14, 2022

Vraag je je ook weleens af welke vogelsoorten er allemaal je tuin bezoeken? Met een Raspberry Pi en een microfoon kun je de hele dag het geluid in je tuin opnemen. De software BirdNET-Pi herkent daarin vogeldeuntjes en toont je in een webinterface handige statistieken van wanneer welke vogels te horen zijn.

/images/birdnetpi-overzicht.png

BirdNET is een app voor Android en iOS die vogelsoorten kan onderscheiden aan de hand van geluidsopnames. Ideaal als je tijdens een wandeling je afvraagt welke vogel dat speciale deuntje laat horen. Onder de motorkap draait een neuraal netwerk van onderzoekers van Cornell University dat op geluiden van drieduizend vogelsoorten is getraind. Je neemt enkele seconden van het deuntje op je telefoon op, de app analyseert die vervolgens en benoemt daarna de vogels waarvan het geluid daar het meest op lijkt.

Luistervinkende Raspberry Pi

Patrick McGuire heeft met BirdNET-Pi een versie van BirdNET gemaakt die op de Raspberry Pi werkt. De software luistert continu naar het geluid van een usb-microfoon en herkent daarin realtime vogelgeluiden. Dat werkt op een Raspberry Pi 4B, 3B+ en Zero W 2. Om van alle mogelijkheden te kunnen genieten, is een Raspberry Pi 4B wel aangeraden.

Als je BirdNET ook als app op je telefoon kunt draaien, wat is dan het voordeel van BirdNET-Pi? Het belangrijkste pluspunt is dat BirdNET-Pi continu luistert, waardoor je het de klok rond vogelgeluiden kunt laten herkennen. Voor vogelliefhebbers levert dat ook interessante statistieken op, zoals hoe laat je het meeste kans maakt om specifieke vogelsoorten te spotten.

/images/birdnetpi-statistieken.png

Een ander voordeel is dat BirdNET-Pi zijn analyses volledig offline doet, terwijl de smartphone-app elke audio-opname naar de servers van het BirdNET-project moet doorsturen om daar de analyse uit te voeren. Je zou dus een Raspberry Pi met BirdNET-Pi op een plaats zonder netwerktoegang kunnen installeren en na een dag de opgeslagen detecties kunnen raadplegen.

In een artikel van me op id.nl lees je hoe je BirdNET-Pi op een Raspberry Pi installeert en configureert.

Ce samedi 19 novembre, à 11h30, je donnerai une conférence à Toulouse au Capitole du Libre sur le thème :

Attention ! Cette conréfence n’est pas une conréfence sur le cyclimse ! Merci de votre compréhension !

Ça parlera de culture, de libre, de culture libre, de liberté de culture et de l’importance d’échapper à la monoculture, de libérer nos cerveaux en libérant nos références culturelles.

J’en profiterai pour annoncer quelques bonnes surprises dont je n’ai pas encore eu l’occasion de parler sur ce blog.

Après la conf, je resterai jusqu’au dimanche après-midi, assis à une table entre Pouhiou et David Revoy, excusez du peu, pour dédicacer mon roman Printeurs et, première surprise, mon nouveau recueil de nouvelles qui n’est officiellement pas encore sorti.

Quand je dis « pas encore sorti », je veux dire que même moi je ne l’ai pas encore vu. Mon éditeur a envoyé une poignée des tout premiers exemplaires imprimés directement à Toulouse. Je le découvrirai donc devant vous à la table de dédicace. Une exclusivité, comme le cassoulet, 100% toulousaine. Je peux cependant déjà vous dire que si vous êtes partisan du respect de la vie privée, si vous pestez contre l’invasion des smartphones ou que vous vous êtes déjà retrouvé face à un extra-terrestre gluant dans des toilettes qui ne correspondaient pas à votre sexe, vous devriez trouver votre bonheur dans ce recueil qui devrait arriver dans toutes les librairies françaises, belges et suisses en janvier.

Si vous êtes dans le coin, n’hésitez pas à venir faire un tour et, si le cœur vous en dit, à m’aborder.

J’ai découvert aux Imaginales à Épinal que plusieurs lecteurs hésitaient à m’aborder pour « ne pas me déranger ». J’avoue que je déteste être dérangé quand je suis concentré, mais si je vais à une conférence, c’est précisément pour rencontrer des gens. Donc pour être dérangé ! Et vous n’avez pas à vous justifier de ne pas acheter mes livres. D’ailleurs, les ventes seront assurées par un libraire professionnel.

Pour ceux qui préfèrent les versions électroniques, mais souhaitent malgré tout une dédicace, je vous propose de venir avec le morceau de papier/carton de votre choix. Je le glisserai dans ma machine à écrire pour vous faire un petit mot.

Parce qu’au fond de moi, ça me fait sacrément plaisir de retrouver une conf libriste, de me retrouver au milieu de geeks et geekettes en t-shirts déglingués, de discuter de vive voix après parfois près de vingt années d’échanges sur le net, de m’associer à d’autres Vimeurs pour troller les utilisateurs d’Emacs, de parler de Gemini et de naviguer en ligne de commande avec des gens qui trouvent ça normal (je peux vous faire une démo d’Offpunk, mon navigateur web offline), de vous donner des nouvelles de ma boulangère …

Bref, je suis sacrément content de vous (re)voir !

(non je ne suis pas malade en voiture. C’est juste que quand je suis content je vomis. Et là je suis hypercontent !)

Recevez les billets par mail ou par RSS. Max 2 billets par semaine, rien d’autre. Adresse email jamais partagée et définitivement effacée lors du désabonnement. Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, lisez, offrez et partagez des livres.

Ce texte est publié sous la licence CC-By BE.

November 12, 2022

We now invite proposals for presentations. FOSDEM offers open source and free software developers a place to meet, share ideas and collaborate. Renowned for being highly developer-oriented, the event brings together some 8000+ geeks from all over the world. The twenty-third edition will take place on Saturday 4th and Sunday 5th February 2023 at the usual location, ULB Campus Solbosch in Brussels. It will also be possible to participate online. Developer Rooms For more details about the Developer Rooms, please refer to the Calls for Papers that are being added to https://fosdem.org/2023/news/2022-11-07-accepted-developer-rooms/ when they are issued. Main Tracks Main舰

The Debian Videoteam has been sprinting in Cape Town, South Africa -- mostly because with Stefano here for a few months, four of us (Jonathan, Kyle, Stefano, and myself) actually are in the country on a regular basis. In addition to that, two more members of the team (Nicolas and Louis-Philippe) are joining the sprint remotely (from Paris and Montreal).

Videoteam sprint

(Kyle and Stefano working on things, with me behind the camera and Jonathan busy elsewhere.)

We've made loads of progress! Some highlights:

  • We did a lot of triaging of outstanding bugs and merge requests against our ansible repository. Stale issues were closed, merge requests have been merged (or closed when they weren't relevant anymore), and new issues that we found while working on them were fixed. We also improved our test coverage for some of our ansible roles, and modernized as well as improved the way our documentation is built. (Louis-Philippe, Stefano, Kyle, Wouter, Nicolas)
  • Some work was done on SReview, our video review and transcode tool: I fixed up the metadata export code and did some other backend work, while Stefano worked a bit on the frontend, bringing it up to date to use bootstrap 4, and adding client-side filtering using vue. Future work on this will allow editing various things from the webinterface -- currently that requires issuing SQL commands directly. (Wouter and Stefano)
  • Jonathan explored new features in OBS. We've been using OBS for our "loopy" setup since DebConf20, which is used for the slightly more interactive sponsor loop that is shown in between talks. The result is that we'll be able to simplify and improve that setup in future (mini)DebConf instances. (Jonathan)
  • Kyle had a look at options for capturing hardware. We currently use Opsis boards, but they are not an ideal solution, and we are exploring alternatives. (Kyle)
  • Some package uploads happened! libmedia-convert-perl will now (hopefully) migrate to testing; and if all goes well, a new version of SReview will be available in unstable soon.

The sprint isn't over yet (we're continuing until Sunday), but loads of things have already happened. Stay tuned!

November 11, 2022

Hello dear MySQL Community !

As you may already know FOSDEM 2023 is again going to be held in-person. FOSDEM will take place February 4th and 5th 2023.

We have also decided to put our pre-FOSDEM MySQL Day on track for a fifth edition.

As for the last edition, the event will be spread over 2 days.

These 2 extra days related to the world’s most popular open source database will take place just before FOSDEM, the 2nd and 3rd February at the usual location in Brussels.

Please don’t forget to register as soon as possible as you may already know, the seats are limited !

Register on eventbrite: https://mysqldays2023.eventbrite.com

And, please don’t forget, that if you have register for the event and you cannot make it, please free back your ticket for somebody else.

Sessions

The agenda is not yet determined and you may have noticed that the MySQL Devroom during FOSDEM will be very short with a very limited amount of sessions.

Therefore, we will certainly include in the pre-FOSDEM agenda some of the sessions that will not appear in the Devroom agenda.

Also, if you would like to propose a session exclusively for the pre-FOSDEM MySQL Days, as the duration of the sessions is more flexible, please do not hesitate to contact me.

I am very happy to see you all again !

November 10, 2022

I published the following diary on isc.sans.edu: “Do you collect “Observables” or “IOCs”?“:

Indicators of Compromise, or IOCs, are key elements in blue team activities. IOCs are mainly small pieces of technical information that have been collected during investigations, threat hunting activities or malware analysis. About the last example, the malware analyst’s goal is identify how the malware is behaving and how to indentify it.

Most common IOCs are… [Read more]

The post [SANS ISC] Do you collect “Observables” or “IOCs”? appeared first on /dev/random.

Next week, November 16th, I will participate to the MySQL Innovation and Cloud Virtual Day in French.

My colleagues will present what’s new in MySQL 8.0.31 and also summarize all the big news that was announced at Oracle Cloud World in Las Vegas.

Attendees will learn about the MySQL HeatWave offering in OCI.

I will be presenting something that is only available in MySQL on-prem and in OCI as a managed service: MySQL Document Store.

The event is in French and attendees will have the opportunity to discuss and chat with MySQL experts (included Olivier!!) during the event.

Registration is required to attend this free event: Register Here.

On se voit la semaine prochaine !

November 09, 2022

I published the following diary on isc.sans.edu: “Another Script-Based Ransomware“:

In the past, I already found some script-based ransomware samples written in Python or Powershell. The last one I found was only a “proof-of-concept” (my guess) but it demonstrates how easy such malware can be developed and how they remain undetected by most antivirus products.
I found a malicious VisualBasic script that attracted my attention. The SHA256 is 8c8ed4631248343f8732a83193828471e005900fbaf144589d57f6900b9c8996 and its VT score is only 3/57!. It’s no flagged as malicious but, even more, it’s reported as a simple mallicious script… [Read more]

The post [SANS ISC] Another Script-Based Ransomware appeared first on /dev/random.

November 07, 2022

Recently, I wrote three articles on how to analyze queries and generate a slow query log for MySQL Database Service on OCI:

In these post, we were generating a slow query log in text or JSON directly in Object Storage.

Today, we will see how we can generate a slow query log in text directly using MySQL Shell and form Performance Schema.

The generated log can be used to digest the queries with a tool like pt-query-digest.

The MySQL Plugin used is logs and is available on my GitHub repo dedicated to my MySQL Shell Plugins.

This is an example of an output using pt-query-digest:

# Query 3: 0.40 QPS, 0.00x concurrency, ID 0xF70E8D59DF2D27CB4C081956264E69AB at byte 104128
# This item is included in the report because it matches --limit.
# Scores: V/M = 0.01
# Time range: 2022-11-06T23:12:14 to 2022-11-06T23:12:19
# Attribute    pct   total     min     max     avg     95%  stddev  median
# ============ === ======= ======= ======= ======= ======= ======= =======
# Count          1       2
# Exec time      8    13ms    87us    13ms     6ms    13ms     9ms     6ms
# Lock time      1     4us       0     4us     2us     4us     2us     2us
# Rows sent      0       1       0       1    0.50       1    0.71    0.50
# Rows examine   0       1       0       1    0.50       1    0.71    0.50
# Rows affecte   0       0       0       0       0       0       0       0
# Merge passes   0       0       0       0       0       0       0       0
# Tmp tables     0       0       0       0       0       0       0       0
# Tmp disk tbl   0       0       0       0       0       0       0       0
# Query size     1     420     210     210     210     210       0     210
# Cpu time       0       0       0       0       0       0       0       0
# Max memory     2   2.55M   1.15M   1.40M   1.28M   1.40M 178.87k   1.28M
# String:
# Bytes sent   n/a
# Databases    test
# Execution en PRIMARY
# Full join    no
# Full scan    no (1/50%), yes (1/50%)
# Hosts        n/a
# No index use no (1/50%), yes (1/50%)
# Tmp table    no
# Tmp table on no
# Tmp tbl size n/a
# Users        n/a
# Query_time distribution
#   1us
#  10us  ################################
# 100us
#   1ms
#  10ms  ################################
# 100ms
#    1s
#  10s+
# Tables
#    SHOW TABLE STATUS FROM `test` LIKE 'scott'\G
#    SHOW CREATE TABLE `test`.`scott`\G
# EXPLAIN /*!50100 PARTITIONS*/
select doc->"$.holeScores[6].number" hole_number, regexp_substr(json_search(replace(replace(replace(doc->"$.holeScores[*].score", ', ' , '","'), '[' ,'["'), ']', '"]'), 'one', 1), '[0-9]+') score_idx from scott\G

MySQL Shell for Visual Studio Code and OCI

It’s also possible to use this plugin with MySQL Shell for Visual Studio Code and generate a slow query log for an MySQL instance running on OCI.

You need to copy the plugins in ~/.mysqlsh-gui/plugin_data/gui_plugin/shell_instance_home/plugins.

For the Windows user, the path is C:\Users\<user_name>\AppData\Roaming\MySQL\mysqlsh-gui\plugin_data\gui_plugin\shell_instance_home\plugins.

Let’s connect using a Free Bastion Host:

Once the plugin is used with the method genetateSlowQueryLog(), a prompt is displayed to enter the filename: We can then open the file that was saved locally:

Conclusion

Here is another solution to generate MySQL Slow Query Log directly from Performance Schema. The generated file is as close as possible to a real slow query log file with a few slight differences, some more information and some less.

As you could see, the plugin can be also used with MySQL Database Service in OCI where we don’t have access to the slow query log file.

Of course, using Performance Schema has some limitations, like the max amount of entries in the table and the possibility to miss some queries during the table truncation if we use the option.

But well used, this can be a very nice source to digest some queries during a limited time.

Enjoy MySQL and good hunting for query optimization candidates !

November 06, 2022

We are pleased to announce the developer rooms that will be organised at FOSDEM 2023. Developer rooms are assigned to self-organising groups to work together on open source projects, to discuss topics relevant to a broader subset of the community, etc. The individual developer room organisers will issue their calls for participation in the next few days. The list below will be updated accordingly. Topic Call for Participation Binary Tools CfP BSD CfP Collaboration and Content Management CfP Community CfP Confidential Computing CfP Containers CfP Continuous Integration and Continuous Deployment CfP Declarative and Minimalistic Computing CfP Distributions CfP DNS舰
A underground subway tunnel with an incoming train.

It's no secret that I don't like using Facebook, Instagram or Twitter as my primary platform for sharing photos and status updates. I don't trust these platforms with my data, and I don't like that they track my friends and family each time I post something.

For those reasons, I set a personal challenge in 2018 to take back control over my social media, photos and more. As a result, I quit Facebook and stopped using Twitter for personal status updates. I still use Twitter for Drupal-related updates.

To this date, I still occasionally post on Instagram. The main reason I still post on Instagram is that it's simply the easiest way for friends and family to follow me. But every time I post a photo on Instagram, I cringe. I don't like that I cause friends and family to be tracked.

My workflow is to upload photos to my website first. After uploading photos to my website, I occasionally POSSE a photo to Instagram. I have used this workflow for many years. As a result, I have over 10,000 photos on my website, and only 300 photos on Instagram. By all means, my website is my primary platform for sharing photos.

I decided it was time to make it a bit easier for people to follow my photography and adventures from my website. Last month I silently added a photo stream, complete with an RSS feed. Now there is a page that shows my newest photos and an easy way to subscribe to them.

While an RSS feed doesn't have the same convenience factor as Instagram, it is better than Instagram in the following ways: more photos, no tracking, no algorithmic filtering, no account required, and no advertising.

My photo stream and photo galleries aspire to the privacy of a printed photo album.

I encourage you to subscribe to my photos RSS feed and to unfollow me on Instagram.

Step by step, I will continue to build my audience here, on my blog, on the edge of the Open Web, on my own terms.

From a technical point of view, my photo stream uses responsive images that are lazy loaded — it should be pretty fast. And it uses the Open Graph Protocol for improved link sharing.

PS: The photo at the top is the subway in Boston. Taken on my way home from work.

Also posted on IndieNews.

November 01, 2022

Il y a un peu moins d’un an, j’ai supprimé mon compte Twitter. Un compte vérifié avec la célèbre icône bleue, suivi par près de 7000 autres comptes Twitter.

Si ce n’est pas exceptionnel, ce compte n’en était pas moins relativement « influent » sur l’échelle Twitter. J’ai pourtant décidé de tenter l’expérience de m’en passer complètement, pour voir. Je savais que j’avais un an pour faire marche arrière. Durant un an, mon compte serait « réactivable » avant d’être définitivement supprimé. Il me reste donc quelques semaines pour changer d’avis.

Et pourtant, cela ne me viendrait pas à l’esprit.

Un sentiment de liberté

Je m’étais déjà déconnecté de mon compte Twitter pendant des périodes plus ou moins longues, jusqu’à trois mois. Mais la suppression est complètement différente. En quelques jours, j’ai tout simplement arrêté de penser à ce qui se passait sur Twitter. J’ai arrêté de penser que ce réseau existait. L’expérience a été la même que pour Facebook et LinkedIn : en supprimant le compte, je me suis ôté un poids énorme. C’est pour Twitter, le réseau auquel j’étais le plus accroc, que la sensation a été la plus forte.

Il est simple de se retirer des réseaux que l’on n’aime pas ou qu’on utilise peu. Le réel changement vient d’accepter de se retirer d’un réseau dont on connait la nocivité, pour soi et pour le monde, sans pourtant pouvoir s’en passer. Parce qu’on est persuadé d’en tirer plus de bien que de mal.

Je suis un blogueur et écrivain qui cherche la gloire. Qui cherche le buzz. Qui cherche à vendre des livres, à être lu, à toucher des lecteurs. Les réseaux sociaux sont littéralement conçus pour les gens comme moi. Et pourtant, ils me prennent beaucoup en m’apportant bien peu. Les pseudobénéfices ne sont que du vent. Une icône bleue à côté de mon nom ? La belle affaire ! Un nombre à quatre chiffres sous la marque « abonnés » ? Un simple attrape-nigaud pour que je tente à tout prix de le faire augmenter.

Twitter ne me manque pas. Au contraire. Je me demande sans cesse pourquoi je ne l’ai pas quitté plus tôt. Pourquoi ceux qui, comme moi, ont la conscience de la nocivité de cette plateforme la crédibilisent en restant dessus. (je ne vais pas leur jeter la pierre, j’en faisais partie il y a moins d’un an).

La migration vers Mastodon

Je souris de la naïveté de certains utilisateurs qui s’indignent de l’arrivée d’Elon Musk à la tête de Twitter. C’était pourtant clair depuis le début, non ? Vous êtes des « utilisateurs ». Vous êtes une marchandise, vous créez la valeur de l’entreprise, que ce soit pour Elon Musk ou un autre. Vous êtes les pigeons et votre indignation ne fait qu’alimenter les débats, les interactions et donc les intérêts publicitaires.

Il n’y a pas de bonne manière d’utiliser un réseau propriétaire. En créant un compte, nous acceptons d’être utilisés, manipulé et que chacune de nos interactions y soit désormais monétisée.

Il y a déjà 5 ans, je tentais de promouvoir Mastodon, une alternative libre et décentralisée à Twitter. Je lis souvent des remarques comme quoi c’est beaucoup plus compliqué.

Non.

C’est juste différent. Si tu n’arrives pas à utiliser Mastodon, c’est que tu n’en as tout simplement pas envie.

C’est plus facile de manger un burger au Macdo qu’un plat équilibré avec des légumes. C’est plus facile de balancer ses déchets dans un parc plutôt que de faire du tri. L’argument de la facilité n’en est pas un. Le monde se modèle selon l’énergie que nous y mettons.

Il faut créer un compte avec un mot de passe et tout ? Sur Twitter aussi. C’est juste que tu as l’habitude. C’est juste que tu t’es connecté sur un serveur Mastodon, que tu as découvert que tu avais 0 follower, que tu n’avais plus la petite icône bleue, que tu ne savais plus « promouvoir tes tweets », que tu ne voyais plus des likes s’afficher en direct sous tes messages. Que tu n’es plus revenu et que donc tu as oublié ton mot de passe. Que c’est plus facile d’accuser le logiciel libre d’être compliqué plutôt que d’affronter sa propre vacuité.

Si tu n’as pas l’envie d’apprendre à utiliser Mastodon, c’est compréhensible. Personne ne te force. Mais n’accuse pas la plateforme.

Twitter est très bon pour te faire croire que tu es un utilisateur important. Mais sur le Fediverse, le réseau décentralisé auquel participent les serveurs Mastodon, il n’y a pas d’utilisateurs, encore moins des importants. Il y a juste des personnes qui sont toutes sur le même pied d’égalité.

La plateforme ne peut pas remplacer l’introspection

C’est peut-être ça le plus difficile à accepter : ce n’est pas Mastodon ni le logiciel libre le problème. C’est toi. C’est toi qui cherches à te sentir important, à être valorisé. En achetant des vêtements chers, le dernier iPhone le jour de sa sortie, une grosse voiture ou une icône bleue à côté de ton pseudo Twitter. Ou tout simplement en ayant beaucoup de retweets, en étant beaucoup cité dans les débats.

La question n’est pas de faire un Twitter-sans-Elon-Musk. La question est de savoir ce que nous voulons, ce que nous cherchons. De faire la part entre ce que nous prétendons (« communiquer ») et ce que nous voulons réellement (« être valorisé », »avoir le sentiment d’exister »).

J’ai plusieurs fois suggéré de supprimer dans Mastodon l’affichage public du nombre de followers pour couper court à toute potentielle spéculation sur cet indicateur. Si je me rends compte qu’un compte à beaucoup de followers, j’ai tendance instinctivement à penser que cette personne est importante, que son avis vaut la peine d’être écouté. Je ne pense pas être le seul.

Tentez l’expérience

Si votre morale personnelle réprouve ce qu’est ou ce que devient Twitter, je ne peux que vous inviter à tenter l’expérience de supprimer votre compte. Rappelez-vous : vous avez un an pour changer d’avis.

Faites fi de ces conversations tellement importantes, de cette communauté que vous ne pouvez pas « abandonner », de ces ennemis virtuels qui verront votre départ comme une victoire. Tentez simplement de vous aligner avec vos propres valeurs morales. Juste pour voir.

Supprimer votre compte est également la seule et unique manière de protester, de toucher l’entreprise là où ça lui fait mal.

Bien sûr, vous pouvez aussi venir sur Mastodon. Mais ce n’est pas nécessaire. C’est même peut-être contreproductif. Si vous étiez accroc à Twitter, vous serez tenté de voir dans Mastodon un substitut. Il est sans doute préférable de se sevrer de Twitter avant de découvrir autre chose.

Bonus

Installez l’extension libredirect dans votre navigateur, de manière à pouvoir continuer à consulter Twitter à travers l’interface Nitter (vous pouvez même suivre les comptes Twitter dans votre lecteur RSS). Sur Android, utilisez Fritter.

=> https://libredirect.codeberg.page/ Libredirect pour Chrome/Firefox
=> https://f-droid.org/en/packages/com.jonjomckay.fritter/ Fritter pour Android

Recevez les billets par mail ou par RSS. Max 2 billets par semaine, rien d’autre. Adresse email jamais partagée et définitivement effacée lors du désabonnement. Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, lisez, offrez et partagez des livres.

Ce texte est publié sous la licence CC-By BE.

October 31, 2022

Writing tests can be hard. The key is to get started. When I start work on a new Drupal module, I like to start off by adding a very simple test and making it pass. Once I have one simple test passing, it becomes easy to add more. The key is get started with the simplest test possible. This page explains how I do that.

The most basic PHPUnit test for Drupal

The most basic test looks something like this:


   */
  protected static $modules = ['my_module'];

  /**
   * Theme to enable. This field is mandatory.
   *
   * @var string
   */
  protected $defaultTheme = 'stark';

  /**
   * The simplest assert possible.
   */
  public function testOne() {
    $this->assertTrue(TRUE);
  }
}

Drupal uses PHPUnit for testing, and the test above is a PHPUnit test.

The test lives in docroot/modules/custom/my_module/tests/src/Functional/MyModuleTest.php.

Installing PHPUnit for Drupal

Drupal does not ship with PHPUnit out-of-the-box, so it needs to be installed.

The best way to install PHPUnit on a Drupal site is by installing the drupal/core-dev package. It can be installed using Composer:

$ composer require drupal/core-dev --dev --update-with-all-dependencies

The command above will download and configure PHPUnit, along with other development dependencies that are considered a best-practice for Drupal development (e.g. PHPStan).

Configuring PHPUnit for Drupal

Once installed, you still have to configure PHPUnit. All you need to do is set two variables:

  • SIMPLETEST_BASE_URL should be the URL of your site (e.g. http://localhost/).
  • SIMPLETEST_DB should be your database settings (e.g. mysql://username:password@localhost/database).

You can specify these directly in docroot/core/phpunit.xml.dist, or you can set them as environment variables in your shell.

Running a PHPUnit test for Drupal

With PHPUnit installed, let's run our basic test:

$ cd docroot/core
$ ../../vendor/bin/phpunit ../modules/custom/my_module/tests/src/Functional/MyModuleTest.php

We have to execute our test from inside the docroot/core/ directory so PHPUnit can find our test.

Using PHPUnit with Lando

I use Lando for my local development environment. There are a couple of things you can add to your .lando.yml configuration file to make using PHPUnit a bit easier:

services:
  appserver:
    overrides:
      environment:
        SIMPLETEST_BASE_URL: "http://localhost/"
        SIMPLETEST_DB: "mysql://username:password@localhost/database"
        BROWSERTEST_OUTPUT_DIRECTORY: "/app/phpunit-results"
tooling:
  test:
    service: appserver
    cmd: "php /app/vendor/bin/phpunit --verbose -c /app/docroot/core/phpunit.xml.dist"

The first 6 lines sets the variables and the last 4 lines create a new lando test command.

After changing my .lando.yml, I like to rebuild my Lando environment, though that might not be strictly necessary:

$ lando rebuild -y

Now, you can run the PHPUnit test as follows:

$ lando test modules/custom/my_module/tests/src/Functional/MyModuleTest.php

About this page

This page is part of my digital garden. It's like a page in my personal notebook rather than a finished blog post. I documented this information for myself, but I'm putting it out there as it might be useful for others. Contrary to my blog posts, pages in my digital garden are less finished, and updated and refined over time. It grows and lives like a plant in a garden, and I'll tend to it from time to time.

October 26, 2022

Gisteren was Frank Vander linden in De Ideale Wereld. Mijn Kosovaarse vriendin vroeg me wie die interessante kale vent op de TV was. Ik antwoordde haar: “oh ik denk een drummer van De Kreuners of zo”.

Dat was onvergefelijk fout, natuurlijk. Als boetedoening speel ik vandaag alle nummers van De Mens af hier in de huiskamer. Zodat ze een beetje onze Vlaamse muziekcultuur kan waarnemen.

Bij deze ook mijn uitgebreide excuses in deze open brief aan de Frank: sorry dat ik mij totaal vergist heb. Ik heb er heel de nacht over wakker gelegen. Hoe kon zoiets toch gebeuren?

October 24, 2022

It has been a while since I did not take time to write a security conference wrap-up. With all these COVID restrictions, we were stuck at home for a while. Still today, some events remain postponed and, worse, canceled! The energy crisis in Europe does not help, some venues are already increasing their prize to host events! How will this evolve? No idea, but, fortunately, they are motivated people who are motivated to organize excellent events! Still no hack.lu this year but, people from CIRCL.lu decided to organize the first edition of the Cyber Threat Intelligence Summit in Luxembourg. Before the pandemic, there was also the MISP summit organized close to hack.lu. This event, held last week, was a mix of pure CTI- and MISP-related presentations. Here is a quick wrap-up and some links to interesting content.

The first-day keynote was performed by Patrice Auffret and was about “Ethical Internet Scanning in 2022”. Patrice is the founder of Onypthe. If you don’t know the service, it’s a very good concurrent to Shodan. They collect data about Internet connected objects and URLs. Today, scanning is accepted by most of network owners. IMHO, if you care about port-scanning, you failed. It’s part of the “background noise”. What about the ethical aspect of this? Questions that won’t be covered: law aspects, is it useful/useless? Patrice made 10 recommendations to follow when scanning the Internet:

  • Explain the purpose
  • Put an opt-out
  • Provide abuse contacts
  • Provides lists of scanners IP addresses
  • Have good reverse DNS
  • Handle abuse requests
  • Don’t fuzz, just use standard packets/protocols
  • Scan slowly
  • Use fixed IP addresses (no trashable ones)
  • Remote collected data (upon request – GDPR)

They are many active players (Shodan, Censys, ZoomEye, LeakIX, …). Don’t blame them because they scan you. Buy an account, use their REST API and query the information they know about you and increase your footprint visibility. Don’t forget that bad guys do it all the time, they know your assets better than you. Also “You can’t’ secure assets you don’t know

Robert Nixon presented two talks related to MISP: “In Curation we trust” and, later, “MISP to Power BI”. He explained the common problem that we are all facing when starting to deal with threat intel. The risks of “noise” and low-level information. If you collect garbage, you’ll enrich garbage and generate more garbage. Think about the classic IOC “8.8.8.8”. He explained the process of curating MISP events to make them more valuable inside an organization. Example of processing:

  • Remove potential false positive
  • Reduce the lack of contextualization or inconsistencies
  • Are IOCs actionable?

Robert stayed on stage for the 2nd part of his presentation: “MISP + Power BI”. The idea is to access the MISP SQL db from Power BI (of course in a safe way) and create powerful dashboards based on data available in MISP. Be careful with the tables you will open (correlation is not the best one). Robert explained how data can be prepared for better performances (transform some data), convert them, and remove unwanted columns).

The next presentation was called “What can time-based analysis say in Ransomware cases?” by Jorina Baron & david Rufenacht. Like many MSPP, they’ve been busy with ransomware for a while. The compiled data from multiple incidents were used to generate some useful statistics. They focused on:

  • Initial access
  • Lateral movement
  • Effect on target

The dataset was based on 31 ransomware attacks. Some facts:

  • 14d between initial to lateral
  • 40h between lateral to encryption

Interesting approach for ransomware attacks, besides the classic technical details.

Then Sami Mokaddem presented “Automation with MISP workflows”. Recently, a new feature was added to MISP: Workflows. You may roughly compare it to a small XOAR tool inside MISP that helps to trigger actions based on parameters. Example of uses:

  • Chat notifications
  • Prevent publication with a specific sanity check 

After lunch, we had some lightning talks and “HZ Rat goes China – Following the tail of an unknown backdoor” by Axel Wauer. The malware was analyzed and some numbers were reported:

  • 120 samples analyzed
  • 2 delivery techniques
  • 3 malware versions identified
  • C2 servers are online
  • Campaign is ongoing for 2y
  • Focus on Asia/China

As usual, Andras Iklody presented some quick updates about MISP. What changed recently, what’s in the pipe. The amount of work is impressive: 16 releases, 3768 commits, and 100+ contributors.

Cyril Bras came on stage to present his view on sharing IOCs with customers and partners.

And we had another interesting tool to expand the MISP capabilities. Koen Van Impe presented his tool “Web Scraper“. The idea is to feed a MISP instance with instructed informations or structured (CSV, TXT, …). You definitely need to check it if you are struggling with a lot of documents to ingest in MISP!

Another topic that comes often on the table about MISP: How to use it and the cloud and, more important, how to scale it properly? Mikesh Nagar came to talk about his experience with high-level MISP instances hosted in the cloud.

The last presentation of the day was not public.

The second started with another keynote presented by Gregory Boddin: “Internet Leak Exchange Platform“. LeakIx could be seen as a competitor of Onyphe but they focus more on searching for vulnerabilities instead of “mapping the Internet”. The idea could be resumed to “Get there before the threat actors”. Gregory explained how the platform works, what they are searching for, and how they handle collected data.

Then, Markus Ludwig came with a presentation called “Communities – the underestimated super power of CTI“. Not technical at all, the idea was to have a broader view of the huge amount of security researchers. They are more free researchers than paid ones! The community is big and people deserve credits to keep them motivated and, in this case, they tend to contribute more! Great presentation!

Paul Jung presented “How to fail your TI usage?“. I like the idea. When you have to support multiple infrastructures, tools and customers, it’s very easy to make mistake and to fail in distributing high-value CTI. Paul reviewed his top-14 fails and annoying stuff when using CTI for detection

  • RFC1918 IPs, multicast IP, … (Tip: use warning lists from MISP)
  • Wrong URLs http://p or http://https://xxx -> Validate FQDN
  • Human FP – use top Alexa, 8.8.8.8 in reports?
  • NDA & other funny limitations (TLP, sharing across multiple customers)
  • Automation issues
  • Babel issues (deal with tags/taxonomies/humans). Naming conventions!
  • Keep control (control what if you have … or not)
  • Hardware limitations: SSL Ex: www.pastebin.com/xxxxx This FQDN is useless without ID/path/…
  • Hardware limitation: SIEMs : Parameters to URLs can be removed, will never match
  • Hardware limitation: SIEMs’2: Too much data to ingest, try to reduce
  • Decaying and life-cycle
  • Maintaining TI takes time
  • Avoid “Void-ification” of events…An IP alone is not relevant (context)
  • Keep data relevant for an analyst (what kind of threat is it? ingress? egress? 

Then Louise Taggart presented her view about “Strategic Intelligence“. Threat Intelligence does not only rely on technical information. They are different levers:

  • Tactical “what” -> IOCs
  • Operational “how” -> TTPs
  • Strategic “why? what’s next? so what?” (context, motivation, …)

Then, we switched back to technical content with a presentation by the French ANSSI: “Problematic of air-gapped MISP instances“… MISP is a tool that requires a lot of networking resources. By design, a MISP instance must exchange information (events) with other instances. But sometimes, MISP is used in high-security environment that are disconnected from the Internet (air-gapped). In this case, how to use MISP? How to deploy it and how to feed it? Some processes were describe, like how to build and deploy a MISP instance without connectivity. A tool was presented: sftp2misp. I already worked with air-gapped MISP instances and, trust me, it’s not easy!

———-

Paul Rascagneres came on stage to talk about “The state of the art of webshells in 2022”. Webshell remains a classic way to interact with a compromised computer. Once a vulnerability can be exploited, attackers drop a webshell to perform the next steps of the attack. Paul reviewed different types of webshells (of course, not all of them have the same quality and anti-detection defenses). The example of in-memory webshell based on a Tomcat Valve was pretty cool.

After the lunch and a new serie of lighthing talks, Antoine Cailliau presented his tool: DocIntel. We all have thousands of CTI-related documents like blog posts, PDF reports, RSS feeds, … The goal of DocIntel is to index them, extract useful information and enrich them. This way, you’ll build your own CTI repository. I just installed my own instance as a test and it deserves to be invstigated.

The next talk was about the similar topic: “Report Curation and Threat Library – How to organize your. Knowledge” by Patrick Grau.

Koen came back on stage to present “CTI Operational Procedures with Jupyter Notebooks and MISP“. Jupiter notebooks are pretty popular for a while. Koen explained how to use MISP, PyMISP with Jupyter notebooks to document your CTI operational procedures. 

The next talk was “The holy grail for STIX and MISP format” by Christian Studer.

Quentin Jerome presented “WHIDS Update“. WHIDS is an open source EDR for Windows. Great project that is directly linked to MISP to get IOC and detect suspicious activites.

During the lightning talks, some cool projects/tools were presented:

The event was intensive: 2 days, talks of 30 mins with great content. Of course, the social dimension was the best one. It’s so cool to see good old (and new!) friends. Teasing: a second edition of the CTI-Summit should be organized next year in parallel to hack.lu.

All talks (expect the non-TLP:Clear ones) have been recoreded and are available on Youtube, thanks to Cooper who made an awesome job as usual!

The post CTI-Summit 2022 Luxembourg Wrap-Up appeared first on /dev/random.

October 22, 2022

Normaal gesproken

Normaal gaan we in Vlaanderen niet al te extreem doen met onze houtkachel. Per slot van rekening hebben we allemaal wel of gas, elektriciteit of een warmtepomp of nog iets anders om onze woning te verwarmen.

De houtkachel werd of wordt meestal gebruikt om gezelligheid te bereiken.

Maar dit jaar hebben we een redelijk zware energiecrisis. Die zorgt ervoor dat ons gas heel erg duur is. Daarom gaan we een aantal alternatieven uitproberen.

Wat ik in mijn tuin zoal vond waren de eikels van een eikenboom. Enkele problemen met het verbranden van eikels:

  • Je moet de eikels verzamelen
  • De eikels zijn (vaak te) vochtig
  • De (verzamelde) eikels zijn gemengd met bladeren
  • Je moet de eikels snel in de kachel krijgen

De eikels verzamelen

Dit doe ik met een grashark. Ik geef voorkeur aan eikels die op verharde grond liggen. We moeten niet autistisch doen en proberen alle eikels te verzamelen. Meestal zijn er voldoende eikels die op de verharde grond liggen. Bovendien gebruik ik de hark om de bladeren te scheiden van de eikels (we zitten hier op aarde niet in een vacuüm en dus vallen de eikels sneller dan de bladeren. Je bent een ingenieur? Dan kan je die kennis gebruiken).

De eikels zijn te vochtig

Ik verzamel de eikels wanneer het droog weer is. Bovendien houd ik de eikels een paar dagen (zelfs weken) bij in een (open) emmer of container alvorens ik ze in de kachel gebruik. Hoe langer je ze kan drogen, hoe beter.

De eikels zijn (nog steeds) gemengd met bladeren

Die bladeren zijn vaak vuil en vochtig, wat rook veroorzaakt wanneer we de verzameling verbranden. Daarom heb ik een ijzeren raster op de emmers waarin ik de eikels laat vallen gelegd. Zo kunnen de bladeren eenvoudig gefilterd worden.

Je moet de eikels snel in de kachel krijgen.

Hoe langer het deurtje van de kachel open is, hoe meer rook er ontsnapt. Dat is niet goed want ongezond voor de mensbeesten die zich in de buurt van de kachel bevinden. Daarom heb ik geïnvesteerd in een ijzeren blik om de eikels uit een grotere emmer te kunnen scheppen in voldoende hoeveelheid per keer. Ik leg de eikels ook rondom het brandende vuur in plaats van in het vuur. Zodat pas later, wanneer de deur van de kachel reeds gesloten is (en er geen rook in de leefruimte meer kan ontsnappen), de eikels zullen verbranden.

Tot slot

Tot slot heb ik een luchtzuiveraar in de leefruimte staan. Iedere ochtend open ik ook alle ramen (d.i. vrij belangrijk). Tijdens de avond zijn er vaak ook twee ramen die in een lijn van West naar Oost liggen open. De wind in Vlaanderen waait meestal van West naar Oost. Geloof je me niet? Kijk maar eens hoe de landingsbanen van de luchthaven van Zaventem liggen. En zoek op waarom dat zo is. Je wil dat de wind snel door je leefruimte waait, zodat het zoveel mogelijk fijnstof meeneemt.

Uiteraard verlies je door de raam open te zetten wat van de door de kachel gegenereerde warmte. Maar als je daar dan weer een beetje over nadenkt, dan doet dat er niet zoveel toe: je hebt die warmte gratis vergekregen door gebruik te maken van eikels. Of niet?

Of is het beter dat je longen gevuld worden met fijnstof?

Je kan ook even na de kacheldeur gesloten is die ramen weer sluiten. Uit een goede kachel die goed geïnstalleerd is ontsnapt letterlijk niets van rook naar de leefruimte. Als vanzelfsprekend wil je dit enkel met een goede kachel doen.

Natuurlijk

Is dit allemaal goed voor de luchtkwaliteit van Vlaanderen? Vast niet.

Mijn kachel is een erg moderne DDG kachel die m.a.w. een zogenaamde volledige verbranding doet (en ook aangesloten is op de centrale verwarming). Maar dit is uiteraard niet goed voor de luchtkwaliteit van het land.

Ik ben er niet zeker van of men momenteel veel betere alternatieven heeft: Russisch gas verbranden aan meer dan duizend euro per maand is ook niet echt een goede optie. Elektriciteit is onwaarschijnlijk duur. Dus ja.

De wet

Je mag niets anders dan onbewerkt hout verbranden. De eikels zijn volgens mij onbewerkt hout. Maarja. Het hangt vast van een interpretatie af.

October 20, 2022

With the latest MySQL release (8.0.31), MySQL adds support for the SQL standard INTERSECT and EXCEPT table operators.

Let’s have a look how to use them.

We will use the following table:

CREATE TABLE `new` (
  `id` int NOT NULL AUTO_INCREMENT,
  `name` varchar(20) DEFAULT NULL,
  `tacos` int DEFAULT NULL,
  `sushis` int DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB

For our team meeting, we will order tacos and sushi’s.

Each record represent the order of each team member:

select * from new;
+----+-------------+-------+--------+
| id | name        | tacos | sushis |
+----+-------------+-------+--------+
|  1 | Kenny       |  NULL |     10 |
|  2 | Miguel      |     5 |      0 |
|  3 | lefred      |     4 |      5 |
|  4 | Kajiyamasan |  NULL |     10 |
|  5 | Scott       |    10 |   NULL |
|  6 | Lenka       |  NULL |   NULL |
+----+-------------+-------+--------+

Intersect

The manual says that INTERSECT limits the result from multiple SELECT statements to those rows which are common to all. INTERSECT operator is part of the ANSI/ISO SQL standard (ISO/IEC 9075-2:2016(E))

We want to run two queries, the first one will list all records where the team member chose tacos and the second one will return all records where the person chose sushi’s.

The two separate queries are:

(query 1) select * from new where tacos>0;

(query 2) select * from new where sushis>0;

The only record that is present in both results is the one with id=3.

Let’s use INTERSECT to confirm that:

select * from new where tacos > 0 
intersect 
select * from new where sushis > 0;
+----+--------+-------+--------+
| id | name   | tacos | sushis |
+----+--------+-------+--------+
|  3 | lefred |     4 |      5 |
+----+--------+-------+--------+

Excellent, on previous versions of MySQL, the result of such query would have been:

ERROR 1064 (42000): You have an error in your SQL syntax; 
check the manual that corresponds to your MySQL server version 
for the right syntax to use near 
'intersect select * from new where sushis > 0' at line 1

Except

In the manual, we can read that EXCEPT limits the result from the first SELECT statement to those rows which are (also) not found in the second.

Let’s find out all team members that will only eat tacos using EXCEPT:

select * from new where tacos > 0 
except 
select * from new where sushis > 0;
+----+--------+-------+--------+
| id | name   | tacos | sushis |
+----+--------+-------+--------+
|  2 | Miguel |     5 |      0 |
|  5 | Scott  |    10 |   NULL |
+----+--------+-------+--------+

And if we want to perform the reverse and get all those that will only eat sushi’s we inverse the queries order like this:

select * from new where sushis > 0 
except 
select * from new where tacos > 0;
+----+-------------+-------+--------+
| id | name        | tacos | sushis |
+----+-------------+-------+--------+
|  1 | Kenny       |  NULL |     10 |
|  4 | Kajiyamasan |  NULL |     10 |
+----+-------------+-------+--------+

Conclusion

MySQL 8.0.31 continues the legacy of 8.0 to include support for SQL standards such as Window Functions, Common Table Expressions, Lateral Derived Tables, JSON_TABLES, JSON_VALUE, …

Enjoy MySQL !

October 17, 2022

Once again, the latest version of MySQL 8.0 includes several contributions from the MySQL Community.

MySQL 8.0.31 was released on October 11th 2022.

I would like to thank all contributors on behalf of the entire Oracle MySQL team !

This new release contains patches from Facebook/Meta, Dennis Gao, Lou Shuai, Caza Zhang, Zhang Simon from Tencent, Dimitry Kudryavtsev, Rahul Malik from Percona, Alex Xing, Marcelo Altmann from Percona, Matthew Steeples, Adam Croot, Luis Pinto and Evgeniy Patlan from Percona for all his patches on MySQL Shell.

Here is the list of the above contributions and related bugs:

Replication

  • #101056 – Ack_receiver may lost the slave semi-sync ack due to net timeout – Dennis Gao (fixed based on 8.0.30)
  • #104491 – Fix race between binlog sender heartbeat timeout – Facebook

Group Replication

  • #107635 – event scheduler cause error on group replication – Lou Shuai

Optimizer

  • #104040 – update histogram with json data – Caza Zhang

Server DDL

  • #105903 – data is incorrect after instant add column – Zhang Simon from Tencent
  • #107633 – fixing a type-o, should be \”truncate\” – Dimitry Kudryavtsev

InnoDB

  • #106616 – failure occurred while upgrading a MySQL instance with a MySQL 5.6 data directory containing user-created table with a particular table ID to 8.0 – Rahul Malik from Percona
  • #106952 – it is better to clear upd_t in trx_undo_prev_version_build – Alex Xing
  • #107613 – Assertion failure: dict0dd.cc:1693:dd_column_is_dropped(old_col) – Marcelo Altmann from Percona (based on 8.0.30)

Connectors

Connector/Net

  • #108091 – Add PackageLicenseExpression so that NuGet can display it – Matthew Steeples
  • #108290 Fix malformed URL – Adam Croot

Connector/C++

  • #108355 – Fix windows compilation issue in cmake/libutils/save_linker_o – Luis Pinto

MySQL Shell

  • #107412 – Fix requested python version – Evgeniy Patlan from Percona
  • #107415 – Fix incorrect lin in doc – Evgeniy Patlan from Percona
  • #107411 – Improve wording in postrm actions – Evgeniy Patlan from Percona
  • #107413 – Add libssh to requirements – Evgeniy Patlan from Percona
  • #107416 – Fix path in doc – Evgeniy Patlan from Percona
  • #107414 – Fix cmake version needed for build – Evgeniy Patlan from Percona

As usual, those contributions cover many different sections of MySQL. We can notice that we start receiving more MySQL Shell contributions.

If you have patches and you also want to be part of the MySQL Contributors, it’s easy, you can send Pull Requests from MySQL’s GitHub repositories or send your patches on Bugs MySQL (signing the Oracle Contributor Agreement is required).

And don’t forget, that if you want to extend MySQL, the best way is to create a component. Now there is a complete series to learn how to start using the MySQL Component Infrastructure (part 1) !

Thank you again to all our contributors !

October 14, 2022

In the previous part, we created our application and our 2 functions.

Now we need to create an API Gateway to be able to call these functions from outside OCI. Using a third party scheduler, our laptop, etc…

Before creating the gateway, we need to create some policy to allow the API Gateway to call our functions. I’ve tried to create the policy after, it seems to not work as expected.

Security

Dynamic Group

We need to create a Dynamic Group that will match our gateway:

The matching rule is checking the type of resource that should be ApiGateway and my compartment’s id:
All {resource.type='ApiGateway', resource.compartment.id = 'ocid1.compartment.oc1..xxxxxxxxx'}

Policy

Now that we have created the dynamic group, we need to create a policy:

The Policy Statements are sentences describing what is allowed:

These are my statement in text, lefred_api_grp is the dynamic group’s name and my compartment is sandbox-lefred. You need to replace them by yours:

Allow dynamic-group lefred_api_grp to manage fn-function in compartment sandbox-lefred
Allow dynamic-group lefred_api_grp to read fn-app in compartment sandbox-lefred
Allow dynamic-group lefred_api_grp to use fn-invocation in compartment sandbox-lefred
Allow dynamic-group lefred_api_grp to use virtual-network-family in compartment sandbox-lefred
Allow dynamic-group lefred_api_grp to manage public-ips in compartment sandbox-lefred
Allow dynamic-group lefred_api_grp to use functions-family in compartment sandbox-lefred

API Gateway

We have everything we need to create our API Gateway and defines the routes to call our functions:

Once the Gateway is active, we can create a deployment:

To not make it too complicate, I’m bypassing the authentication.

And we need to define two routes:

  • /slowlog_json : to call the export in JSON
  • /slowlog_plain : to call the export of a plain text slow query log

Using the API Gateway

Now we can use the API with curl for example like this, from my laptop, replace the xxxxx by your gateway’s endpoint:

$ curl https://xxxxx.apigateway.us-ashburn-1.oci.customer-oci.com/slowlog_plain \
  -X POST -H 'Content-Type: application/json' -d '{"mds_host": "10.0.1.127",
  "mds_user": "admin", "mds_port": "3306", "mds_password": "Passw0rd!",
   "mds_name": "lefred-mysql"}'
{"message": "MySQL Slow Log saved: slow_lefred-mysql_202210140748.log"}

And the file is now available in Object Storage:

We can download the file and use it with pt-query-digest for example:

We can also get the JSON file and use it with other third party tool. This is an example of the file opened in Firefox:

The output also contains information like on which engine was the query executed, InnoDB (PRIMARY) or HeatWave (SECONDARY):

This is an example of Grafana Loki when loading those generated JSON files:

Scheduled Dumps

It is possible to call the application from external tools like EasyCron or Cronless.

I’ve decided to use a cronjob directly from my Always Free Ampere instance. This cronjob is calling the /slowlog_plain route every 10 mins:

If we check the cron daemon’s output, we can see the list of generated files:

We can see that all to logs are sent to Object Storage:

Limitations

Using this method to export slow query log information from the MySQL Database Service has some limitations. But in general, the reported information should be more than sufficient to have a decent overview of the workload.

These are the limitations:

Conclusion

This method allows the MySQL DBA to easily retrieve slow query log information directly from Object Storage for all MySQL database instances running on OCI.

This approach may not be suitable for large workloads. If you need more information about your workload in OCI and want to get details about your readings, I recommend using MySQL Enterprise Monitor (MEM) or any other third-party tool on a Compute Instance in your Tenancy.

Examples of MEM Queries Dashboard:

Enjoy MySQL and happy query optimization !

In my previous post, I explained how to deal with Performance_Schema and Sys to identify the candidates for Query Optimization but also to understand the workload on the database.

In this article, we will see how we can create an OCI Fn Application that will generate a slow query log from our MySQL Database Service instance and store it to Object Storage.

The creation of the function and its use is similar to the one explained in the previous post about creating a logical dump of a MySQL instance to Object Storage.

The Application

We need to create an application and 2 functions, one to extract the data in JSON format and one in plain text. We also need to deploy an API Gateway that will allow to call those function from anywhere (publicly):

Let’s start by creating the application in OCI console:

We need to use the Public Subnet of our VCN to be able to reach it later from our API Gateway:

After we click on Create, we can see our new Application created:

We then need to follow the statement displayed on the rest of the page. We use Cloud Shell:


This looks like this:

fdescamp@cloudshell:~ (us-ashburn-1)$ fn update context registry iad.ocir.io/i***********j/lefred
Current context updated registry with iad.ocir.io/i***********j/lefred
fdescamp@cloudshell:~ (us-ashburn-1)$ fn update context registry iad.ocir.io/i***********j/lefred
Current context updated registry with iad.ocir.io/i***********j/lefred
fdescamp@cloudshell:~ (us-ashburn-1)$ docker login -u 'idinfdw2eouj/fdescamp' iad.ocir.io
Password: **********
WARNING! Your password will be stored unencrypted in /home/fdescamp/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded
fdescamp@cloudshell:~ (us-ashburn-1)$ fn list apps
NAME            ID
slow_query_log  ocid1.fnapp.oc1.iad.xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxq

After that in Cloud Shell, we initialize our two new functions:

fdescamp@cloudshell:~ (us-ashburn-1)$ fn init --runtime python mysql_slowlog_txt
Creating function at: ./mysql_slowlog_txt
Function boilerplate generated.
func.yaml created.
fdescamp@cloudshell:~ (us-ashburn-1)$ fn init --runtime python mysql_slowlog
Creating function at: ./mysql_slowlog
Function boilerplate generated.
func.yaml created.

Both functions are initialized, we will start with the one dumping the queries in JSON format.

Function mysql_slowlog

As this is the first function of our application, we will define a Dockerfile and the requirements in a file (requirements.txt) in the folder of the function:

fdescamp@cloudshell:~ (us-ashburn-1)$ cd mysql_slowlog
fdescamp@cloudshell:mysql_slowlog (us-ashburn-1)$ ls
Dockerfile  func.py  func.yaml  requirements.txt

We need to add the following content in the Dockerfile:

FROM fnproject/python:3.9-dev as build-stage
WORKDIR /function
ADD requirements.txt /function/

RUN pip3 install --target /python/  --no-cache --no-cache-dir -r requirements.txt && rm -fr ~/.cache/pip /tmp*  requirements.txt func.yaml Dockerfile .venv && chmod -R o+r /python

ADD . /function/

RUN rm -fr /function/.pip_cache

FROM fnproject/python:3.9
WORKDIR /function

COPY --from=build-stage /python /python
COPY --from=build-stage /function /function

RUN chmod -R o+r /function && mkdir -p /home/fn && chown fn /home/fn

ENV PYTHONPATH=/function:/python
ENTRYPOINT ["/python/bin/fdk", "/function/func.py", "handler"]

The requirements.txt file needs to contain the following lines:

fdk>=0.1.48
oci
mysql-connector-python

We also need to modify the content of func.yaml file to increase the memory to 2048:

memory: 2048

All the magic of the function resides in the Python file func.py.

Modify the content of the file with the code of the file linked above.

Once done, we can deploy the function:

fdescamp@cloudshell:mysql_slowlog (us-ashburn-1)$ fn -v deploy --app slow_query_log
Deploying mysql_slowlog to app: slow_query_log
Bumped to version 0.0.1
Using Container engine docker
Building image iad.ocir.io/i**********j/lefred/mysql_slowlog:0.0.1
Dockerfile content
-----------------------------------
FROM fnproject/python:3.9-dev as build-stage
WORKDIR /function
ADD requirements.txt /function/

RUN pip3 install --target /python/  --no-cache --no-cache-dir -r requirements.txt && rm -fr ~/.cache/pip /tmp*  requirements.txt func.yaml Dockerfile .venv && chmod -R o+r /python

ADD . /function/

RUN rm -fr /function/.pip_cache

FROM fnproject/python:3.9
WORKDIR /function

COPY --from=build-stage /python /python
COPY --from=build-stage /function /function

RUN chmod -R o+r /function && mkdir -p /home/fn && chown fn /home/fn

ENV PYTHONPATH=/function:/python
ENTRYPOINT ["/python/bin/fdk", "/function/func.py", "handler"]


-----------------------------------
FN_REGISTRY:  iad.ocir.io/i**********j/lefred
Current Context:  us-ashburn-1
Sending build context to Docker daemon  9.728kB
Step 1/13 : FROM fnproject/python:3.9-dev as build-stage
 ---> 808c3fde4a95
Step 2/13 : WORKDIR /function
 ---> Using cache
 ---> 7953c328cf0e
Step 3/13 : ADD requirements.txt /function/
 ---> Using cache
 ---> 5d44308f3376
Step 4/13 : RUN pip3 install --target /python/  --no-cache --no-cache-dir -r requirements.txt && rm -fr ~/.cache/pip /tmp*  requirements.txt func.yaml Dockerfile .venv && chmod -R o+r /python
 ---> Using cache
 ---> 608ec9527aca
Step 5/13 : ADD . /function/
 ---> ae85dfe7245e
Step 6/13 : RUN rm -fr /function/.pip_cache
 ---> Running in 60421dfa5e4d
Removing intermediate container 60421dfa5e4d
 ---> 06de6b9b1860
Step 7/13 : FROM fnproject/python:3.9
 ---> d6c82f055722
Step 8/13 : WORKDIR /function
 ---> Using cache
 ---> b6bf41dd40e4
Step 9/13 : COPY --from=build-stage /python /python
 ---> Using cache
 ---> c895f3bb74f7
Step 10/13 : COPY --from=build-stage /function /function
 ---> b397ec7769a1
Step 11/13 : RUN chmod -R o+r /function && mkdir -p /home/fn && chown fn /home/fn
 ---> Running in 5af6a775d055
Removing intermediate container 5af6a775d055
 ---> fac578e4290a
Step 12/13 : ENV PYTHONPATH=/function:/python
 ---> Running in fe0bb2f24d6e
Removing intermediate container fe0bb2f24d6e
 ---> c0460b0ca6f9
Step 13/13 : ENTRYPOINT ["/python/bin/fdk", "/function/func.py", "handler"]
 ---> Running in 0ed370d1b391
Removing intermediate container 0ed370d1b391
 ---> 6907b3653dac
Successfully built 6907b3653dac
Successfully tagged iad.ocir.io/i************j/lefred/mysql_slowlog:0.0.1

Parts:  [iad.ocir.io i*************j lefred mysql_slowlog:0.0.1]
Using Container engine docker to push
Pushing iad.ocir.io/i********j/lefred/mysql_slowlog:0.0.1 to docker registry...The push refers to repository [iad.ocir.io/i**********j/lefred/mysql_slowlog]
50019643244c: Pushed 
b2b65f9f6bdd: Pushed 
4ae76999236e: Layer already exists 
9dbf415302a5: Layer already exists 
fcc297df3f46: Layer already exists 
79b7117c006c: Layer already exists 
05dc728e5e49: Layer already exists 
0.0.1: digest: sha256:e0a693993c7470557fac557cba9a2a4d3e828fc2d21789afb7ebe6163f4d4c14 size: 1781
Updating function mysql_slowlog using image iad.ocir.io/i**********j/lefred/mysql_slowlog:0.0.1...

Function mysql_slowlog_txt

Now we go in the ../mysql_slowlog_txt directory and we modify the func.yaml file to increase the memory to match the same amount as th previous function (2048).

Then we copy the content of this files in to func.py.

When done we can deploy that function too:

fdescamp@cloudshell:mysql_slowlog (us-ashburn-1)$ cd ../mysql_slowlog_txt/
fdescamp@cloudshell:mysql_slowlog_txt (us-ashburn-1)$ vi func.yaml 
fdescamp@cloudshell:mysql_slowlog_txt (us-ashburn-1)$ vi func.py 
fdescamp@cloudshell:mysql_slowlog_txt (us-ashburn-1)$ fn -v deploy --app slow_query_log

Variables

Our application requires some variables to work. Some will be sent every time the function is called, like which MySQL Instance, the user’s credentials, which Object Storage bucket to use, … Some will be “hardcoded” to not having to specify them each time (like the tencancy, oci user, …).

We use again Cloud Shell to specify those that won’t be specified each time:

fn config app slow_query_log oci_fingerprint "fe:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:XX:3d"
fn config app slow_query_log oci_tenancy 'ocid1.tenancy.oc1..xxxxx'
fn config app slow_query_log oci_user "ocid1.user.oc1..xxxxxx"
fn config app slow_query_log namespace "i********j"
fn config app slow_query_log bucket "lefred-bucket"
fn config app slow_query_log oci_region "us-ashburn-1"

We also need to provide an OCI key as a string. The content of the string can be generated using base64 command line program:

And then we add it in Cloud Shell like this:

fn config app slow_query_log oci_key '<THE CONTENT OF THE STRING ABOVE>'

Testing

If we have the security list well configured (Private Subnet accepting connection on port 3306 from Public Subnet) and if we have an Object Storage bucket ready (with the name configured earlier), we can already test our functions directly form Cloud Shell:

fdescamp@cloudshell:~ (us-ashburn-1)$ echo -n '{"mds_host": "10.0.1.127",
 "mds_user": "admin", "mds_port": "3306", "mds_password": "Passw0rd!", 
 "mds_name": "lefred-mysql"}' | fn invoke slow_query_log mysql_slowlog
{"message": "MySQL Slow Log saved: slow_lefred-mysql_202210132114.json"}

fdescamp@cloudshell:~ (us-ashburn-1)$ echo -n '{"mds_host": "10.0.1.127",
 "mds_user": "admin", "mds_port": "3306", "mds_password": "Passw0rd!",
 "mds_name": "lefred-mysql"}' | fn invoke slow_query_log mysql_slowlog_txt
{"message": "MySQL Slow Log saved: slow_lefred-mysql_202210132124.log"}

We can see the files in Object Storage:

In the next article, we will see how to configure the API Gateway to call our application and store the statements on Object Storage.

And finally, we will see the content of those files and how to use them.

Stay tuned !

October 13, 2022

If like me you are an old experienced MySQL DBA, to analyze your MySQL workload, you certainly have used the slow query log with long_query_time set to 0.

The slow query log is a file that contains all the queries whose execution time is greater than the value of long_query_time. This file can be huge and uses up all available disk space on busy systems.

Constantly writing to that file can also add an overhead to the server.

For those reasons, the slow query log is not available in MySQL Database Service (HeatWave) in OCI.

Plan B ?

As a DBA, what are my options for finding the queries that need to be optimized? As usual, the queries that consume the most time are the ones that need some attention

It can be a very long query or a short query executed too many times.

Currently the MySQL DBAs use Performance_Schema to manage the queries executed on their database.

I already wrote an article on how to use Performance_Schema and Sys, let’s point out the query consuming most of the execution time (latency) with this new rewritten query (using new functions):

SQL> select schema_name, format_pico_time(total_latency) tot_lat,
            exec_count, format_pico_time(total_latency/exec_count)
            latency_per_call, query_sample_text
     from sys.x$statements_with_runtimes_in_95th_percentile as t1
     join performance_schema.events_statements_summary_by_digest as t2
       on t2.digest=t1.digest 
    where schema_name not in ('performance_schema', 'sys')
 order by (total_latency/exec_count) desc limit 1\G
*************************** 1. row ***************************
      schema_name: employees
          tot_lat: 21.54 s
       exec_count: 4
 latency_per_call: 5.38 s
query_sample_text: select * from salaries where salary > 80000
1 row in set (0.0127 sec)

Sys schema also contains a statement analysis view that can be used containing plenty of information to hunt the bad queries.

Let’s see an example to illustrate all the available information:

SQL> select * from sys.statement_analysis 
     where db not in ('performance_schema', 'sys') limit 1\G
*************************** 1. row ***************************
            query: SELECT `new_table` . `title` , ... `title` ORDER BY `salary` DESC
               db: employees
        full_scan: 
       exec_count: 11
        err_count: 0
       warn_count: 0
    total_latency: 38.96 s
      max_latency: 5.15 s
      avg_latency: 3.54 s
     lock_latency: 33.00 us
      cpu_latency:   0 ps
        rows_sent: 77
    rows_sent_avg: 7
    rows_examined: 13053117
rows_examined_avg: 1186647
    rows_affected: 0
rows_affected_avg: 0
       tmp_tables: 22
  tmp_disk_tables: 11
      rows_sorted: 77
sort_merge_passes: 0
           digest: 922701de9e5c51847f9f7de245b88fef4080b515ba8805082cd90c32830714eb
       first_seen: 2022-10-12 20:45:50.770465
        last_seen: 2022-10-13 11:49:13.140228
1 row in set (0.0022 sec)

Slow Query Log

But as I always say, old habits die hard… and sometimes people wants to deal with a slow query log anyway.

To generate something like this for example:

MySQL Database Server – queries in Grafana Loki

Or digest them offline with a tool like pt-query-digest that most of the MySQL DBAs have already used.

In the next article, I will show you how to use OCI Fn applications to generate a slow query log (in JSON or plain text) from MySQL Database Service and store it to Object Storage.

Stay tuned !

October 12, 2022

apple cider vinegar and rust

What one night in apple cider vinegar does to rust:

 

One night in recycled apple cider vinegar:

Een nacht in gerecycleerde appelciderazijn:

October 10, 2022

Today, at the peak of the biggest chess drama in years, I silently reached Elo 1300 on Chess.com.

Elo is the rating system used in chess, and Elo 1300 means I finally learned not to hang my pieces.

As a next goal, I'd like to get to Elo 1500. That would take me from "decent social player" to "beginning club player".

I shared this news with my family, and got something between a blank stare and an eye roll. I don't blame them.

If you think chess is boring, I encourage you to take a look at game 6 of the 2021 World Chess Championship.

I'll let you know when I get to Elo 1500. Being busy and all, that will likely take me at least nine months — if I can get there at all.

M5Stack ontwikkelt ESP32-microcontrollerbordjes met behuizing en scherm die je zo in je woonkamer zou kunnen zetten. Het zijn de ideale producten om een dashboard voor je smarthome van te maken. Met UIFlow is dat ook vrij eenvoudig te programmeren, op een grafische manier. Zo hoef je geen kaas gegeten te hebben van programmeercode.

Ik heb in en rond m'n huis allerlei temperatuursensoren staan. De binnen- en buitentemperatuur wilde ik ergens eenvoudig tonen. De M5Stack Core Ink leek me daarvoor ideaal: het is een ESP32-microcontrollerbordje met 1,54inch e-ink-scherm.

/images/uiflow-core-ink.jpg

Ik heb dit soort dashboards al op verschillende manieren geprogrammeerd: met Homepoint, ESPHome en openHASP. Nog een andere manier is UIFlow, een webgebaseerde ontwikkelomgeving voor de producten van M5Stack.

UIFlow ondersteunt alleen de hardware van M5Stack, maar geen willekeurig ESP32-microcontrollerbordjes. Het voordeel is dat de drivers van hun eigen producten allemaal ingebouwd zijn. Je kunt ook de grafische interface die je op het scherm van bijvoorbeeld de Core Ink wilt tonen 'tekenen' in de webinterface:

/images/uiflow-tekenen.png

Verder 'programmeer' je je software in Blockly, een grafische programmeertaal waarbij je blokjes met elkaar verbindt. Onderliggend wordt je Blockly-code overigens in MicroPython omgezet. Functionaliteit die niet in UIFlow is voorzien, kun je dan ook als MicroPython-code toevoegen.

In een artikel van me op id.nl lees je hoe ik de binnen- en buitentemperatuur van sensoren via MQTT inlees en op de Core Ink toon. De volledige code ziet er als volgt uit:

/images/uiflow-code.png

En 2007, espérant monétiser ma réputation naissante de blogueur influent, j’ai installé des publicités Google sur mon blog. Les premiers mois furent de bonnes surprises (entre 100€ et 200€ par mois) avant que Google ne m’annonce qu’en raison des clics frauduleux sur mes publicités, la somme serait revue à la baisse (pour tomber très vite à 30-50€ par mois).

Ne cliquant pas sur mes propres publicités et étant d’une naïve honnêteté, ma première réalisation fut que Google pouvait me raconter absolument n’importe quoi, qu’ils pouvaient décider de leur tarif, que je n’avais de toute façon aucun recours. Que 30€, c’était mieux que rien.

Vraiment ? À ce moment précis, je me fis la réflexion que tout modèle publicitaire était intenable sur le long terme. Que la publicité ne pouvait fonctionner que si elle était une fraction d’un véritable échange économique, mais que, toute seule, la publicité ne permet rien. Si une entreprise paye pour une publicité, c’est parce qu’elle espère augmenter son bénéfice d’une somme supérieure à celle dépensée.

Ce qui nous place devant deux alternatives :

Soit la publicité fonctionne et elle entraine que les personnes exposées (à savoir les lecteurs de mon blog) dépensent plus d’argent que s’ils n’y avaient pas été exposés.

Soit la publicité ne fonctionne pas vraiment et toute l’industrie n’est qu’une gigantesque arnaque de laquelle il vaut mieux rester le plus éloigné possible.

Dans le premier cas, l’affichage de publicités sur mon blog était hautement immoral (je poussais mes lecteurs à dépenser tout en n’en obtenant qu’un très maigre bénéfice), dans l’autre cas je participais à une arnaque qui risquait de s’écrouler rapidement.

Force est de constater que j’étais un peu trop en avance : quinze années plus tard, l’arnaque semble encore fonctionner. Mais les premières fissures semblent apparaitre dans la pyramide. Dans tous les cas, il est clair que la publicité est hautement immorale. Que toutes les personnes qui y contribuent d’une manière ou d’une autre sont de sacrés enfoirés. Oui. Vous ! Votre métier consiste à attirer le plus possible l’attention des gens, à polluer leur esprit afin qu’ils consomment plus et polluent plus la planète. Pourrir la vie des gens tout en pourrissant la planète.

La publicité en ligne, arnaque ou pas ?

Tim Hwang, dans « Subprime Attention Crisis », et Cory Doctorow, dans « How to destroy surveillance capitalism », défendent tout deux la théorie de l’arnaque. Comme le souligne Cory Doctorow, à lire les rapports aux actionnaires de Facebook, leur plateforme serait une espèce de rayon magique invisible pour manipuler l’esprit des gens. Hautement improbable !

Cependant la publicité a un effet très clair : attirer notre attention, nous distraire, détourner les idées et créer des envies. Il semblerait que la publicité ciblée ne fonctionne en réalité pas beaucoup mieux que des publicités génériques. Voire pas du tout mieux. Cela n’empêche pas Google et Facebook de nous espionner sous toutes les coutures en espérant faire croire à leurs clients que ça fonctionne.

Une chose est certaine : Google (Alphabet) et Facebook (Meta) sont parmi les cinq plus grandes entreprises du monde. Avec Twitter et TikTok, elles occupent le centre des vies connectées (et donc des vies tout court) de milliards d’individus. Il est impossible d’acheter un téléphone sans Google. Il est impossible de lire un article de presse sans liens vers Twitter (Rick Falkvinge se plaisait à appeler la presse « Printed tweets from yesterday »). En rue ou dans les transports, l’immense majorité de la population est en permanence sur Instagram (Meta), Whatsapp (Meta également) ou TikTok (une entreprise chinoise qui a largement démontré son irrespectabilité).

L’immense majorité du chiffre d’affaires de ces quatre sociétés est généré par la publicité. On parle de chiffres plus gros que le budget de tout un pays.

Postuler que la publicité est un business « honnête » revient à dire que ces milliards d’euros ne seraient pas dépensés sans elle. Et donc revient à démontrer que la publicité appauvrit la population d’une manière démentielle en entrainant une surconsommation qui détruit littéralement la planète.

Grâce au smartphone, la publicité a colonisé chaque seconde de nos vies. Il n’en reste pas moins que ces vies sont limitées à une poignée de milliards de secondes et que nos portefeuilles le sont également.

Force est de constater que la publicité est, au moins en partie, une gigantesque arnaque. L’autre partie étant un business parfaitement immoral en pleine compétition avec l’industrie de l’armement pour savoir qui entrainera le plus rapidement la disparition de l’humanité.

Des monopoles à la fois juge et partie

Comme je l’avais découvert avec Google en 2007, le véritable problème de ce marché publicitaire est que les entreprises sont à la fois arbitre et partie prenante. Lorsque vous achetez des publicités chez un de ces géants, tout ce qu’ils vous offrent en échange sont… des chiffres qu’ils inventent.

Pour promouvoir un de mes livres, j’avais fait l’expérience d’acheter des publicités sur Twitter et Facebook en spécifiant que je voulais cibler la Belgique. Sur Twitter, j’ai obtenu des milliers « d’impressions » (le terme pour dire que mon tweet a été affiché chez un utilisateur) et des dizaines de retweets. À l’inspection, tous ces retweets provenaient de comptes vraisemblablement faux (comptes sans interactions, dans une langue étrangère, mais, comme par hasard, contentant tous un drapeau belge dans leur description). De son côté, Facebook m’a informé que des centaines de personnes avaient cliqué sur le lien de ma campagne de financement. Chose étrange, aucun de ces clics n’apparaissait dans les statistiques du site de crowdfunding. L’impact sur les ventes a été absolument nul.

Ces réseaux sont tellement remplis de faux comptes et de fausses interactions que cela en devient gênant. Elon Musk a notamment fait trainer son offre de rachat de Twitter lorsqu’il s’est rendu compte de l’importance du problème (ce qui en dit long sur sa méconnaissance de ce type de business). LinkedIn (Microsoft) est confronté à un problème similaire : la majorité des interactions y sont fausses. Même vos statistiques Google Analytics sont majoritairement remplies de visites de robots.

La situation est simple : nous voulons tous plus de « clics » sur nos contenus et des followers, les grandes entreprises ont pris le contrôle du Web pour nous offrir ces clics et ces followers. Ils nous offrent des compteurs de clics et un moyen de faire augmenter ces clics avec une simple carte de crédit. Ils nous vendent une monnaie de singe de leur propre inventions contre notre argent véritable, un peu comme certaines sociétés de jeux en ligne. Tant qu’il y’a des pigeons pour payer, pourquoi s’en priver ?

Le géant Procter&Gamble a d’ailleurs annoncé avoir coupé complètement tous ses budgets publicitaires pour les réseaux sociaux et n’avoir perçu absolument aucune différence dans les ventes.

La faible qualité de l’attention vendue

Tout n’est pas complètement faux. Facebook s’est fait une spécialité de vous permettre de créer une « communauté », mais de vous forcer à payer pour envoyer des messages à une partie de cette communauté. Avec mes 2500 followers sur Facebook, j’avais découvert à l’époque que chaque message touchait en moyenne 1%, mais que, en payant, je pouvais monter à 5% voire 10% de « ma communauté ». Facebook me faisait croire que j’atteignais un public alors qu’en réalité, je devais payer pour contacter moins de gens que si je leur avais conseillé de s’abonner par mail ou RSS.

Mais même lorsque payer permet réellement d’entrer en contact avec un être humain, cette interaction est généralement d’une qualité incroyablement faible. Dans le train, le bus ou l’avion, observez l’écran des autres voyageurs. Ils font défiler à toute vitesse, s’arrêtant parfois une seconde pour mettre un like sans même prendre le temps de lire. Lorsque la publicité n’est pas une simple arnaque, tout ce qu’elle vous offre est donc d’apparaitre quelques fractions de seconde sur un écran en train de défiler.

Sur LinkedIn, j’ai durant quelques mois tenté d’accepter les offres d’emplois qui m’arrivaient par messages privés. Alors que pour les besoins de l’expérience j’acceptais toutes les conditions sans discuter, aucune des dizaines de demandes n’a jamais débouché sur quoi que ce soit. En fait, je n’ai reçu que quelques réponses à mes dizaines d’acceptations et, malgré des relances de ma part, toutes ces discussions ont tourné en eau de boudin, mes interlocuteurs étant passés à autre chose.

Même lorsque les géants publicitaires remplissent leur part du contrat de manière honnête, force est de constater qu’il s’agit d’une arnaque.

La morbidité des métriques

Comme d’habitude, tout n’est que question de métrique. Si vous êtes dans un job où votre chef vous demande d’augmenter les clics sur un site web ou d’obtenir des followers, alors cela a complètement du sens d’acheter ces clics ou ces followers.

Mais si vous avez un peu plus de recul, la véritable question est « Quel est l’objectif du business ? » et « Quelles sont les métriques véritablement corrélées à cet objectif ».

Ce n’est pas parce que Google, Facebook et les autres vous offrent un beau pack de métriques toutes faites et d’un moyen de les optimiser que cela convient à votre business. En fait, de manière générale, tout fournisseur qui prétend vous vendre l’observation de métriques décidées par lui, mesurées par lui et optimisées par lui est par essence un escroc.

À part dans de rares cas, il est même probable que le fait d’avoir un Google Analytics sur votre site soit préjudiciable à votre business. Le simple fait d’avoir accès à ces statistiques va vous faire prendre des décisions pour augmenter le nombre de visites sur votre site, ce qu’on appelle SEO. Ensuite, vous allez tenter de comprendre pourquoi vos visiteurs qui affluent ne passent pas commande alors que vos techniques de SEO ont justement fait affluer une masse de gens (lorsque ce ne sont pas des robots) qui ne sont a priori pas concernés par votre business. En essayant d’être trouvé par tout le monde, vous perdez de vue le cœur de votre marché, à savoir ceux qui vous cherchent.

L’omniprésence des métriques web fournies par Facebook, Google et compagnie fait que plus personne n’imagine ne pas optimiser ces métriques, même les services publics, les écoles, les restaurants, les business de proximité, les entreprises qui ont une relation humaine avec leurs clients. Bref, l’immense majorité de l’économie.

Échapper aux influences néfastes

C’est ce qui a fait la fortune de Google et Facebook : ils ont confisqué l’économie, l’ont pris en otage en convainquant le monde entier qu’il n’y avait pas d’autre manière de faire du business qu’en augmentant les clics sur le web. Et que la seule manière d’y arriver, c’est de passer par leur monopole.

Tant qu’il y’aura des pigeons pour payer des publicités, ces entreprises survivront. Même si les publicités sont pour promouvoir des services vivant eux-mêmes de la publicité ? Sur Android, par exemple, les jeux font souvent la publicité d’autres jeux publicitaires. Sur Facebook, beaucoup de médias, y compris ceux financés par l’argent public, font leur publicité afin d’attirer des visiteurs pour visionner… leurs propres publicités.

Il n’est pas compliqué de comprendre que tout cela tourne en boucle et que le jour où tout cela s’écroulera, ce ne sera pas très beau à voir. Que tous vos investissements pour obtenir des abonnés, pour obtenir des vues et des clics seront réduits à néant.

Briser l’incroyable puissance de ces monopoles morbides ne passe pas par l’utilisation d’alternatives ou de succédanés, mais par la réalisation profonde que nous n’avons tout simplement pas besoin d’eux.

Qu’ils ne survivent qu’en nous faisant croire à la fable qu’ils sont indispensables.

Que sous les apparences de technologie et de scientisme, ces business ne sont que de simples religions qui contrôlent ceux qui veulent bien y croire, qui extorquent l’argent de leurs fidèles en leur faisant croire que c’est la seule option viable pour gagner le paradis.

Recevez les billets par mail ou par RSS. Max 2 billets par semaine, rien d’autre. Adresse email jamais partagée et définitivement effacée lors du désabonnement. Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, lisez, offrez et partagez des livres.

Ce texte est publié sous la licence CC-By BE.

October 09, 2022

Yesterday I explained how to make a scrap computer do OCR on your scanner/printer’s scanned PDFs in case you have a SMB file share (a Windows file share) where the printer will write to.

I also promised I would make the E-Mail feature of the printer send E-mails with the PDFs in that E-mail being OCR scanned.

I had earlier explained how you can make your old scanner/printer support modern SMTP servers that have TLS, by introducing a scrap computer running Postfix to forward the E-mails for you. This article depends on that, of course. As we will let the scrap computer now do the OCR part. If you have not yet done that, first do it before continuing here.

I looked at Xavier Merten‘s CockooMX, and decided to massacre it until it would do what I want it to do. Namely call ocrmypdf on the application/pdf attachments and then add the resulting PDF/A (which will have OCR text) to the E-mail.

First install some extra software: apt-get install libmime-tools-perl . It will provide you with MIME::Tools, we will use MIME::Parser and MIME::Entity.

Create a Perl script called /usr/local/bin/ocrpdf.pl (chmod 755 it) that looks like this (which is Xavier’s CockooMX massacred and reduced to what I need – Sorry Xavier. Maybe we could try to make CockooMX have a plugin like infrastructure? But writing what looks suspicious to a database ain’t what I’m aiming for here):


#!/usr/bin/perl

# Copyright note

use Digest::MD5;
use File::Path qw(make_path remove_tree);
use File::Temp;
use MIME::Parser;
use Sys::Syslog;
use strict;
use warnings;

use constant EX_TEMPFAIL => 75; # Mail sent to the deferred queue (retry)
use constant EX_UNAVAILABLE => 69; # Mail bounced to the sender (undeliverable)

my $syslogProgram	= "ocrpdf";
my $sendmailPath	= "/usr/sbin/sendmail";
my $syslogFacility	= "mail";
my $outputDir		= "/var/ocrpdf";
my $ocrmypdf		= "/usr/bin/ocrmypdf";

# Create our working directory
$outputDir = $outputDir . '/' . $$;
if (! -d $outputDir && !make_path("$outputDir", { mode => 0700 })) {
  syslogOutput("mkdir($outputDir) failed: $!");
  exit EX_TEMPFAIL;
}

# Save the mail from STDIN
if (!open(OUT, ">$outputDir/content.tmp")) {
  syslogOutput("Write to \"$outputDir/content.tmp\" failed: $!");
  exit EX_TEMPFAIL;
}
while() {
  print OUT $_;
}
close(OUT);

# Save the sender & recipients passed by Postfix
if (!open(OUT, ">$outputDir/args.tmp")) {
  syslogOutput("Write to \"$outputDir/args.tmp\" failed: $!");
  exit EX_TEMPFAIL;
}
foreach my $arg (@ARGV) {
  print OUT $arg . " ";
}
close(OUT);

# Extract MIME types from the message
my $parser = new MIME::Parser;
$parser->output_dir($outputDir);
my $entity = $parser->parse_open("$outputDir/content.tmp");

# Extract sender and recipient(s)
my $headers = $entity->head;
my $from = $headers->get('From');
my $to = $headers->get('To');
my $subject = $headers->get('Subject');
chomp($from);
chomp($subject);

syslogOutput("Processing mail from: $from ($subject)");

processMIMEParts($entity);
deliverMail($entity);
remove_tree($outputDir) or syslogOuput("Cannot delete \"$outputDir\": $!");

exit 0;

sub processMIMEParts
{
  my $entity = shift || return;
  for my $part ($entity->parts) {
    if($part->mime_type eq 'multipart/alternative' ||
       $part->mime_type eq 'multipart/related' ||
       $part->mime_type eq 'multipart/mixed' ||
       $part->mime_type eq 'multipart/signed' ||
       $part->mime_type eq 'multipart/report' ||
       $part->mime_type eq 'message/rfc822' ) {
         # Recursively process the message
         processMIMEParts($part);
     } else {
       if( $part->mime_type eq 'application/pdf' ) {
         my $type = lc  $part->mime_type;
         my $bh = $part->bodyhandle;
         syslogOutput("OCR for: \"" . $bh->{MB_Path} . "\" (" . $type . ") to \"" . $bh->{MB_Path} . ".ocr.pdf" . "\"" );
         # Perform the OCR scan, output to a new file
         system($ocrmypdf, $bh->{MB_Path}, $bh->{MB_Path} . ".ocr.pdf");
         # Add the new file as attachment
         $entity->attach(Path   => $bh->{MB_Path} . ".ocr.pdf",
                         Type   => "application/pdf",
                         Encoding => "base64");
      }
     }
   }
   return;
}

#
# deliverMail - Send the mail back
#
sub deliverMail {
  my $entity = shift || return;

  # Write the changed entity to a temporary file
  if (! open(FH, '>', "$outputDir/outfile.tmp")) {
    syslogOutput("deliverMail: cannot write $outputDir/outfile.tmp: $!");
    exit EX_UNAVAILABLE;
  }
  $entity->print(\*FH);
  close(FH);

  # Read saved arguments
  if (! open(IN, "$outputDir/args.tmp")) {
    syslogOutput("deliverMail: Cannot read $outputDir/args.tmp: $!");
    exit EX_TEMPFAIL;
  }
  my $sendmailArgs = ;
  close(IN);
	
  # Read mail content from temporary file of changed entity
  if (! open(IN, "$outputDir/outfile.tmp")) {
    syslogOutput("deliverMail: Cannot read $outputDir/content.txt: $!");
    exit EX_UNAVAILABLE;
  }
	
  # Spawn a sendmail process
  syslogOutput("Spawn=$sendmailPath -G -i $sendmailArgs");
  if (! open(SENDMAIL, "|$sendmailPath -G -i $sendmailArgs")) {
    syslogOutput("deliverMail: Cannot spawn: $sendmailPath $sendmailArgs: $!");
    exit EX_TEMPFAIL;
  }
  while() {
    print SENDMAIL $_;
  }
  close(IN);
  close(SENDMAIL);
}

#
# Send Syslog message using the defined facility
#
sub syslogOutput {
  my $msg = shift or return(0);
  openlog($syslogProgram, 'pid', $syslogFacility);
  syslog('info', '%s', $msg);
  closelog();
}

Now we just do what Xavier’s CockooMX documentation also tells you to do: add it to master.cf:

Create a UNIX user: adduser ocrpdf

Change the smtp service:

smtp      inet  n       -       -       -       -       smtpd
-o content_filter=ocrpdf

Create a new service

ocrpdf  unix  -       n       n       -       -       pipe
user=ocrpdf argv=/usr/local/bin/ocrpdf.pl -f ${sender} ${recipient}

October 08, 2022

Modern printers can do OCR on your scans. But as we talked about last time, aren’t all printers or scanners modern.

We have a scrap computer that is (already) catching all E-mails on a badly configured local SMTP server, to then forward it to a well configured SMTP server that has TLS. Now we also want to do OCR on the scanned PDFs.

My printer has a so called Network Scan function that scans to a SMB file share (that’s a Windows share). The scrap computer is configured to share /var/scan using Samba as ‘share’, of course. The printer is configured to use that share. Note that you might need in smb.conf this for very old printers:

client min protocol = LANMAN1
server min protocol = LANMAN1
client lanman auth = yes
client ntlmv2 auth = no
client plaintext auth = yes
ntlm auth = yes
security = share

And of course also something like this:

[scan]
path = /var/scan
writable = yes
browsable = yes
guest ok = yes
public = yes
create mask = 0777

First install software: apt-get install ocrmypdf inotify-tools screen bash

We need a script to perform OCR scan on a PDF. We’ll here use it in another script that monitors /var/scan for changes. Later in another post I’ll explain how to use it from Postfix’s master.cf on the attachments of an E-mail. Here is /usr/local/bin/fixpdf.sh:

! /bin/sh
a=$1
TMP=`mktemp -d -t XXXXX`
DIR=/var/scan
mkdir -p $DIR/ocr
cd $DIR
TIMESTAMP=`stat -c %Y "$a"`
ocrmypdf --force-ocr "$a" "$TMP/OCR-$a"
mv -f "$TMP/OCR-$a" "$DIR/ocr/$TIMESTAMP-$a"
chmod 777 "$DIR/ocr/$TIMESTAMP-$a"
cd /tmp
rm -rf $TMP

Note that I prepend the filename with a timestamp. That’s because my printer has no way to give the scanned files a good filename that I can use for my archiving purposes. You can of course do this different.

Now we want a script that monitors /var/scan and launches that fixpdf.sh script in the background each time a file is created.

My Xerox WorkCentre 7232 uses a directory called SCANFILE.LCK/ for its own file locking. When it is finished with a SCANFILE.PDF it deletes that LCK directory.

Being bad software developers the Xerox people didn’t use a POSIX rename for SCANFILE.PDF to do an atomic write operation at the end.

It looks like this:

inotifywait -r -m  /var/scan | 
while read file_path file_event file_name; do
echo ${file_path}${file_name} event: ${file_event}
done
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/var/scan/ event: OPEN,ISDIR
/var/scan/ event: ACCESS,ISDIR
/var/scan/ event: ACCESS,ISDIR
/var/scan/ event: CLOSE_NOWRITE,CLOSE,ISDIR
/var/scan/ event: OPEN,ISDIR
/var/scan/ event: ACCESS,ISDIR
/var/scan/ event: ACCESS,ISDIR
/var/scan/ event: CLOSE_NOWRITE,CLOSE,ISDIR
/var/scan/XEROXSCAN003.LCK event: CREATE,ISDIR
/var/scan/XEROXSCAN003.LCK event: OPEN,ISDIR
/var/scan/XEROXSCAN003.LCK event: ACCESS,ISDIR
/var/scan/XEROXSCAN003.LCK event: CLOSE_NOWRITE,CLOSE,ISDIR
/var/scan/ event: OPEN,ISDIR
/var/scan/ event: ACCESS,ISDIR
/var/scan/ event: ACCESS,ISDIR
/var/scan/ event: CLOSE_NOWRITE,CLOSE,ISDIR
/var/scan/ event: OPEN,ISDIR
/var/scan/ event: ACCESS,ISDIR
/var/scan/ event: CLOSE_NOWRITE,CLOSE,ISDIR
/var/scan/XEROXSCAN003.PDF event: CREATE
/var/scan/XEROXSCAN003.PDF event: OPEN
/var/scan/XEROXSCAN003.PDF event: MODIFY
/var/scan/XEROXSCAN003.PDF event: MODIFY
...
/var/scan/XEROXSCAN003.PDF event: MODIFY
/var/scan/XEROXSCAN003.PDF event: MODIFY
/var/scan/XEROXSCAN003.PDF event: CLOSE_WRITE,CLOSE
/var/scan/XEROXSCAN003.PDF event: ATTRIB
/var/scan/XEROXSCAN003.LCK event: OPEN,ISDIR
/var/scan/XEROXSCAN003.LCK/ event: OPEN,ISDIR
/var/scan/XEROXSCAN003.LCK event: ACCESS,ISDIR
/var/scan/XEROXSCAN003.LCK/ event: ACCESS,ISDIR
/var/scan/XEROXSCAN003.LCK event: ACCESS,ISDIR
/var/scan/XEROXSCAN003.LCK/ event: ACCESS,ISDIR
/var/scan/XEROXSCAN003.LCK event: CLOSE_NOWRITE,CLOSE,ISDIR
/var/scan/XEROXSCAN003.LCK/ event: CLOSE_NOWRITE,CLOSE,ISDIR
/var/scan/XEROXSCAN003.LCK/ event: DELETE_SELF
/var/scan/XEROXSCAN003.LCK event: DELETE,ISDIR

The printer deleting that SCANFILE.LCK/ directory is a good moment to start our OCR script (call it for example /usr/local/bin/monitorscan.sh):

! /bin/bash
inotifywait -r -m -e DELETE,ISDIR /var/scan |
while read file_path file_event file_name; do
if [ ${file_event} = "DELETE,ISDIR" ]; then
if [[ ${file_name} == *"LCK" ]]; then
suffix=".LCK"
filename=`echo ${file_name} | sed -e "s/$suffix$//"`.PDF
/usr/local/bin/fixpdf.sh $filename &
fi
fi
done

Give both scripts 755 permissions with chmod and now you just run screen /usr/local/bin/monitorscan.sh

When your printer was written by good software developers, it will do POSIX rename. That looks like this (yes, also when done over a SMB network share):

inotifywait -r -m  /var/scan | 
while read file_path file_event file_name; do
echo ${file_path}${file_name} event: ${file_event}
done
Setting up watches. Beware: since -r was given, this may take a while!
Watches established.
/var/scan/ event: OPEN,ISDIR
/var/scan/ event: ACCESS,ISDIR
/var/scan/ event: CLOSE_NOWRITE,CLOSE,ISDIR
/var/scan/ event: OPEN,ISDIR
/var/scan/ event: ACCESS,ISDIR
/var/scan/ event: CLOSE_NOWRITE,CLOSE,ISDIR
/var/scan/ event: OPEN,ISDIR
/var/scan/ event: ACCESS,ISDIR
/var/scan/ event: CLOSE_NOWRITE,CLOSE,ISDIR
/var/scan/ event: OPEN,ISDIR
/var/scan/ event: ACCESS,ISDIR
/var/scan/ event: CLOSE_NOWRITE,CLOSE,ISDIR
/var/scan/.tmp123.GOODBRANDSCAN-123.PDF event: CREATE
/var/scan/.tmp123.GOODBRANDSCAN-123.PDF event: OPEN
/var/scan/.tmp123.GOODBRANDSCAN-123.PDF event: MODIFY
...
/var/scan/.tmp123.GOODBRANDSCAN-123.PDF event: MOVED_FROM
/var/scan/GOODBRANDSCAN-123.PDF event: MOVED_TO

That means that your parameters for inotifywait could be -r -m -e MOVED_TO and in ${file_name} you’ll have that GOODBRANDSCAN-123.PDF. This is of course better than Xerox’s way with their not invented here LCK things that probably also wouldn’t be necessary with a POSIX rename call.

I will document how to do this to the E-mail feature of the printer with Postfix later.

I first need a moment in my life where I actually need this hard enough that I will start figuring out how to extract certain attachment MIME parts from an E-mail with Posix’s master.cf. I guess I will have to look into CockooMX by Xavier Mertens for that. Update: that article is available now.

.

October 06, 2022

It's been a while since I wrote about my views on the Content Management System (CMS) and Digital Experience Platform (DXP) markets.

It's also been a while since I gave an update on Acquia's product vision and strategy, like I did in 2009, 2011 or 2017.

In this post, I want to give you an update on both. I will share my views on the Digital Experience Platform market and talk more about Acquia's vision and strategy.

Throughout the article, I will also give practical examples of how Drupal (the Open Source project I started) and Acquia (the company I started) fit into a 'Composable Enterprise'.

The Composable Enterprise

The Composable Enterprise has been one of the most important trends in the software industry. Analyst firm Gartner popularized the term, and defines the Composable Enterprise as follows:

A Composable Enterprise is an organization that can innovate and adapt to changing business needs through the assembly and combination of packaged business capabilities. To enable the Composable Enterprise, organizations will need to adapt the way they source and deliver applications as vendors deliver more modular capabilities.

The main idea behind the Composable Enterprise is that when organizations compose or assemble solutions from existing building blocks, they are able to move faster. It also provides them the flexibility to adjust to changing business needs.

Per Gartner's thought-leadership, architectural modularity is key to composability. But composability is also much more than modularity. Composability defines an end-to-end approach, and not only a software architecture. Composability is a philosophy of business agility, architecture and governance.

A Composable Digital Experience Platform

In this article, I will apply the ideas of the Composable Enterprise to the DXP market.

Specifically, I will expand upon six core tenets that I believe are key to Composable DXPs:

  1. Principle 1: Software architecture needs to be modular
  2. Principle 2: Components need to be discoverable and orchestrated
  3. Principle 3: All business stakeholders need to be empowered with low-code / no-code
  4. Principle 4: Data makes the difference, but it needs to be unified and automated
  5. Principle 5: Multi-experience content demands strong content management
  6. Principle 6: A platform approach requires diverse experience composition and delivery methods

MACH and Jamstack

Before jumping into the six key principles, I wanted to acknowledge that composability in the DXP market has given rise to trends like MACH and Jamstack.

  • MACH stands for Microservices, API-first, Cloud-native SaaS, Headless. In a MACH architecture, each piece of business functionally is a cloud service. The cloud services are built and managed by independent vendors and integrated by the customer. The typical MACH service is a backend service with either no frontend (headless service) or a decoupled frontend. As a result, MACH services offer web service APIs for integration and frontend development.
  • Jamstack stands for JavaScript, APIs and Markup. Jamstack is an architectural approach that decouples the web experience layer from data and business logic. The web experience layer is often pre-rendered as static pages (markup), served via a CDN.

In short, MACH and Jamstack primarily describe architectural approaches. MACH is mostly about the backend architecture of web applications, and Jamstack is mostly about the frontend architecture of web applications.

Both MACH and Jamstack represent developer-centric approaches to the challenge of composability. Neither are prescriptive about the capabilities of a DXP. Both MACH and Jamstack can be part of a Composable DXP.

Principle 1: Software architecture needs to be modular

Hand-drawing of six connected building blocks. Each building block represents a feature.Composing an experience by assembling different building blocks.

At the core of a Composable DXP are modular software design principles. Organizations should reject software monoliths and focus on software that is modular.

Software monoliths are inflexible. Monoliths prevent teams from moving fast, hinder innovation and make it harder to deliver digital products and services.

Modular software is decomposed into smaller pieces with standardized interfaces. Modular software allows solutions to be created by combining reusable chunks of code.

This idea is not new — modular software has been around since the 1960s — but a lot of software still doesn't live up to these basic design principles.

Drupal proof points

Open Source software is almost always better than proprietary software with regards to APIs and modularity.

For over 10 years, Drupal has been pushing the concept of the Assembled Web.

Today, Drupal's open and modular architecture allows over 10,000 active contributors to build and maintain 46,000 modules for the more than 1 million websites running Drupal.

Integrations include third-party commerce platforms, digital asset management platforms, analytics platforms, CRM systems, marketing automation software, frontend frameworks, and many more.

Principle 2: Components need to be discoverable and orchestrated

Hand-drawing of two robots operating components on a conveyor belt.Component discovery and maintenance.

A Composable Architecture is an architecture in which the individual components of the stack can be replaced without affecting other parts of the system.

The idea is to enable organizations to select and assemble a customized solution that best fits their needs.

Combining components from different providers gives organizations a lot of flexibility, but it also comes with added cost and complexity:

  • Components can be hard to discover; e.g. what are all the available components for my DXP, what do they do and how do I use them?
  • Components can be hard to orchestrate; e.g. some components might not work together, components might have dependencies, etc.

A software stack that consists of many components can be hard to build and maintain. In fact, Composable Architectures shift much of the discovery, orchestration, integration and testing burden from vendors to the end user.

Composability necessitates an approach that simplifies the discovery, installation, assembly, and maintenance of components. A good Composable DXP offers the following solutions to manage diverse components:

  • Packaged Business Capabilities (PBCs) combine components and their configuration into higher-level building blocks. This is helpful to make solution delivery more repeatable.
  • A Marketplace where both components and PBCs can be searched for and discovered.
  • A Component Manager to provide a unified mechanism to install and update components from different providers. The Component Manager resolves component dependencies (i.e. component A requires component B to be installed as well) and manages component compatibility (i.e. component A only works with v2 of component B but not yet with v3). A Component Manager also knows when there is a new version of a component (e.g. component B has a security release and needs to be updated) and orchestrates its update (e.g. updating component B for security reasons, requires updating component C as well).
  • A Component Catalog is a registry or database of components and the metadata required to manage components. This includes the component's location, version information, dependency chain, compatibility restrictions, release details, etc. Marketplaces and Component Managers operate on top of one or more Component Catalogs.
  • A CI/CD Pipeline for continuous integration, management and updating of components. New versions of components are released every day by independent providers. Automated integration and automated testing is required to deliver composable software reliably and quickly.

Drupal and Acquia proof points

Drupal has 46,000 components, called modules. Drupal also offers more than 1,000 PBCs, which Drupal calls distributions and recipes. Distributions and recipes not only combine modules, but also ship with data schemas, configuration, content and data to make everything work well together. Example distributions include:

Modules (components) and distributions (PBCs) can be searched, browsed and filtered in at least three ways:

Since the release of Drupal 8 in 2015, Drupal sites are managed with Composer (a Component Manager) and Packagist (a Component Catalog). Composer downloads and installs modules, along with any third-party dependencies. Composer also takes care of updates, version management, compatibility management and dependency management.

Last but not least, Acquia Code Studio offers a CI/CD pipeline. It continuously checks for new releases of components. When there is an update, it will install the new version and test it in a staging environment. The automated tests include integration testing, security testing, unit testing, performance testing and more.

Principle 3: All business stakeholders need to be empowered with low-code / no-code

Hand-drawing of a person moving UI components in front of a very large screen.No-code lets you deliver customer experience faster.

To deliver the best customer experiences, all departments (engineering, marketing, sales, customer success and HR) must participate in the creation of these experiences.

Being modular and component-driven is great for developers, but it doesn't necessarily enable other business stakeholders to take part in the experience creation.

This is where low-code and no-code solutions come in. Low-code and no-code technologies use a graphical user interface (GUI) to speed up experience building. They empower every business stakeholder to create customer experiences without the help of software developers.

A Composable DXP with low-code / no-code helps organizations in at least three ways:

  1. When separate business functions own their technology, rather than relying on inside or outside engineering teams, they can move faster.
  2. Low-code / no-code solutions free up internal development teams to focus on unique, differentiated innovation. The low-code / no-code value proposition isn't exclusive to business users; it also speeds up development teams.
  3. Low-code / no-code solutions reduce an organization's reliance on software engineering capacity and skills that can be harder to find.

In short, low-code / no-code solutions enable cross-functional teams to deliver great customer experiences faster.

Drupal and Acquia proof points

For low-code / no-code to be effective across all of an experience's creation, it needs to be available throughout the Composable DXP; from the content layer, to the data layer, to the orchestration layer.

Content layer
  • Drupal's UI enables developers and non-developers to compose or assemble websites without having to write code. Over the past 20+ years, Drupal has helped democratize website design using its low-code / no-code approach.
  • Acquia Site Studio makes Drupal even more accessible to business users and cross-functional teams. For example, Acquia Site Studio enables organizations to set up a composable design system. Organizations can design visual components (e.g. header, footer, gallery, buttons, etc.) using no-code tools, organize them in a catalog, and use them across multiple websites.
Data layer
  • Acquia's Customer Data Platform (CDP) provides 300 out-of-the-box connectors along with low-code tooling to quickly create custom integrations. Using a drag-and-drop UI, organizations can build simple to sophisticated data integrations. Despite being low-code, these integrations can handle complex data transformations, data deduplication, conditional triggers, and much more.
Orchestration layer
  • Acquia Campaign Studio allows organizations to create customer journeys. Engaging workflows, beautiful emails and landing pages can be built in minutes with an easy-to-use, drag-and-drop interface.

Principle 4: Data makes the difference, but it needs to be unified and automated

Hand-drawing of 3 rockets taking off. The rocket with the label 'unified data' goes the fastest and is the furthest along.Unified data takes you further!

Organizations are increasingly using data to create superior, personalized customer experiences. The goal is simple: improve customer satisfaction, loyalty and advocacy via tailored experiences.

Tomorrow's applications will consume data from multiple sources to develop a fine-grained user profile for each user. In turn, these user profiles are used for personalization.

The first problem is that customer data is everywhere: in CRM systems, accounting systems, websites, marketing tools, commerce systems, point of sale systems, analytics platforms, customer support software, mobile applications, chatbots, call center software, and more.

Despite the clear need to tap all data sources, data is locked up in different databases and in disparate formats. Data silos are one of the biggest barriers to delivering personalized experiences.

To use data to its fullest potential, data needs to be unified. It needs to break free from these silos.

The second problem is scalability. Humans can't manually compose an experience for every single single customer and every single customer interaction.

When you have millions of user profiles and millions of interactions, automation is key. Organizations must rely on machine learning algorithms to tailor experiences to people's preferences.

Last but not least, it goes without saying that organizations also need and want to respect their customers' data privacy, and remain in compliance with regulations like GDPR and other local data privacy laws. This adds a third layer of complexity to managing and using user data.

Drupal and Acquia proof points

Acquia's Customer Data Platform (CDP) helps companies manage both consumer privacy and deliver personalized experiences at scale.

First, by integrating different data sources, Acquia CDP provides all teams with direct access to unified customer data. Data is integrated, cleaned up, and de-duped.

Second, Acquia's machine learning platform helps teams use that data to deliver personalized experiences or drive targeted campaigns. Acquia delivers over 1 trillion machine learning recommendations a year, or 3 billion personalization recommendations a day.

With Acquia CDP, users can easily leverage pre-built families of predictions, personas or next-best experience models such as 'Likelihood to pay full price', 'Product affinity segments' or 'Next-best channel' models. Users can also implement, manage and publish their own custom ML models.

Principle 5: Multi-experience content demands strong content management

Hand-drawing of a DJ table specialized in mixing content rather than mixing music.Remix content and deploy it to different channels.

Multi-experience refers to a user's end-to-end experience with one organization across a variety of digital touchpoints — websites, mobile applications, chatbots, voice assistants, wearables, augmented reality, metaverses, and more.

Great content is at the core of any great (multi-)experience. Quality content helps organizations stand out from the competition. This raises expectations for organizations to create content that customers care about, and to deliver that content on the channels they prefer to use. Composable content is key.

A Composable DXP should provide a Content Platform designed to address the challenges of managing content across all channels. These challenges include:

  • Streamlining content management and content operations by eliminating content silos.
  • Leveraging content models and headless delivery capabilities to provide content across channels and customer touchpoints.
  • Improving content governance and brand consistency across all customer touchpoints.

To address these challenges, a Content Platform needs the following core capabilities:

  • Content modeling – Breaks content up into common elements so they can be shared and remixed for different channels. It also enables content to be assembled and reassembled quickly, based on data-driven insights into each customer's particular interests. This includes models for articles, blogs, pages, etc., along with atomized digital assets and product information.
  • Content operations – Organizes all people and processes responsible for strategizing, creating, distributing and analyzing content.
  • Content delivery – Delivers content to different channels. A Content Platform acts as a content repository that makes content available to different channels using web services (headless) and/or a traditional presentation layer (coupled).
  • Content governance – Ensures that content is consistent across all engagement channels. Engagement channels can't be disjointed or siloed.
  • Content personalization – Personalizes content in real time. Personalization can be based on anonymous behavioral actions and/or explicit preferences in the customer profile.
  • Journey orchestration – Seamlessly coordinates customer journeys from inbound to outbound touchpoints, and from native to federated channels. Journey orchestration acts on real-time events, inferred insights, decisioning through machine learning and business rules to deliver next-best actions in-moment for every individual at scale.

Composable content does not necessarily mean headless CMSes are de facto the best Content Platform. Headless solutions have pros and cons relative to alternative approaches:

  • A Traditional CMS provides both a content repository and no-code tools to build experiences with. Traditional CMSes empower marketers and developers to build web experiences using a UI. Because traditional CMSes do not have strong web service APIs, they have limited ability to push content to multiple channels.
  • A Headless CMS is primarily a content repository, and generally lacks no-code tools for building digital experiences. Headless CMSes excel at pushing content to multiple channels. They are frontend agnostic and require developers to build custom frontends. A headless CMS often leaves marketers at a disadvantage, but is more flexible for developers.
  • A Hybrid CMS offers headless capabilities, but also comes with an optional out-of-the-box frontend. It provides developers with the web service APIs required to deliver content across channels, while at the same time ensuring marketers have no-code tools to build digital experiences.

Nearly all Traditional CMSes have evolved to be Hybrid CMSes. It's no longer relevant to talk about Traditional CMSes. Today, the choice is really between Headless and Hybrid.

Drupal and Acquia proof points

  • Drupal is decidedly a Hybrid CMS. Drupal evolved from a Traditional CMS to a Hybrid CMS in 2012, more than 10 years ago, well before the term 'headless' became popular. Today, you can use Drupal as a Traditional CMS, a Headless CMS or a combination of both. Drupal is API-first, but not API-only.

    When Drupal is used in headless mode, organizations can use their preferred JavaScript framework to build a frontend. This includes React, Next.js, Vue, Svelte, Angular and more.

    • Example 1: Drupal integrates with Gatsby. Gatsby enables developers to build static websites using React. The content for the React website is provided by Drupal.
    • Example 2: Drupal has excellent Next.js integration that includes support for JSON:API, GraphQL, authentication, seamless editing, live previews, form building, search, internationalization, and much more.

    To streamline the building of headless applications with Drupal, Acquia provides headless SDKs, Node.js hosting, and more.

  • Acquia DAM offers deep governance across distributed marketing teams for properly managing digital assets and storing rich product information. It provides brand automation that ensures brand consistency across different channels.

Principle 6: A platform approach requires diverse experience composition and delivery methods

Hand-drawing of a person juggling different dimensions. Dimensions are balls with labels: cost, simple, complex, etc.Juggling different experience composition and delivery options.

If you're like most organizations, the number of digital experience applications you have continues to grow, not shrink.

Different sites also have different scale, functionality, complexity and longevity. Some experiences are continuously developed, while others are only around for a few months. Some are built by IT, others by marketing. Some sites get a thousand visitors a month, others get 100 million visitors a month.

When managing a portfolio of digital experiences, a one-size-fits-all approach simply doesn't work. Organizations need to balance development approaches and operational costs.

For one website, an organization might want to use a marketer-friendly page builder and static site hosting. For another website, the same organization might require a headless CMS and PaaS delivery platform.

Furthermore, CIOs and CMOs often face cost-cutting and acceleration pressures at the same time. They are constantly having to figure out how to do more with less.

So how do you manage a diverse portfolio of digital experience applications, maximize velocity, minimize cost and meet growing security demands — all without sacrificing quality?

This is done by standardizing on a platform or ecosystem that scales from small to large, from simple to complex, and from IT to marketing. Specifically, this type of platform lets you choose between different experience composition and delivery options:

  • Experience composition options – A platform that offers both APIs and no-code tooling; from web service APIs, to templates, to page builders, to different frontend JavaScript frameworks.
  • Experience delivery options – A platform that can deploy static sites or dynamic sites on either SaaS or PaaS.

These different composition and delivery models also need to be underpinned by a standard set of shared services:

  • Services that enable the sharing of content.
  • Services that enable the sharing of data.
  • Services that streamline cloud-native delivery.
  • Services that encourage consistent operational workflows, including release management, deployment, security and compliance best-practices.
  • Services to manage a global catalog of components, PBC, and design systems.
  • Services to manage a global portfolio of sites and digital experiences.

In short, managing a portfolio of digital experiences requires a standard technology footprint, build around certain core services, but with the option to vary approaches to experience building and experience delivery.

Drupal and Acquia proof points

More and more, we see organizations standardize on Drupal. Why? Because Drupal is one of the few solutions that can scale from very small to extremely large.

Drupal also has the depth and breadth of functionality to support thousands of different use cases; including blogs, marketing sites, employee experience sites, corporate intranets, commerce sites and extremely high-traffic event websites.

Because Drupal is a hybrid CMS, you can use no-code tools to build experiences, or JavaScript frameworks to build experiences.

For websites that require custom code, Acquia Cloud is the leading Drupal Platform-as-a-Service (Drupal PaaS). It offers high security, high-availability, on-demand elasticity, staging environments and many developer tools.

For templated sites, Acquia Site Factory allows you to assemble tens, hundreds or thousands of unique digital experiences.

Regardless of the delivery model used, services like Acquia Content Hub and Acquia CDP help you share content and data across your portfolio of sites. In addition, Acquia provides global visibility over all your Drupal applications and experiences.

Conclusion

After decades of contending with rigid, inflexible systems, enterprises crave the agility and speed that comes with composability.

This is particularly important in today's economy, where organizations gain competitive advantages based on having a tailored digital customer experience.

I predict that the majority of enterprises will have migrated to a composable model within the next 10 years.

October 01, 2022

Dat vind ik een goed idee. Maar dan wel wanneer ook vrouwen moeten gaan.

Sterven voor je land is geen zwaar werk. Vrouwen zijn daar net zo geschikt voor als mannen. Het enige wat je moet doen is in een zelfgegraven put of in de loopgraven (die tegenwoordig door graafmachines, wat een luxe, voor je gemaakt zijn) rustig te wachten tot de artillerie van de tegenstander de benen en armen van je lijf kapotbombarderen. Gedurende dat spektakel zie je meestal eerst een aantal keer hetzelfde gebeuren bij je vriendinnen in het putteke een beetje verderop.

Tijdens dat wachten heb je diarree en ben je daardoor constant uitgedroogd, komen de ratten aan je tenen knauwen, worden je strijdende vriendinnen geweldadig zot (want ze hebben hun vriendinnen aan gort geschoten zien worden), is er de hele nacht constant bombardement waarbij iedere knal een knal op jouw putteke had kunnen zijn, zie je mensen levend verbrand worden door fosfor-regen en krijg je continu idiote opdrachten van de leidinggevende commandanten (die, trouwens, vrouwen kunnen zijn: geen probleem he, dat kan ook. Dat kan ook).

Dat gaat van een zomer of lente-situatie uit. Want in de winter is het ook nog eens negatief een paar graden gedurende de hele nacht dat jij daar zit. Teminste als je hier bij ons gestationeerd bent, en niet dichter bij Rusland. Waar het min 20 en min 30 is en je gewoonweg doodvriest. Terwijl de ratten aan je tenen knauwen en je diarree hebt. Just in case you forgot about that.

Om op dat moment getrained te worden is iets dat geheel ‘equal’ hoort te zijn. Zoals de toiletten moeten we unisex-loopgraven hebben. Zoals de huidige generatie het wil moeten we dus ook unisex-legerdienst hebben.

Ik ben helemaal voorstander. Zolang de vrouwtjes ook moeten gaan. We moeten allemaal ons steentje bijdragen. Niet waar?

September 29, 2022

I have a dual boot on my desktop pc: Windows 11 and Ubuntu Linux. I hardly every use the Windows installation. Maybe for some games, but Steam has gotten better and better at supporting games on Linux. Or when you need to login on some government website with your eID and you can’t use the ItsMe app.

Many moons ago I did a boo-boo: for some reason I felt that I had to make my EFI system partition bigger. Which also meant resizing and moving all other partitions. Linux didn’t flinch but Windows pooped in its pants. Apparently that operating system is soooo legacy that it can’t cope with a simple partition move. I tried to fix it using a Windows system repair disk but the damn thing just couldn’t be arsed.

The partitions on my first hard disk

For a long time I just couldn’t be bothered with any further repair attempts. I don’t need that Windows anyway. I can always run Windows in VirtualBox if I really need it. It also means that I can nuke a 414 GiB partition and use that space for better things. As you can see in the screenshot, I mounted it on /mnt/windows with the intention of copying the directory Users/Amedee to Linux, in case there was still something of value there. Probably not, but better safe than sorry.

There’s just one small snag: for the life of me, I couldn’t find a Windows activation key, or remember where I put it. It’s not an OEM PC so the key isn’t stored in the BIOS. And I didn’t want to waste money on buying another license for an operating system that I hardly ever use.

I googled for methods to retrieve the Windows activation key. Some methods involve typing a command on the command prompt of a functioning Windows operating system, so those were not useful for me. Another method is just reading the activation key from the Windows Registry:

Computer\HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SoftwareProtectionPlatform\BackupProductKeyDefault

I don’t need a working Windows operating system to read Registry keys, I can just mount the Windows filesystem in Linux and query the Registry database files in /Windows/System32/config/. I found 2 tools for that purpose: hivexget and reglookup.

hivexget

This one is the simplest, it directly outputs the value of a registry key.

Installation:

sudo apt install --yes libhivex-bin

Usage:

hivexget /mnt/windows/Windows/System32/config/SOFTWARE \
     "\Microsoft\Windows NT\CurrentVersion\SoftwareProtectionPlatform" \
     BackupProductKeyDefault
XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

reglookup

This requires a bit more typing.

Installation:

sudo apt install --yes reglookup

Usage:

reglookup -p "/Microsoft/Windows NT/CurrentVersion/SoftwareProtectionPlatform/BackupProductKeyDefault" \
     /mnt/windows/Windows/System32/config/SOFTWARE
PATH,TYPE,VALUE,MTIME
/Microsoft/Windows NT/CurrentVersion/SoftwareProtectionPlatform/BackupProductKeyDefault,SZ,XXXXX-XXXXX-XXXXX-XXXXX-XXXXX,

The output has a header and is comma separated. Using -H removes the header, and then cut does the rest of the work;

reglookup -H -p "/Microsoft/Windows NT/CurrentVersion/SoftwareProtectionPlatform/BackupProductKeyDefault" \
     /mnt/windows/Windows/System32/config/SOFTWARE \
     | cut --delimiter="," --fields=3
XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

September 28, 2022

We now invite proposals for developer rooms for FOSDEM 2023. FOSDEM offers open source and free software developers a place to meet, share ideas and collaborate. Renowned for being highly developer-oriented, the event brings together some 8000+ geeks from all over the world. The twenty-third edition will take place Saturday 4th and Sunday 5th February 2023 in Brussels, Belgium. Developer rooms are assigned to self-organising groups to work together on open source and free software projects, to discuss topics relevant to a broader subset of the community, etc. Most content should take the form of presentations. Proposals involving collaboration舰

Last week, over 1,200 Drupalists gathered in Prague for DrupalCon Europe. It was great to see everyone together in person.

In good tradition, I delivered my State of Drupal keynote. You can watch the video of my keynote or download my slides (380 MB).

Text on slide reads

Why the Open Web should win

Today the web is ingrained in every aspect of our daily lives. We use it for work, to socialize, to pay our bills, get healthcare, and gain access to information.

As more transactions, collaborations and interactions are taking place online, there is now a greater responsibility to ensure that the web is inclusive of every person, and accounts for everyone's safety.

When people are excluded from being able to access online experiences, they are also excluded from rewarding careers, independent lifestyles, and the social interactions and friendships that bring people together.

For those reasons, we need to do everything we can to protect and grow the Open Web.

Text on slide reads

Caring about Drupal is caring about the Open Web

In my keynote, I made the point that Drupal is an important player in the preservation and growth of the Open Web.

I kicked off my keynote talking about some of my personal reasons for using Drupal, most of which have to do with the future of the Open Web.

After all these years, I continue to upload my photos to my website, despite there being plenty of simpler alternatives (e.g. Facebook or Instagram). I do this for a number of reasons.

First, my photos are precious to me, and I don't want them to get lost. I look at news stories about MySpace and Facebook losing users' content. I like that I'm in control of my own data, including my backups.

Second, I don't like how proprietary platforms limit my creative freedom. Pages and templates within a closed platform tend to look the same. It's hard to stand out, or even just express yourself the way you want to. With Drupal, I'm unrestricted in how I share my photos.

Third, I don't like how these platforms treat my friends and family. Many of them use invasive tracking. For that reason, I don't use trackers or ads on my website. My site aspires to the privacy of a printed book.

All of these are reasons why I want both Drupal and the Open Web to win. We don't want to live in a world where proprietary platforms reign supreme. We need more "Good Software". Software that is open, flexible, secure, accessible, and pro-privacy.

Making security, privacy, accessibility, multilingual capabilities, usability, and ease of maintenance top priorities is hard work work, but it's worth it. Caring about Drupal is the same as caring about the Open Web.

Drupal's growth and influence

Text reads

The good news is that Drupal has grown into a powerful platform for ambitious site builders. Drupal empowers millions of ambitious site builders to create Open Web experiences.

However, we have to get better at promoting what we're good at to ensure more people understand how powerful and influential Drupal is. That influence becomes really clear when you look at Drupal's end users.

Making Drupal's high bar easier to achieve

Because of Drupal's impact on the digital landscape, we keep a high bar with regards to stability, reliability, accessibility, security, backwards compatibility, and more. With great impact comes great responsibility.

This high bar can make contribution difficult and slow, and is sometimes what keeps people from contributing back more to Drupal.

Unfortunately, we can't lower the bar. However, we can make it easier to achieve our high bar. That's why we are moving from Drupal's homegrown collaboration tools to GitLab.

GitLab streamlines and automates various steps of the contribution process. Check out the Drupal.org GitLab video from the Drupal Association for an update.

Welcoming more people to Drupal 10

Another way to accelerate innovation is to grow our capacity and attract more people to Drupal.

The best way to attract new people to our community is by making easy-to-use software that solves real world problems.

Since my previous DrupalCon keynote, I'm happy to report that we've made a lot of progress on our key initiatives. A few key highlights:

  • Olivero became stable/default
  • CKEditor5 became stable/default
  • We're almost PHP 8.2 ready
  • We upgraded to Symfony 6
  • And we made Drupal Core smaller

Each of these initiatives is significant because they make Drupal and the Open Web more approachable.

In addition, the Project Browser initiative and Automatic Updates initiative saw tremendous progress. Videos below.

To learn more about Drupal's strategic initiatives, you can watch the Drupal Core Initiative Leads keynote. Highly recommended for those that want to contribute.

Upgrading to Drupal 10

Drupal 10 is scheduled to be released in mid-December. And after the Drupal 10 release, users will have 11 months to upgrade from Drupal 9 to Drupal 10. Drupal 10 will require PHP 8.1, something to plan for as well.

A timeline that shows the Drupal 10 release date and the Drupal 9 end-of-life date.Drupal 10 will be released on December 14, 2022. This gives site owners until November 2023 to update from Drupal 9 to Drupal 10.

Luckily, the upgrade path has never been easier. In the video below, we compare the upgrade path from Drupal 8 to Drupal 9 to the upgrade from Drupal 9 to Drupal 10. As you can see, the Drupal 9 to 10 upgrade is much more automated. It will be the easiest yet!

The update to Drupal 10 is required because some of Drupal 9's third-party dependencies will reach end-of-life. If you don't upgrade within 11 months, you will be running a site with unmaintained third-party dependencies.

A pie chart that shows that many modules are ready, or close to ready, for Drupal 10.We recently released the first Drupal 10 beta release. At the time we released Drupal 10 beta-1, 3 times as many modules were ready compared to when we released Drupal 9 beta-1. More than a 1,000 modules are already ready for Drupal 10!A bell curve chart that shows Drupal getting easier to use and maintain. Drupal 9 and Drupal 10 are easier than Drupal 8.Thanks to various focused initiatives, Drupal is starting to get easier. Drupal 8 was peak difficulty.

Thank you

I hope we can all find ways to care about building the web we want to see for the future. Making Drupal better means making the Open Web better.

I'd like to thank everyone that was involved in making DrupalCon Prague, Drupal 10, and the key initiatives described above a resounding success.

Last but not least, I'd like to encourage even more people to get involved. It might not always be easy, but it's well worth it.

September 27, 2022

When I started learning how to develop Bluetooth Low Energy (BLE) applications with Zephyr RTOS, I was immediately impressed by the excellent documentation and the wealth of sample code of this open source real-time operating system. But what really got me understand the way of working with BLE on Zephyr was just going through each line of code in a sample and find out the corresponding definition in Zephyr's header files, which are surprisingly readable.

As an exercise in this approach, I want to go through the iBeacon sample application with you. You'll find that this deceptively simple application already hides a lot of details that are important to know before starting with more complex Zephyr code.

Here's the sample code to broadcast an iBeacon advertisement:

/*
 * Copyright (c) 2018 Henrik Brix Andersen <henrik@brixandersen.dk>
 *
 * SPDX-License-Identifier: Apache-2.0
 */

#include <zephyr/types.h>
#include <stddef.h>
#include <zephyr/sys/printk.h>
#include <zephyr/sys/util.h>

#include <zephyr/bluetooth/bluetooth.h>
#include <zephyr/bluetooth/hci.h>

#ifndef IBEACON_RSSI
#define IBEACON_RSSI 0xc8
#endif

/*
 * Set iBeacon demo advertisement data. These values are for
 * demonstration only and must be changed for production environments!
 *
 * UUID:  18ee1516-016b-4bec-ad96-bcb96d166e97
 * Major: 0
 * Minor: 0
 * RSSI:  -56 dBm
 */
static const struct bt_data ad[] = {
    BT_DATA_BYTES(BT_DATA_FLAGS, BT_LE_AD_NO_BREDR),
    BT_DATA_BYTES(BT_DATA_MANUFACTURER_DATA,
              0x4c, 0x00, /* Apple */
              0x02, 0x15, /* iBeacon */
              0x18, 0xee, 0x15, 0x16, /* UUID[15..12] */
              0x01, 0x6b, /* UUID[11..10] */
              0x4b, 0xec, /* UUID[9..8] */
              0xad, 0x96, /* UUID[7..6] */
              0xbc, 0xb9, 0x6d, 0x16, 0x6e, 0x97, /* UUID[5..0] */
              0x00, 0x00, /* Major */
              0x00, 0x00, /* Minor */
              IBEACON_RSSI) /* Calibrated RSSI @ 1m */
};

static void bt_ready(int err)
{
    if (err) {
        printk("Bluetooth init failed (err %d)\n", err);
        return;
    }

    printk("Bluetooth initialized\n");

    /* Start advertising */
    err = bt_le_adv_start(BT_LE_ADV_NCONN, ad, ARRAY_SIZE(ad),
                  NULL, 0);
    if (err) {
        printk("Advertising failed to start (err %d)\n", err);
        return;
    }

    printk("iBeacon started\n");
}

void main(void)
{
    int err;

    printk("Starting iBeacon Demo\n");

    /* Initialize the Bluetooth Subsystem */
    err = bt_enable(bt_ready);
    if (err) {
        printk("Bluetooth init failed (err %d)\n", err);
    }
}

First, this includes a couple of header files. Then the code defines the IBEACON_RSSI constant, which sets the measured power in the iBeacon advertisement. The value 0xc8 is equal to decimal value 200. As this field in the iBeacon specification's advertising data structure is encoded as a signed integer, this is interpreted as 200 - 256 = -56 dBm.

Advertising data structures in Zephyr

Then, you see the definition of an array of struct bt_data elements. The bt_data struct describes an advertising data structure with a type, length, and a pointer to the data. Its definition (in zephyr/bluetooth/bluetooth.h) is:

/** Description of different data types that can be encoded into
 * advertising data. Used to form arrays that are passed to the
 * bt_le_adv_start() function.
 */
struct bt_data {
        uint8_t type;
        uint8_t data_len;
        const uint8_t *data;
};

In the iBeacon program, the array consists of two elements. This is consistent with the iBeacon specification, which shows that an iBeacon advertising packet consists of two advertising structures: flags and manufacturer-specific data.

In principle, you could fill the array with bt_data structs you create yourself, but Zephyr defines some helper macros. Their definition (again, from zephyr/bluetooth/bluetooth.h) looks like this:

/** @brief Helper to declare elements of bt_data arrays
 *
 * This macro is mainly for creating an array of struct bt_data
 * elements which is then passed to e.g. @ref bt_le_adv_start().
 *
 * @param _type Type of advertising data field
 * @param _data Pointer to the data field payload
 * @param _data_len Number of bytes behind the _data pointer
 */
#define BT_DATA(_type, _data, _data_len) \
        { \
                .type = (_type), \
                .data_len = (_data_len), \
                .data = (const uint8_t *)(_data), \
        }

/** @brief Helper to declare elements of bt_data arrays
 *
 * This macro is mainly for creating an array of struct bt_data
 * elements which is then passed to e.g. @ref bt_le_adv_start().
 *
 * @param _type Type of advertising data field
 * @param _bytes Variable number of single-byte parameters
 */
#define BT_DATA_BYTES(_type, _bytes...) \
        BT_DATA(_type, ((uint8_t []) { _bytes }), \
                sizeof((uint8_t []) { _bytes }))

So, with the BT_DATA macro, you construct a bt_data struct with the type, data, and data length you specify (using a pointer to the data). BT_DATA_BYTES is a macro that uses the BT_DATA macro and automatically fills it with the right length for data with a known size.

If you return to the iBeacon program, you see that it fills the array with two elements using the BT_DATA_BYTES macro.

The first element is of type BT_DATA_FLAGS and with data BT_LE_AD_NO_BREDR. You can find the definition of these constants in zephyr/bluetooth/gap.h (GAP stands for Generic Access Profile). For non-connectable advertising packets, the flags data structure is optional.

The second element is of type BT_DATA_MANUFACTURER_DATA. The data used to construct this element follows the format listed in the iBeacon specification: Apple's two-byte company ID, the type (0x02), and length (0x15 or 21 bytes) for the iBeacon data type, and then the UUID, major, minor, and measured power.

All in all, the result of all these macros is an array with advertising data structures that looks like this, schematically:

/images/zephyr-ibeacon-bt-data.png

An iBeacon packet's advertisement data in Zephyr

Enabling Bluetooth

Until now, the code has been all about setting up the right data structures to advertise. Let's have a look at the main() function. This actually does just one thing: calling the bt_enable() function. You have to call this function before any other calls that use the board's Bluetooth hardware. It initializes the Bluetooth subsystem and returns 0 on success and a negative error code otherwise.

The argument to the bt_enable() function is a callback function that's called when initializing Bluetooth is completed. It's in this callback function, bt_ready(), that the core of the program resides.

Advertising

The callback function is called with one argument -- an error code that is 0 on success, or an error code consisting of a negative value otherwise. So, in bt_ready(), you first check whether Bluetooth initialization failed, and, if not, you start advertising.

The definition of bt_le_adv_start() (still in zephyr/bluetooth/bluetooth.h), the function used to start advertising, is:

/** @brief Start advertising
 *
 * Set advertisement data, scan response data, advertisement parameters
 * and start advertising.
 *
 * When the advertisement parameter peer address has been set the advertising
 * will be directed to the peer. In this case advertisement data and scan
 * response data parameters are ignored. If the mode is high duty cycle
 * the timeout will be @ref BT_GAP_ADV_HIGH_DUTY_CYCLE_MAX_TIMEOUT.
 *
 * @param param Advertising parameters.
 * @param ad Data to be used in advertisement packets.
 * @param ad_len Number of elements in ad
 * @param sd Data to be used in scan response packets.
 * @param sd_len Number of elements in sd
 *
 * @return Zero on success or (negative) error code otherwise.
 * @return -ENOMEM No free connection objects available for connectable
 *                 advertiser.
 * @return -ECONNREFUSED When connectable advertising is requested and there
 *                       is already maximum number of connections established
 *                       in the controller.
 *                       This error code is only guaranteed when using Zephyr
 *                       controller, for other controllers code returned in
 *                       this case may be -EIO.
 */
int bt_le_adv_start(const struct bt_le_adv_param *param,
                    const struct bt_data *ad, size_t ad_len,
                    const struct bt_data *sd, size_t sd_len);

So, to call this function, you need to supply advertising parameters, the advertising data structure's array and its length, as well as the array of data structures for the scan response packet and its length.

The iBeacon example program doesn't use any scan response data, so the last two arguments are NULL and 0. The array of advertising data structures is ad, which you constructed in the beginning of the code. With ARRAY_SIZE, a macro defined in zephyr/sys/util.h, you compute the number of elements in the array. When using a C compiler, this macro is defined as:

#define ARRAY_SIZE(array) (sizeof(array) / sizeof((array)[0]))

The first argument to the bt_le_adv_start() function specifies the advertising parameters. In this example, this argument is BT_LE_ADV_NCONN. If you look up its definition in zephyr/bluetooth/bluetooth.h, you'll see that this defines non-connectable advertising with a private address:

/** Non-connectable advertising with private address */
#define BT_LE_ADV_NCONN BT_LE_ADV_PARAM(0, BT_GAP_ADV_FAST_INT_MIN_2, \
                                        BT_GAP_ADV_FAST_INT_MAX_2, NULL)

The macro BT_LE_ADV_PARAM is another helper macro:

/**
 * @brief Helper to declare advertising parameters inline
 *
 * @param _options   Advertising Options
 * @param _int_min   Minimum advertising interval
 * @param _int_max   Maximum advertising interval
 * @param _peer      Peer address, set to NULL for undirected advertising or
 *                   address of peer for directed advertising.
 */
#define BT_LE_ADV_PARAM(_options, _int_min, _int_max, _peer) \
        ((struct bt_le_adv_param[]) { \
                BT_LE_ADV_PARAM_INIT(_options, _int_min, _int_max, _peer) \
         })

This teaches you that BT_GAP_ADV_FAST_INT_MIN_2 and BT_GAP_ADV_FAST_INT_MAX_2 are the minimum and maximum advertising interval. You can find their definitions in zephyr/bluetooth/gap.h:

#define BT_GAP_ADV_FAST_INT_MIN_2               0x00a0  /* 100 ms   */
#define BT_GAP_ADV_FAST_INT_MAX_2               0x00f0  /* 150 ms   */

So, your iBeacon will use an advertising interval of between 100 ms and 150 ms.

BT_LE_ADV_PARAM uses another macro in the same header file, BT_LE_ADV_PARAM_INIT, and its definition is:

/**
 * @brief Initialize advertising parameters
 *
 * @param _options   Advertising Options
 * @param _int_min   Minimum advertising interval
 * @param _int_max   Maximum advertising interval
 * @param _peer      Peer address, set to NULL for undirected advertising or
 *                   address of peer for directed advertising.
 */
#define BT_LE_ADV_PARAM_INIT(_options, _int_min, _int_max, _peer) \
{ \
        .id = BT_ID_DEFAULT, \
        .sid = 0, \
        .secondary_max_skip = 0, \
        .options = (_options), \
        .interval_min = (_int_min), \
        .interval_max = (_int_max), \
        .peer = (_peer), \
}

All in all, this means that the code starts non-connectable advertising with its default Bluetooth ID, a private address, no special advertising options, and with an advertising interval of between 100 ms and 150 ms.

Summary

When reading source code of Zephyr applications, you can definitely feel discouraged by all these macros at first. However, after you've seen a couple of Zephyr programs, you start to see how they all fit together. The beauty of Zephyr is that its source code is completely open. If you don't understand what a specific macro or function is doing, just look up its declaration in the header file or even in the corresponding C file where it's implemented.

This approach has thought me a lot about developing Bluetooth Low Energy applications with Zephyr. I hope it will be a useful approach to you too.

September 26, 2022

An alphabetical list of software that I really like.

Apache: Not as popular as it used to be, but I know it and so still use it

apt/aptitude/apt-get: The origin of all 'apps' and the most user friendly and secure way to manage software

Audacity: Or is that too controversial now with their telemetry?

bash: The best interface to my computers

convert: From Imagemagick, for batch converting images 

Civilization IV: Yes 4, the only software that is not (yet?) open source

cron: It wakes me up every day (using mpv and bash)

Debian: I run it on almost everything

ffmpeg: to convert video formats

find: because I tend to forget where I put stuff :)

Firefox: because an alternative browser is crucial

FreeCAD: Amazing CAD software, and I like math

gcc: Because coding in C is fun!

gimp: Easy image manipulation

git: How did I ever live without it?

gramps: To keep track of my ancestors

grep: Often very useful.

Homebrew: To make a MacBook usable

Inkscape: Because vector graphics are essential

Linux: in background of course, like so many other tools that I am rarely aware of

make: For software that is not part of Debian

mariadb(mysql): So easy to build a database

mpv: For audio and video (and for their API

Openshot: To create Lego NXT or Prusa video's

PrusaSlicer: A 3D printer opens a whole new world of possibilities 

Python: I hate this language, but it is used so much and coding is still fun

return Youtube dislike: How else is Youtube usable?!

rsync: For backups

Singlefile: because web pages tend to disappear forever 

ssh: Multitool for anything remote

Thunderbird: Reading my mail since 1996 (as Netscape Mail)

tmux: used daily, configured to a quiet and relaxing layout

Video downloadhelper: because video's tend to disappear forever

vim: By far the best editor for text, code, config files and more.

wine: To play that one game on Debian (though I wish its 32-bit had as much memory as 32-bit Windows XP)

Wireshark: The best tool to learn about networking

XFCE: My GUI since almost 20 years

xfce-screenshooter: Because screenshots are useful

 

Inspiration HN.

September 25, 2022

Back after two virtual editions, FOSDEM 2023 will take place at the ULB on the 4th and 5th of February 2023. As has become traditional, we offer free and open source projects a stand to display their work “in real life” to the audience. You can share information, demo software, interact with your users and developers, give away goodies, sell merchandise or accept donations. All is possible! We offer you: One table (180x80cm) with a set of chairs and a power socket. Fast wireless internet access. A spot on stands.fosdem.org. You can choose if you want the spot for舰

For one of the projects I'm currently involved in, we want to have a better view on sustainability within IT and see what we (IT) can contribute in light of the sustainability strategy of the company. For IT infrastructure, one would think that selecting more power-efficient infrastructure is the way to go, as well as selecting products whose manufacturing process takes special attention to sustainability.

There are other areas to consider as well, though. Reusability of IT infrastructure and optimal resource consumption are at least two other attention points that deserve plenty of attention. But let's start at the manufacturing process...

Certifications for products and companies

Eco certifications are a good start in the selection process. By selecting products with the right certification, companies can initiate their sustainable IT strategy with a good start. Such certifications look at the product and manufacturing, and see if they use proper materials, create products that can have extended lifetimes in the circular (reuse) economy, ensure the manufacturing processes use renewable energy and do not have harmful emissions, safeguard clean water, etc.

In the preliminary phase I am right now, I do not know yet which certifications make most sense to pursue and request. Sustainability is becoming big business, so plenty of certifications exist as well. From a cursory search, I'd reckon that the following certifications are worth more time:

  • EcoVadis provides business sustainability ratings that not only cover the ecological aspect, but also social and ethical performance.
  • ISO 14001 covers environmental management, looking at organizations' processes and systematic improvements contributing to sustainability.
  • Carbon Neutral focus on transparency in measurements and disclosure of emissions, and how the company is progressing in their strategy to reduce the impact on the environment.
  • TCO Certified attempts to address all stages of a manufacturing process, from material selection over social responsibility and hazardous substances up to electronic waste and circular economy.
  • Energy Star focuses on energy efficiency, and tries to use standardized methods for scoring appliances (including computers and servers).

Power efficiency

A second obvious part is on power efficiency. Especially in data center environments, which is the area that I'm interested in, power efficiency also influences the data center's capability of providing sufficient cooling to the servers and appliances. Roughly speaking, a 500 Watt server generates twice as much heat as a 250 Watt server. Now, that's oversimplifying, but for calculating heat dissipation in a data center, the maximum power of infrastructure is generally used for the calculations.

Now, we could start looking for servers with lower power consumption. But a 250 Watt server is most likely going to be less powerful (computing-wise) than a 500 Watt server. Hence, power efficiency should be considered in line with the purpose of the server, and thus also the workloads that it would have to process.

We can use benchmarks, like SPEC's CPU 2017 or SPEC's Cloud IaaS 2018 benchmarks, to compare the performance of systems. Knowing the server's performance for given workloads and the power consumption, allows architects to optimize the infrastructure.

Heat management (and more) in the data center

A large consumer of power in a data center environment are the environmental controls, with the cooling systems taking a big chunk out of the total power consumption. Optimizing the heat management in the data center has a significant impact on the power consumption. Such optimizations are not solely about reducing the electricity bill, but also about reusing the latent heat for other purposes. For instance, data center heat can be used to heat up nearby buildings.

A working group of the European Commission, the European Energy Efficiency Platform (E3P), publishes an annual set of best practices in the EU Code of Conduct on Data Center Energy Efficiency which covers areas such as airflow design patterns, operating temperature and humidity ranges, power management features in servers and appliances, infrastructure design aspects (like virtualization and appropriate, but no over-engineered redundancy), etc.

This practice goes much beyond the heat management alone (and is worth a complete read), covering the complete data center offering. Combining these practices with other areas of data center design (such as redundancy levels, covered by data center tiering) allows for companies that are looking at new data centers to overhaul their infrastructure and be much better prepared for sustainable IT.

Circular ecosystem

Another part that often comes up in sustainability measures is how reusable the infrastructure components are after their "first life". Infrastructure systems, which frequently renew after 4 to 5 years of activity, can be resold rather than destroyed. The same can be said for individual components.

Companies that deal with sensitive data regularly employ "Do Not Return" clauses in the purchases of storage devices. Disks are not returned if they are faulty, or just swapped for higher density disks. Instead, they are routinely destroyed to make sure no data leakage occurs.

Instead of destroying otherwise perfect disks (or disks that still have reusable components) companies could either opt for degaussing (which still renders the disk unusable, but has better recyclability than destroyed disks) or data wiping (generally through certified methods that guarantee the data cannot be retrieved).

Extended lifecycle

Systems are often working perfectly beyond their 4 to 5 year lifespans. Still, these systems are process-wise automatically renewed to get more efficient and powerful systems in place. But that might not always be necessary - beyond even the circular ecosystem remarks above (where such systems could be resold), these systems can even get extended lifecycle within the company.

If there is no need for a more powerful system, and the efficiency of the system is still high (or the efficiency can be improved through minor updates), companies can seek out ways to prolong the use of the systems. In previous projects, I advised that big data nodes can perfectly remain inside the cluster after their regular lifetime, as the platform software (Hadoop) can easily cope with failures if those would occur.

Systems can also be used to host non-production environments or support lab environments. Or they can be refurbished to ensure maximal efficiency while still being used in production. Microsoft for instance has a program called Microsoft Circular Centers which aims at a zero-waste sustainability within the data center, through reuse, repurpose and recycling.

Right-sizing the infrastructure

Right-sizing is to select and design infrastructure to deal with the workload, but not more. Having a set of systems at full capacity is better than having twice as many systems at half capacity, as this leads to power inefficiencies.

To accomplish right-sizing isn't as easy as selecting the right server for a particular workload. Workload is distributed, and systems are virtualized. Virtualization allows for much better right-sizing as you can distribute workload more optimally.

Companies with large amounts of systems can more efficiently distribute workload across their systems, making it easier to have a good consumption pattern. Smaller companies will notice that they need to design for the burst and maximum usage, whereas the average usage is far, far below that threshold.

Using cloud resources can help to deal with bursts and higher demand, while still having resources on-premise to deal with the regular workload. Such hybrid designs, however, can be complex, so make sure to address this with the right profiles (yes, I'm making a stand for architects here ;-)

Standardizing your infrastructure also makes this easier to accomplish. If the vast majority of servers are of the same architecture, and you standardize on as few operating systems, programming languages and what not, you can more easily distribute workload than when these systems have different architectures and purposes.

Automated workload and power management

Large environments will regularly have servers and infrastructure that is not continuously used at near full capacity. Workloads are frequently following a certain curve, such as higher demand during the day and lower at night. Larger platforms use this curve to schedule appropriate workload (like running heavy batch workload at night while keeping the systems available for operational workload during the day) so that the resources are more optimally used.

By addressing workload management and aligning power management, companies can improve their power usage by reducing active systems when there are less resource needs. This can be done gradually, such as putting CPUs in lower power modes (CPU power takes roughly 30% of a system's total power usage), but can expand to complete hosts being put in idle state.

We can even make designs where servers are shut down when unused. While this is frequently frowned upon, citing possible impact on hardware failures as well as reduced reactivity to sudden workload demand, proper shutdown techniques do offer significant power savings (as per a research article titled Quantifying the Impact of Shutdown Techniques for Energy-Efficient Data Centers).

Conclusion

Sustainability within IT focuses on several improvements and requirements. Certification helps in finding and addressing these, but this is not critical in any company's strategy. Companies can address sustainability easily without certification, but with proper attention and design.

Feedback? Comments? Don't hesitate to drop me an email, or join the discussion on Twitter.

September 21, 2022

Over the 15+ years of my development career, I have learned several things that significantly increase my effectiveness. In this post, I share those learnings with you.

Structure:

  • Generic Advice — Important context and motivation for the technical advice
  • Technical Advice — The main course
  • Recommended Reading — Links to high-quality books and blogs that are great for getting started

This blog post mentions and links valuable concepts that you can explore further as you see fit.

Generic Advice for Juniors

1. Code is not the Point

As developers, we like writing code. Most of us want to be given a nice unambiguous task. A fun technical puzzle to solve without paying attention to the rest of the world.

Put reasonable effort into making sure that you are solving the right problem. To quote Peter Drucker: There is nothing so useless as doing efficiently that which should not be done at all. Gather feedback early and often, typically by continuous delivery to real users. Be Agile.

Software development is expensive, with the vast majority of the effort of real-world projects typically going into maintenance. Combine this with the goal being user/business outcomes, the best code is often no code. To quote Bill Gates: “Measuring programming progress by lines of code is like measuring aircraft building progress by weight.”

See also: YAGNI, KISS, The Last Responsible Moment.

2. Software Design Matters

During the first 5 years of my development career, I thought that software design is for software architects or other people with special roles. I was focused on “getting things done”, and saw software design and practices such as writing tests, as a distraction at best. My code worked, and I was getting a lot of things done. Or so I thought.

Then I read Clean Code, by Robert C. Martin. This book motivates caring about software design and contains examples and many technical heuristics. The most conceptual takeaway is the saying that “the only way to go fast is to go well“. In other words, if you make a mess, it will slow you down. See also: TradableQualityHypothesis, DesignStaminaHypothesis

Learning how to write well-designed clean code of course takes time and effort. And when you start, you will be slower and make mistakes. Simple is not Easy.

3. Use the BEST Practices

Writing tests tends to be beneficial. There are exceptions, but most of the time, it makes a lot of sense to write automated tests. Writing tests is an example of a best practice.

If you are new to writing tests, just follow the best practice and write tests for everything. When starting, blindly following the best practice will be better than following your own underdeveloped judgment. Over time you will learn how to write tests effectively, and be able to tell the difference between you have messed up, and situations where writing a test is not worth it. You will also start to understand the value tests bring on a more visceral level, by having experienced the decrease in debugging sessions and the worry-free refactoring enabled by your tests. After developing your judgment, you will be able to transcend the best practice.

This advice applies to best practices in any area that you are a junior in. Automated tests are just an example.

One big gotcha is that it is not easy to tell the difference between a sensible best practice and something nonsensical or even counterproductive. This is made more complicated by most existing code being a mess, and by most developers, including “experienced” and “senior” ones, not knowing software design basics. This makes having a good mentor extremely valuable. Barring that, one piece of advice based on my own experiences is to be wary of best practices specific to the community of your language or framework. Look for evergreen advice that has been around for decades.

Technical Advice for Juniors

Our focus will be on technical topics. Many other areas are essential, such as health, happiness, career, and soft skills. Knowing how to avoid a technical pitfall won’t help you if you are sleep deprived and working on the wrong problem for a toxic boss that underpays you.

4. Write Tests

Write automated tests. Perhaps write tests before the code, such as via Test Driven Development (TDD). This makes it easy to verify your code is correct in a repeatable manner, thus saving you from much manual retresting and from debugging sessions.

You think test-first is difficult? Try debug-after.

Perhaps even more importantly, tests give you the safety net to refactor your code. And continuous refactoring is needed to keep your code clean. Without a reliable test suite, it is all the more likely that your code will rot.

Writing tests is difficult if the design of your code is poor, such as when using inheritance for code reuse, or when using static functions. If on the other hand, you have SOLID classes, with no global dependencies, then writing nice tests is not so difficult.

Test design matters because poorly written tests will slow you down. Avoid binding your tests to implementation details of the code under test or to the structure of the system. Avoid overusing Mocks and write better Test Doubles.

5. Do Not Use Inheritance For Code Reuse

This is one of those best practices that bring to mind the “Use the Best Practices” section. My advice: do not use inheritance for code reuse at all when you are starting out. It is rarely the right call and can do a lot of harm. Favor composition over inheritance.

6. Write Object-Oriented code

Write SOLID code that is not STUPID. There is so much value in understanding these principles and anti-patterns.

Actually create objects. Classes with only static methods are not OO. Try to avoid using static code altogether.

See also: my defense of SOLID.

7. Write Functional Code

(Functional programming is not to be confused with imperative structural programming.)

This point is not about entriely switching to a functional language. You can benefit from using a functional style in your OO language. Minimize state, especially mutable state, and do one thing in your functions. See also: functional core, imperative shell.

8. Use Informed Duplication

Copy-pasting big chunks of code to multiple places is almost always unwise. Any self-respecting developer soon learns this and starts to follow some form of Don’t Repeat Yourself (DRY). Unfortunately, the well-intended pursuit of DRY often leads to overengineering and accidental complexity. This is where the counterpart of DRY comes in: Write Everything Twice (WET). The idea behind WET is to only deduplicate on the third occurrence of duplication.

For a more in-depth look at the costs and benefits of deduplication, see The Fallacy of DRY.

9. Types, Names and Comments

Try to write self-documenting code and avoid comments.

Every time you write a comment, you should grimace and feel the failure of your ability of expression. — Robert C. Martin

Comments are dangerous because they can lie. The code can change without the comment being updated. New code can be added right under the comment. The comment might have been wrong or inaccurate in the first place. When this happens, the comment not only becomes useless, it becomes misleading.

To write self-documenting code:

  • Do one thing in your functions
    • By doing a single thing in a function, you can give it a clear name
    • Feel the need to explain what different sections of your function do by adding comments? Instead, extract each section into its own well-named function.
    • Extract till you drop“: if you can meaningfully extract a function, then you probably should. Don’t be afraid of small functions.
    • Command Query Separation
    • Similar to the Single Responsibility Principle for classes (The S in SOLID)
  • Minimize state
  • Use types. Combined with a test suite that executes the code, you can rely on the types telling the truth.
  • Avoid mixed types. Avoid parameters or return values that can be an integer, a boolean, or a string. This naturally happens if you write focused functions that only do one thing.
  • Write tests. A well-written and comprehensive test suite shows you how the production code can be used, and how it behaves.

Clean Code by Robert C. Martin has some good rules of thumb about naming and comments.

Recommended Reading for Juniors

Books

Blogs

See also: Recommended Reading for Developers by Jeff Atwood

Bonus links

  • Tell Don’t Ask — Encapsulation best practice
  • Law of Demeter — Coupling best practice
  • Domain Driven Design — A sizeable toolbox. More advanced, good to first learn the basics.
  • Object Calisthenics — Rules that restrict what you can do in programming. Nice for learning how to do things differently
  • Pair Programming — A great way to learn from each other
  • Code katas — Simple programming, great for practicing a specific technique or skill such as Test Driven Development

The post Advice for junior developers appeared first on Entropy Wins.

September 15, 2022

This is a warning: only do this at home, instead of don’t do this at home. On your local home network that is behind a firewall and/or gateway.

Edit: Philip Paeps sent me a few suggestions to restrict things even more. I have adapted this blog item to mention all of them.

Unfortunately there are companies that make and made printers and then don’t provide firmware upgrades for it.

You end up with a scanner/printer that works perfectly fine. Except you can’t find any SMTP servers without TLS anymore. Because rightfully so has more or less every E-mail provider turned off plain text SMTP.

Now your printer cannot send the scanned documents over E-mail anymore. Firmware upgrade of your scanner/printer? What if there is none?

What to do? Run a unencrypted postfix smtpd on some scrap computer on your local network, that relays your mails with its smtp client to a TLS enabled SMTP server.

apt-get install postfix # and select Internet site

I didn’t want fake security by having authentication on the smtpd side, as there will be no encryption between printer and our local postfix. When somebody listens on your local network they would not only have the PDFs that you scanned with your scanner/printer, they will also have those authentication tokens.

I suppose you can add authentication, but then at least don’t be silly and use usernames and passwords that you use somewhere else. Note that some really old scanners/printers also can’t do SMTP with authentication.

I used these relay_restrictions for smtpd. We will place the printer’s IP address in/within mynetworks, so we will relay for it through permit_mynetworks.

smtpd_relay_restrictions = permit_mynetworks reject_unauth_destination

The mydestination should be as restricted as possible so we’ll just empty it (no local delivery at all):

mydestination =
local_recipient_maps =
local_transport = error:local delivery is disabled

You can now also remove the local delivery agent from master.cf if you want. We restrict the mynetworks of course too. The scanner/printer is with DHCP on 192.168.0.0/24, so add that:

mydestination =
mynetworks = 127.0.0.0/8 192.168.0.0/24

Even better is to have your scanner/printer on a fixed IP address and then use that one IP address:

mynetworks = 127.0.0.0/8 192.168.0.11

In fact, when the scrap computer has a fixed IP address then you can further restrict things by using inet_interfaces of course:

inet_interfaces = 192.168.0.14, 127.0.0.1

And now we configure the relayhost, we will relay to our TLS enabled SMTP(s) server:

relayhost = [smtps.server.org]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = encrypt
header_size_limit = 4096000
message_size_limit = 524288000

Now make a file /etc/postfix/sasl_passwd to add the username / password. After that of course run postmap /etc/postfix/sasl_passwd

[smtps.server.org]:587 username:password

We now on the printer configure the scrap computer’s local network IP address or DHCP hostname as SMTP server.

Other security considerations that are possible is to place printer and scrap computer on a VLAN or let there be a crossed UTP cable between them. But if you are going to do those things then you also know how to do it yourself. Such things do make sense: your easily hackable LED lamps and even more easily hackable Smart TV don’t need to talk to your printer nor this scrap computer. VLAN everything!

September 14, 2022

It has been long overdue, but I’m finally almost ready to release Autoptimize Pro. The first version of AO Pro (an add-on to be installed on top off Autoptimize) will have Image optimization, CDN, fully automated critical CSS and a number of “boosters” to improve performance even more, with more features are in the backlog. Beta-testing is being finalized, I am currently not looking for additional...

Source

September 13, 2022

FOSDEM 2023 will take place on Saturday 4th and Sunday 5th of February 2023. We are aiming for a physical conference. Update: We initially said "2022" which was a typo, of course. Sorry about that. While there are still a lot of open questions, we are aiming for a physical FOSDEM 2023. It's both exciting and scary. We will open Call for Devrooms etc soon. One note: With the online format, we were able to host more devrooms than ever. In the physical world, we will be constrained by the amount and size of the rooms ULB is able舰

September 06, 2022

Na meer dan een half jaar geshockeerd toe te kijken hoe onze Europese leiders de éne na de andere aankondigingspolitiek deden zonder wat dan ook substantiëels te doen, lijkt de politieke klasse in de Europese wijk in Brussel eindelijk te zijn wakker geschoten.

Dat Rusland tegensancties zou doen nadat wij letterlijk hun nationale reserves afnamen, was verdomd voorspelbaar en kei hard al vanaf half februari een gegeven dat zeker was.

Dat het Russisch gas uiteindelijk zou afgesloten worden was klaar en duidelijk. Hoeveel gas het was was klaar en duidelijk. Dat de godganse Europese economie dan op haar gat zou gaan, was klaar en duidelijk.

We vinden het allemaal heel erg prettig om nu een beetje met Duitsland te staan lachen. Maar heel erg binnenkort zal jouw baan, die afhankelijk is van de Duitse industrie (denk er maar eens twee keer over na), verdwijnen. Het zal gedaan zijn met lachen terwijl je gigantisch hoge energiefacturen mag betalen.

Allemaal klaar en duidelijk. Crystal helder.

Maar wat deden de jongens en meisjes in de Europese wijk te Brussel daar aan? Helemaal niets.

Aankondigingspolitiek. Populistisch zijn. Dissidenten verketteren: was je tegen sancties, dan was je voor Putin. Alsof dit enkel Putin is en niet de gehele elite in Het Kremlin. Enkelen werden zelfs afgedreigd. Ja toch wel ja. WIJ zijn aan het afglijden naar fascisme en censuur. Heel erg duidelijk.

Maar als je je economie in oorlog zet tegen Rusland, dan hoor je ook voorbereid te zijn. Dat zijn en dat waren wij niet. Totaal niet.

De populistische aankondigingspolitiek van het afgelopen jaar zal er voor zorgen dat onze Europese economie gewoon de grond ingeboord zal worden.

We hadden intelligente sancties kunnen bedenken. Maar dat was niet straf genoeg. We hadden kunnen gaan onderhandelen met Rusland. Maar dan werd je verketterd.

Of is dat niet waar? Zie dan naar Macron die probeerde en telkens door de roeptoeters verketterd werd.

Maar zie hoe Erdogan het wel doet en daarmee zijn Turkije helemaal op de kaart heeft gezet. Hoe de hele West-Europese wereld in het conflict van Oekraïne er letterlijk gewoon niet toe doet. Maar Turkije helemaal wel.

Onze Westerse leiders zijn een bende populistische Twitter-accounts maar dan zonder eigen leger. Ze zijn dus niets. Niets dat er toe doet.

Wij hebben géén strategie. Dat zijn wij helemaal verleerd. We schipperen rond op populisme en het om ter hardst roepen. Ik bedoel, tweeten.

En dat, dat werkt niet in oorlogstijden.

August 30, 2022

A notorious ex-DD decided to post garbage on his site in which he links my name to the suicide of Frans Pop, and mentions that my GPG key is currently disabled in the Debian keyring, along with some manufactured screenshots of the Debian NM site that allegedly show I'm no longer a DD. I'm not going to link to the post -- he deserves to be ridiculed, not given attention.

Just to set the record straight, however:

Frans Pop was my friend. I never treated him with anything but respect. I do not know why he chose to take his own life, but I grieved for him for a long time. It saddens me that Mr. Notorious believes it a good idea to drag Frans' name through the mud like this, but then, one can hardly expect anything else from him by this point.

Although his post is mostly garbage, there is one bit of information that is correct, and that is that my GPG key is currently no longer in the Debian keyring. Nothing sinister is going on here, however; the simple fact of the matter is that I misplaced my OpenPGP key card, which means there is a (very very slight) chance that a malicious actor (like, perhaps, Mr. Notorious) would get access to my GPG key and abuse that to upload packages to Debian. Obviously we can't have that -- certainly not from him -- so for that reason, I asked the Debian keyring maintainers to please disable my key in the Debian keyring.

I've ordered new cards; as soon as they arrive I'll generate a new key and perform the necessary steps to get my new key into the Debian keyring again. Given that shipping key cards to South Africa takes a while, this has taken longer than I would have initially hoped, but I'm hoping at this point that by about halfway September this hurdle will have been taken, meaning, I will be able to exercise my rights as a Debian Developer again.

As for Mr. Notorious, one can only hope he will get the psychiatric help he very obviously needs, sooner rather than later, because right now he appears to be more like a goat yelling in the desert.

Ah well.

August 29, 2022

Here’s a scenario: ten years ago, you renovated your bathroom. As a shower, you chose a walk-in shower with a Belgian blue stone which is porous and is a limestone.

That’s why you have to be careful with products like Antikal. Because they will react with the lime in your limestone. In other words, you have to rinse away the excess Antikal quickly.

But you’re a computer geek and because you have a piece of software that you programmed and are already thinking about, you sometimes forget this.

Thus, a drop of Antikal remains on your blue stone.

A few hours later, that drop became a white stain. Which now really seems to be inside your stone.

Unfortunately, it really is. No matter what you try, it cannot be scrubbed off. Unless you let a professional come and polish a layer off of your stone (which costs about 800 euros, by the way).

Unfortunately, not much later the stains come back. Because the stains are deep in your stone and the reaction ‘continues’. That was 800 euros for a few weeks’ peace of mind.

Now what?

Of course you buy a whole gang of HG products such as HG blauwesteen reiniger, HG grafsteenreiniger and HG natuursteen kleurvlekkenverwijderaar.

You spray it on, let it soak in for half an hour; a little of the stains are now indeed better. But not really.

What actually worked better was the Mr. Proper Magic sponge. That sanded the top layer of that white stain away a bit.

But what worked best after that was a coat of Moeller Stone Care HMK P333 Hardsteenolie.

Your Belgian Blue stone will become a Belgian dark blue stone. But the stains (which I had first rubbed out with the Mr. Proper Magic sponge) are less visible and seem to belong to the stone’s natural life.

Then, after two full days letting the stuff get impregnated, a sticky top layer appeared. I had probably used a little too much of that stuff.

But another heavy treatment with the whole gang of HG products, such as HG blauwesteen reiniger, HG grafsteenreiniger and HG natuursteen kleurvlekkenverwijderaar (all at the same time and soak for half an hour) followed by rubbing hard with an iron sponge, and the sticky top layer is gone.

The stone is actually quite beautiful now. Dark blue.

August 26, 2022

The IT world is littered with frameworks, best practices, reference architectures and more. In an ever-lasting attempt to standardize IT, we often get lost in too many standards or specifications. For consultants, this is a gold-mine, as they jump in to support companies - for a fee, naturally - in adopting one or more of these frameworks or specifications.

While having references and specifications isn't a bad thing, there are always pros and cons.

August 22, 2022

Sometimes, it's useful to get a notification that a command has finished doing something you were waiting for:

make my-large-program && notify-send "compile finished" "success" || notify-send "compile finished" "failure"

This will send a notification message with the title "compile finished", and a body of "success" or "failure" depending on whether the command completed successfully, and allows you to minimize (or otherwise hide) the terminal window while you do something else, which can be a very useful thing to do.

It works great when you're running something on your own machine, but what if you're running it remotely?

There might be something easy to do, but I whipped up a bit of Perl instead:

#!/usr/bin/perl -w

use strict;
use warnings;

use Glib::Object::Introspection;
Glib::Object::Introspection->setup(
    basename => "Notify",
    version => "0.7",
    package => "Gtk3::Notify",
);

use Mojolicious::Lite -signatures;

Gtk3::Notify->init();

get '/notify' => sub ($c) {
    my $msg = $c->param("message");
    if(!defined($msg)) {
        $msg = "message";
    }
    my $title = $c->param("title");
    if(!defined($title)) {
        $title = "title";
    }
    app->log->debug("Sending notification '$msg' with title '$title'");
    my $n = Gtk3::Notify::Notification->new($title, $msg, "");
    $n->show;
    $c->render(text => "OK");
};

app->start;

This requires the packages libglib-object-introspection-perl, gir1.2-notify-0.7, and libmojolicious-perl to be installed, and can then be started like so:

./remote-notify daemon -l http://0.0.0.0:3000/

(assuming you did what I did and saved the above as "remote-notify")

Once you've done that, you can just curl a notification message to yourself:

curl 'http://localhost:3000/notify?title=test&message=test+body'

Doing this via localhost is rather silly (much better to use notify-send for that), but it becomes much more interesting if you're going to run this to your laptop from a remote system.

An obvious TODO would be to add in some form of security, but that's left as an exercise to the reader...

August 21, 2022

Door omstandigheden wat minder kunnen lezen, maar bij deze nog eens drie boeken:



De Hel van Deurne-Noord is een verzamling getuigenissen van voetballers en trainers, en reporters, over hun ervaring op den Bosuil. Het dateert van 2016 ofzo denk ik, dus het is vooral dromen van een groot Royal Antwerp FC. Ik denk niet dat er toen iemand was die Naingolan of Alderweireld als spelers van den Antwerp zag.

Het boek leest zeer vlot, elke getuigenis is maar een blad of twee, soms drie. Ideaal voor bus en treinritten.


Ons Antwerpen heeft me blij verrast! Dit is een van de beste boeken dat ik al gelezen heb. Het dateert van rond 1931, de auteurs zijn dus al lang dood, anders had ik hen zeker bedankt.

In dit boek wandel je rond in het Antwerpen van 90 jaar geleden. Er wordt bijvoorbeeld gezegd dat er plannen zijn om een Rubenshuis te maken in de buurt van de Meir/Wapperstraat. Het plein dat nu 'Wapper' heet, bestond nog niet. Er is ook geen Astridplein (dat heette toen Statieplein), maar ze hadden wel net meer dan tienduizend schepen gehad in Antwerpen op 1 jaar tijd.

Ik ben zinnens eens met dit boek rond te lopen, want echt elke straat waar ze zijn wordt vermeld, tot huisnummers toe. De weg zou dus helemaal moeten te volgen zijn.


Starten met C is mijn introductie tot programmeren in C (ik kende totdantoe enkel Basic, Assembler en PASCAL, ah en COBOL). Dit was ons cursusboekje in Leuven in 1991 denk ik... en maakte mij een grote fan van C.

Ik ben het terug aan het lezen nu en beleef er weer veel plezier aan. Nog steeds fan van C dus, helaas ben ik momenteel niet zo'n goeie programmeur. Nu het bijna uit is, zal ik mijn odio applicatie maar snel aanpassen, voordat iemand deze klucht kan zien :)



August 19, 2022

Yesterday I had the day off, so I decided to finally upgrade my Thinkpad X13 to Ubuntu 22.04. Jammy Jellyfish (as the release is nicknamed) is nice but the new default display server, Wayland, blocks Shutter and other non-native screenshot apps from making screenshots. This interfering with my support workflow as I use Shutter not only to make the screenshot but also to edit and upload it to imgur.

Source

August 12, 2022

I run Debian on my laptop (obviously); but occasionally, for $DAYJOB, I have some work to do on Windows. In order to do so, I have had a Windows 10 VM in my libvirt configuration that I can use.

A while ago, Microsoft issued Windows 11. I recently found out that all the components for running Windows 11 inside a libvirt VM are available, and so I set out to upgrade my VM from Windows 10 to Windows 11. This wasn't as easy as I thought, so here's a bit of a writeup of all the things I ran against, and how I fixed them.

Windows 11 has a number of hardware requirements that aren't necessary for Windows 10. There are a number of them, but the most important three are:

  • Secure Boot is required (Windows 10 would still boot on a machine without Secure Boot, although buying hardware without at least support for that hasn't been possible for several years now)
  • A v2.0 TPM module (Windows 10 didn't need any TPM)
  • A modern enough processor.

So let's see about all three.

A modern enough processor

If your processor isn't modern enough to run Windows 11, then you can probably forget about it (unless you want to use qemu JIT compilation -- I dunno, probably not going to work, and also not worth it if it were). If it is, all you need is the "host-passthrough" setting in libvirt, which I've been using for a long time now. Since my laptop is less than two months old, that's not a problem for me.

A TPM 2.0 module

My Windows 10 VM did not have a TPM configured, because it wasn't needed. Luckily, a quick web search told me that enabling that is not hard. All you need to do is:

  • Install the swtpm and swtpm-tools packages
  • Adding the TPM module, by adding the following XML snippet to your VM configuration:

    <devices>
      <tpm model='tpm-tis'>
        <backend type='emulator' version='2.0'/>
      </tpm>
    </devices>
    

    Alternatively, if you prefer the graphical interface, click on the "Add hardware" button in the VM properties, choose the TPM, set it to Emulated, model TIS, and set its version to 2.0.

You're done!

Well, with this part, anyway. Read on.

Secure boot

Here is where it gets interesting.

My Windows 10 VM was old enough that it was configured for the older i440fx chipset. This one is limited to PCI and IDE, unlike the more modern q35 chipset (which supports PCIe and SATA, and does not support IDE nor SATA in IDE mode).

There is a UEFI/Secure Boot-capable BIOS for qemu, but it apparently requires the q35 chipset,

Fun fact (which I found out the hard way): Windows stores where its boot partition is somewhere. If you change the hard drive controller from an IDE one to a SATA one, you will get a BSOD at startup. In order to fix that, you need a recovery drive. To create the virtual USB disk, go to the VM properties, click "Add hardware", choose "Storage", choose the USB bus, and then under "Advanced options", select the "Removable" option, so it shows up as a USB stick in the VM. Note: this takes a while to do (took about an hour on my system), and your virtual USB drive needs to be 16G or larger (I used the libvirt default of 20G).

There is no possibility, using the buttons in the virt-manager GUI, to convert the machine from i440fx to q35. However, that doesn't mean it's not possible to do so. I found that the easiest way is to use the direct XML editing capabilities in the virt-manager interface; if you edit the XML in an editor it will produce error messages if something doesn't look right and tell you to go and fix it, whereas the virt-manager GUI will actually fix things itself in some cases (and will produce helpful error messages if not).

What I did was:

  • Take backups of everything. No, really. If you fuck up, you'll have to start from scratch. I'm not responsible if you do.
  • Go to the Edit->Preferences option in the VM manager, then on the "General" tab, choose "Enable XML editing"
  • Open the Windows VM properties, and in the "Overview" section, go to the "XML" tab.
  • Change the value of the machine attribute of the domain.os.type element, so that it says pc-q35-7.0.
  • Search for the domain.devices.controller element that has pci in its type attribute and pci-root in its model one, and set the model attribute to pcie-root instead.
  • Find all domain.devices.disk.target elements, setting their dev=hdX to dev=sdX, and bus="ide" to bus="sata"
  • Find the USB controller (domain.devices.controller with type="usb", and set its model to qemu-xhci. You may also want to add ports="15" if you didn't have that yet.
  • Perhaps also add a few PCIe root ports:

    <controller type="pci" index="1" model="pcie-root-port"/>
    <controller type="pci" index="2" model="pcie-root-port"/>
    <controller type="pci" index="3" model="pcie-root-port"/>
    

I figured out most of this by starting the process for creating a new VM, on the last page of the wizard that pops up selecting the "Modify configuration before installation" option, going to the "XML" tab on the "Overview" section of the new window that shows up, and then comparing that against what my current VM had.

Also, it took me a while to get this right, so I might have forgotten something. If virt-manager gives you an error when you hit the Apply button, compare notes against the VM that you're in the process of creating, and copy/paste things from there to the old VM to make the errors go away. As long as you don't remove configuration that is critical for things to start, this shouldn't break matters permanently (but hey, use your backups if you do break -- you have backups, right?)

OK, cool, so now we have a Windows VM that is... unable to boot. Remember what I said about Windows storing where the controller is? Yeah, there you go. Boot from the virtual USB disk that you created above, and select the "Fix the boot" option in the menu. That will fix it.

Ha ha, only kidding. Of course it doesn't.

I honestly can't tell you everything that I fiddled with, but I think the bit that eventually fixed it was where I chose "safe mode", which caused the system to do a hickup, a regular reboot, and then suddenly everything was working again. Meh.

Don't throw the virtual USB disk away yet, you'll still need it.

Anyway, once you have it booting again, you will now have a machine that theoretically supports Secure Boot, but you're still running off an MBR partition. I found a procedure on how to convert things from MBR to GPT that was written almost 10 years ago, but surprisingly it still works, except for the bit where the procedure suggests you use diskmgmt.msc (for one thing, that was renamed; and for another, it can't touch the partition table of the system disk either).

The last step in that procedure says to restart your computer!, which is fine, except at this point you obviously need to switch over to the TianoCore firmware, otherwise you're trying to read a UEFI boot configuration on a system that only supports MBR booting, which obviously won't work. In order to do that, you need to add a loader element to the domain.os element of your libvirt configuration:

<loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE_4M.ms.fd</loader>

When you do this, you'll note that virt-manager automatically adds an nvram element. That's fine, let it.

I figured this out by looking at the documentation for enabling Secure Boot in a VM on the Debian wiki, and using the same trick as for how to switch chipsets that I explained above.

Okay, yay, so now secure boot is enabled, and we can install Windows 11! All good? Well, almost.

I found that once I enabled secure boot, my display reverted to a 1024x768 screen. This turned out to be because I was using older unsigned drivers, and since we're using Secure Boot, that's no longer allowed, which means Windows reverts to the default VGA driver, and that only supports the 1024x768 resolution. Yeah, I know. The solution is to download the virtio-win ISO from one of the links in the virtio-win github project, connecting it to the VM, going to Device manager, selecting the display controller, clicking on the "Update driver" button, telling the system that you have the driver on your computer, browsing to the CD-ROM drive, clicking the "include subdirectories" option, and then tell Windows to do its thing. While there, it might be good to do the same thing for unrecognized devices in the device manager, if any.

So, all I have to do next is to get used to the completely different user interface of Windows 11. Sigh.

Oh, and to rename the "w10" VM to "w11", or some such. Maybe.

August 10, 2022

À voir l’affiche et la bande-annonce, l’année du requin s’annonce comme une comédie estivale des plus traditionnelles, sorte de croisement entre « Les gendarmes de Saint-Tropez à la pêche au requin » et « Les bronzés au camping 3 ».

Heureusement, la lecture des critiques m’avait mis la puce à l’oreille. L’année du requin n’est pas une énième comédie franchouillarde de type sous-splendid, au grand plaisir ou au grand dam des commentateurs. Les gags de la bande-annonce s’enchainent dans les premières minutes du film. Comme prévu, le gendarme Maja, Marina Foïs, se prend un seau d’eau et une vanne comique de la part de son collègue Blaise, Jean-Pascal Zadi. Rires bien vite étouffés par la réplique tranchante d’une Marina Foïs qui crève l’écran en gendarme fatiguée par une carrière assez terne dans une ville où la spécialité est de poser ses fesses dans le sable et de regarder la mer : « Ce n’est pas gai de se prendre un seau d’eau lorsqu’on est en service. » Sourires gênés de ses coéquipiers et du public.

Le ton est donné. Le prétexte comédie n’était qu’un attrape-nigaud. Si le film regorge de pépites humoristiques, celles-ci se font discrètes, sans insistance (comme le coup de la garde-robe de Maja, entraperçue une seconde en arrière-plan). Là n’est pas le propos.

Le propos ? Il n’est pas non plus dans l’histoire, assez simple pour ne pas dire simplette : un requin hante les côtes de la station balnéaire de La Pointe et, à la veille de la retraite, la gendarme maritime Maja décide d’en faire son affaire.

Pas de comédie désopilante ? Pas d’histoire ? Mais quel est l’intérêt alors ?

Tout simplement dans l’incroyable panoplie d’humains que la caméra des frères Boukherma va chercher. Chaque personnage est ciselé, la caméra s’attardant longuement sur les défauts physiques, les rides, les visages bouffis, fatigués, vieillis, mais également souriants et pleins de personnalité. Au contraire des frères Dardennes, l’image ne cherche pas à servir un ultra-réalisme social. Il s’agit plutôt de mettre à l’honneur, d’héroïfier ces humains normaux. En contrepoint à ces anti-superhéros, le film offre un maire jeune, lisse et sans caractère ni le moindre esprit de décision (Loïc Richard). Parachuté depuis Paris, il se réfugie, symbole de cette lutte des classes omniprésente, derrière une visière anti-covid. Des Parisiens qui sont à la fois détestés par les locaux, mais nécessaires, car faisant tourner l’économie.

** Entracte publicitaire **

Acteur bordelais, Loïc Richard est réputé pour son travail de la voix. J’ai eu l’occasion de collaborer avec lui lorsqu’il a enregistré la version audiolivre de mon roman Printeurs, disponible sur toutes les plateformes d’audiobook. Autant il joue à merveille le personnage fade et lisse dans le film, autant il peut prendre des intonations sombres et inquiétantes dans sa lecture de Printeurs. Je ne pouvais quand même pas rater de placer cette anecdote 😉

=> https://voolume.fr/catalogue/sf-et-fantasy/printeurs/

** Fin de l’entracte, merci de regagner vos sièges **

Dans la première partie du film, Maja part à la chasse aux requins et tout se passe, à la grande surprise du spectateur, un peu trop facilement. La gendarme devient, malgré elle, une héroïne des réseaux sociaux. Mais au plus rapide est la montée, au plus dure est la chute. Au premier incident, qui n’est clairement pas le fait de Maja, elle devient la bête noire. Harcelée, elle en vient à paniquer dans une courte, mais puissante scène de rêve. Le propos est clair : le véritable requin est l’humain, alimenté par les réseaux sociaux et par les médias, symbolisé par une omniprésente radio réactionnaire qui attise les haines sous un vernis pseudohumoristique. Sous des dehors de petits paradis balnéaires, la haine et la rancœur sont tenaces. Sous la plage, les pavés. L’éden est amer.

À partir de la séquence onirique, le film perd progressivement tout semblant de réalisme et l’humour se fait de plus en plus rare. Les codes sont inversés : si l’humour était filmé de manière réaliste, les images d’action et d’angoisse sont offertes à travers la caméra d’une comédie absurde, l’apogée paradoxal étant atteint avec le rodéo impromptu de Blaise et le réveil surréaliste d’une Maja qui s’était pourtant noyée quelques minutes auparavant. Tout donne l’impression que Maja a continué son rêve, que la lutte contre le requin se poursuit dans son inconscient.

Étrange et déstabilisant, le film fonctionne entre autres grâce à un travail très particulier du cadre et de la couleur. Chaque plan résulte d’une recherche qui porte le propos, l’émotion. Lorsqu’elle est sur son ordinateur, Maja est baignée d’une lumière froide alors que son mari, à l’arrière-plan, représente la douceur chaleureuse du foyer. « Tu devrais arrêter Twitter », lance-t-il machinalement en partant dans la nature alors qu’elle reste enfermée devant son smartphone. Lors des confrontations entre les époux, la caméra se décentre souvent, donnant une perspective, un retrait, mais une intensité aux échanges.

Le titre lui-même porte une critique sociale très actuelle : « L’année passée c’était le covid, cette année le requin. Ce sera quoi l’année prochaine ? ». Le requin est le pur produit d’un réchauffement climatique entrainant des catastrophes face auxquelles tant les politiciens, les écologistes et les réactionnaires sont impuissants. Chacun ne cherchant finalement qu’à se dédouaner de toute responsabilité. Comme le dit le maire : « Ça va encore être la faute de la mairie ! ».

Sans y toucher, le film démontre le succès et la nécessité de décennies de lutte féministe. Le personnage principal est une femme qui s’est consacrée à sa carrière avec le soutien d’un mari effacé et très gentil (Kad Merad, incroyablement humain en mari bedonnant). Son assistante Eugénie est une femme (Christine Gautier). Pourtant, encore une fois, aucune insistance n’est placée sur le sujet. Le sexe des personnages importe peu, les relations étant, à tous les niveaux, purement basées sur leur caractère. Aucune séduction, aucune histoire d’amour autre qu’un mariage de longue date entre Maja et son mari, aucune mièvrerie. Le tout avec des interactions humaines profondément réalistes (dans des situations qui le sont évidemment beaucoup moins).

L’année du requin n’est certes pas le film de la décennie, la faute probablement à un scénario un peu simplet, il offre néanmoins une expérience cinématographique originale, nouvelle. Les frères Boukherma nous gratifiant d’un nouveau genre : celui de la parodie sérieuse qui ne se prend pas la tête. Fourmillant de trouvailles (la radio, la voix off particulièrement originale), le film mêle plaisir, clins d’œil aux cinéphiles, critique sociale et cadre original, le tout servi par des acteurs dont les talents sont particulièrement bien exploités.

Que demander de plus ?

Une morale ? Le film se termine justement sur une morale gentille, mais pas trop bateau et parfaitement appropriée : « Il y a deux types de héros. Ceux qui veulent sauver le monde et ceux qui veulent sauver ceux qu’ils aiment ».

Si l’année du requin ne sauve pas ni ne révolutionne le monde, il saura offrir quelques heures de plaisir à ceux qui cherchent des saveurs nouvelles sans se prendre la tête et qui aiment ce cynisme un peu grinçant qui ne s’inscrit dans aucune case précise. Il m’a clairement donné envie de découvrir Teddy, le premier film de ce jeune tandem de réalisateurs jumeaux. Et si après le loup-garou et le requin, ils décident de s’attaquer à la science-fiction, je suis volontaire pour leur pondre un scénario.

Recevez les billets par mail ou par RSS. Max 2 billets par semaine, rien d’autre. Adresse email jamais partagée et définitivement effacée lors du désabonnement. Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, lisez, offrez et partagez des livres.

Ce texte est publié sous la licence CC-By BE.

August 07, 2022

Si ma déconnexion totale a été un échec, si j’ai repris des connexions intempestives, mon usage de l’ordinateur a cependant été profondément modifié. Il est, par défaut, non connecté. Je suis conscient de chaque connexion. Et je ne regarde mes emails qu’une fois, parfois deux par jour. Ce dernier changement ayant grandement facilité grâce à une action que j’ai commencé il y a près de trois ans : supprimer mes comptes en ligne.

Au cours de ces trois dernières années, j’ai activement supprimé plus de 600 comptes sur différentes plateformes. Chaque fois que je reçois un mail d’une plateforme sur laquelle j’ai un compte inutilisé, je procède aux démarches, parfois longues et fastidieuses, pour le supprimer. Au cours de ces trois années, de nombreuses plateformes sont réapparues dont j’avais oublié jusqu’à l’existence.

Le travail a été de très longue haleine, mais commence à porter ses fruits et m’enseigne énormément sur cette marque de viande en boîte transformée en nom commun par des humoristes anglais déguisés en Vikings : le spam.

Les différents types de spams

J’ai identifié trois sortes de spams : le random-spam, l’expected-spam et le white-collar-spam.

Le random-spam est le pur spam dans la plus ancienne tradition du terme. Des emails envoyés à des millions de personnes sans aucune logique, pour vous vendre du viagra, pour vous convaincre d’installer un spyware, d’aider un prince nigérien à récupérer des millions ou de verser une rançon en bitcoins, car vous avez été soi-disant filmé en train de vous palucher devant un site porno. Une fois que votre adresse est publique, il n’y a rien à faire contre ce type de spam si ce n’est tenter de les filtrer. Il est complètement illégal. C’est d’ailleurs sa caractéristique première : il n’est lié à aucune entité juridique évidente. Vous ne pouvez pas vous plaindre ou vous désinscrire. Si le random-spam était une vraie plaie historiquement, je suis surpris de constater que sur mon adresse la plus spammée, une adresse publiée partout depuis quinze années, présente dans une kyrielle de bases de données publiques, je reçois en moyenne un random-spam tous les deux ou trois jours (il est automatiquement détecté comme tel et placé dans mon dossier spam, les faux négatifs étant très rares). La moitié de ces spams concernent les cryptomonnaies. J’en déduis que sur une adresse relativement récente et peu publique, vous recevrez très peu de ces spams.

L’expected-spam est exactement le contraire : c’est du spam envoyé par des plateformes ou des services sur lesquels vous êtes inscrits de votre plein gré. Notifications, enquête de satisfaction, newsletters ou autres annonces de nouveautés. La particularité est que vous pouvez vous désinscrire, même si ce n’est souvent que très temporairement (comme pour Facebook ou Linkedin, qui s’évertuent à créer des nouvelles newsletters ou catégories d’emails pour se rappeler à vous). Au final, il est très simple de se débarrasser de ce spam : supprimer définitivement votre compte de ce service. En théorie. Parce que certains continuent à vous envoyer des messages dont vous ne pouvez plus vous désabonner vu que vous n’avez plus de compte. Une menace de plainte RGPD suffit généralement à résoudre le « bug » informatique. Il est donc possible de réduire l’expected-spam à zéro (sauf s’il provient de votre employeur. Les entreprises se plaignent du manque de productivité des employés, mais paient des gens pour les assommer sous les newsletters internes complètement inutiles, allez comprendre).

Vient ensuite la troisième catégorie : le white-collar-spam. Le white-collar-spam est en fait du spam qui se donne des fausses impressions de légalité. Ce sont des entreprises qui ont acheté vos données et qui vous contactent comme si vous étiez inscrits chez eux. Un lien de désinscription est généralement toujours disponible. Mais plutôt que de me désinscrire simplement, je contacte chacune des entreprises et demande d’où elles tiennent mes données, les menaçant de poursuite RGPD. J’ai ainsi découvert que l’immense majorité des white-collar-spam proviennent, en francophonie, d’un ou deux fournisseurs. Ces fournisseurs sont absolument peu scrupuleux sur la manière dont ils collectent les données. Ce n’est pas étonnant : leur métier est littéralement d’emmerder les utilisateurs d’emails. Leurs clients sont les entreprises, les organisations non gouvernementales et les services publics. Ils classent les emails en catégories et vendent ces bases de données pour une durée limitée. Ce dernier point est important, car un an après avoir été en contact avec l’un de ces spammeurs-légaux-professionnels et avoir clairement fait comprendre que mes données ne pouvaient plus être vendues, j’ai reçu du spam d’un de leur client. Il s’est avéré que le client, un service public français à vocation culturelle, avait réutilisé une base de données achetée deux ans auparavant, ce qui était interdit par son contrat.

J’ai donné le nom « white-collar-spam », car ce spam n’est guère différent du random-spam illégal si ce n’est qu’il est accompli par des sociétés ayant pignon sur rue très fières de leur métier de spammeur. Au lieu de lutter contre le spam, nous en avons fait une activité honorable et rémunératrice !

Outre ces quelques acteurs professionnels du spam, une grande quantité de white-collar-spam provient indirectement de Linkedin. En effet, certains outils permettent aux professionnels du marketing (le nouveau nom pour spammeur) de récolter les adresses mails, même cachées, de leurs contacts Linkedin. Si vous avez un de ces très nombreux spammeurs dans vos contacts sur ce réseau, vous êtes foutu. La solution la plus simple : supprimer votre compte Linkedin et laisser les spammeurs entre eux (la fonction première de ce réseau). Le simple fait d’effacer mon compte Linkedin a divisé par deux, en quelques semaines, le nombre de spams que je recevais.

La majorité du spam que je reçois aujourd’hui est donc ce white-collar-spam qui est plus ou moins légal et complètement immoral.

Une idée m’est venue pour le combattre très simplement : interdire la revente d’une donnée identifiante sans l’accord de la personne concernée. Simple comme tout : si une société souhaite vendre des données, elle doit en demander l’autorisation à chaque transaction. Cette règle s’appliquerait également en cas de rachat d’une société par une autre ou en cas de transfert d’une entité juridique à une autre. Il semble en effet évident que l’on peut partager ses données avec une entité, mais ne pas vouloir le faire avec une autre. La société machin sur laquelle vous avez un compte se fait racheter par truc ? Vous devez marquer votre accord sans quoi vos données seront effacées après un délai de quelques mois. Simple à implémenter, simple à surveiller, simple à légiférer.

Ce qui signifie que si nous avons du spam, c’est parce que nous le voulons. Comme la cigarette ou la pollution industrielle, le spam fait partie des nuisances dont nous nous plaignons sans réellement oser les combattre parce que nous sommes persuadés qu’il y’a une raison valable pour laquelle ça existe, parce que nous nous y sommes habitués et parce que certains se sont tellement enrichis avec qu’ils peuvent influencer le pouvoir politique et médiatique. Pire : nous admirons même un peu ceux qui gagnent leur vie de cette manière et sommes prêts à travailler pour eux si une offre juteuse se présente.

Les bénéfices insoupçonnés de la suppression de compte

La solution la plus radicale et qui fonctionne à merveille reste de supprimer tous ses comptes. C’est un processus de longue haleine : je me suis découvert plus de 600 comptes au fur et à mesure que je fouillais mon gestionnaire de mot de passe, les comptes liés à mes comptes Google, Facebook et LinkedIn. Chaque fois que je crois avoir fait le tour, des comptes complètement oubliés réapparaissent dans ma boîte mail lorsqu’ils modifient leurs conditions d’utilisation.

Supprimer un compte qu’on n’utilise plus est un processus pénible : réussir à se reconnecter, à trouver la procédure pour supprimer qui est souvent planquée et artificiellement complexe (pas toujours). Mais c’est encore plus difficile lorsqu’il s’agit d’un compte qu’on utilise ou qu’on pense pouvoir réutiliser. Le plus difficile étant lorsqu’un historique existe, historique souvent agrémenté d’un score : nombre d’amis, points, karma, récompenses, badges… Après Facebook et Twitter, Reddit et Quora furent probablement les comptes les plus difficiles à supprimer. Je me suis rendu compte que je tirais une fierté absurde de mon karma et de mes scores alors que je n’ai jamais été un utilisateur assidu de ces plateformes.

Mention spéciale tout de même à ces sites qui ont prétendu avoir effacé mes données sans réellement le faire. Dans le cas d’une chaine de restaurants de sushi, le gestionnaire s’est contenté de rajouter « deleted_ » devant mon adresse email. Ce fut encore pire pour un grand site immobilier belge. Plus d’un an après la suppression totale de mes données, le site s’est soudain mis à m’envoyer journalièrement le résultat d’une recherche que j’avais enregistrée une décennie auparavant. Sans possibilité de désactiver, mon compte étant officiellement supprimé. Il a fallu plusieurs semaines d’échanges par email pour résoudre le problème et obtenir un semblant d’explication : un très vieux backup aurait été utilisé pour restaurer certaines bases de données. Je vous laisse juge de la crédibilité d’une telle raison.

De toutes mes histoires, j’ai appris une généralité : l’immense majorité des services est en réalité incapable de supprimer vos données, que ce soit par malveillance ou par incompétence. Toute donnée entrée sur un site doit être considérée comme définitivement compromise et potentiellement publique. Si j’ai très souvent accordé le bénéfice du doute, attribuant les erreurs ou difficultés à l’incompétence, j’ai plusieurs fois été confronté à ce qui ne pouvait être que des mensonges manifestes et éhontés. Une grande majorité des services web réclamant vos données sont donc soit incompétents, soit profondément malhonnêtes. Soit les deux. L’exception venant des petits services artisanaux, généralement développés par une toute petite équipe. Dans tous les cas de ce genre, l’effacement s’est fait rapidement, proprement et parfois avec un mot gentil personnalisé. Preuve que la suppression n’est pas un acte techniquement insurmontable.

Contrairement à l’abstinence ou au blocage d’accès à ces sites, la suppression du compte a eu chez moi un impact absolument incroyable. Du jour au lendemain, j’ai arrêté de penser à ce qui se passait sur ces plateformes. Du jour au lendemain, j’ai arrêté de penser à ce qui pourrait avoir du succès sur ces plateformes. J’ai arrêté de penser pour ces plateformes. J’ai arrêté de me plier à leurs règles, de travailler inconsciemment pour elles. J’ai arrêté d’avoir envie de les consulter. Et lorsque me vient l’envie d’y poster ou d’y répondre, le fait de devoir recréer un compte pour l’occasion est assez pour m’arrêter dans mon élan et me faire remarquer que j’ai mieux à faire. Lorsqu’une plateforme est soudain vraiment nécessaire, je recrée un compte, si possible avec une adresse jetable et le supprime après emploi. Une fois le réflexe pris, ce n’est plus tellement contraignant.

Plateformes et militantisme

N’ayant pas supprimé mon compte Mastodon, par simple soutien idéologique au projet, je me retrouve mécaniquement à explorer cette plateforme. Plateforme elle-même complètement biaisée (si je devais la considérer comme représentative de la France, Mélenchon aurait dû devenir président avec près de 95% des voix, le reste étant essentiellement des abstentions).

Dans le militantisme, il existe deux écoles. La première prétend qu’il faut aller chercher les gens où ils sont. Militer pour le logiciel libre sur Facebook par exemple. La seconde soutient qu’il faut d’abord être fidèle à ses propres valeurs, ses convictions.

Je suis désormais convaincu de la seconde approche. Je pense avoir soutenu la première approche pendant des années entre autres pour justifier ma quête égotique sur les réseaux propriétaires, pour résoudre mon conflit interne. Car, quelle que soit l’intention derrière un message, son impact sera toujours contrôlé par la plateforme sur laquelle il est posté. Le simple fait d’utiliser une plateforme nous déforme et nous conforme à ladite plateforme.

Je pense également qu’il ne faut pas aller « chercher les gens là où ils sont ». Ne pas crier pour tenter de couvrir le bruit ambiant. Il faut au contraire construire des espaces de calme, des espaces personnels et faire confiance aux humains pour les trouver lorsqu’ils en ont besoin. Le simple fait d’avoir un compte sur une plateforme justifie pour tous vos contacts le fait de rester sur cette plateforme. Le premier qui quitte la plateforme s’exclut du groupe. Le second force le groupe à se poser des questions. Le troisième implique que « le groupe » n’est tout simplement plus sur cette plateforme, que celle-ci est devenue inutile dans le cadre du groupe.

Aucun discours ne convainc autant que montrer l’exemple. Faire plutôt que dire. Être plutôt que convaincre. Vivre ses propres choix, sa propre personnalité et respecter ceux qui en font d’autres en acceptant que cela puisse nous éloigner.

Oui, en supprimant mes comptes j’ai raté des opportunités sociales. Mais soit je ne m’en suis pas rendu compte, ce qui a épargné mon énergie mentale, soit cela a eu pour impact de faire prendre conscience à mon entourage qu’ils ne pouvaient plus faire entièrement confiance à Facebook ou Whatsapp. Dans tous les cas, le rapport coût/bénéfice s’est révélé disproportionnellement en faveur de la suppression.

À chaque compte effacé, j’ai eu le sentiment qu’on m’enlevait un poids des épaules. Je me sentais revivre. Certes, je perdais une « audience potentielle », mais j’y gagnais en liberté, en plaisir d’écrire sur mon blog, sur mon gemlog voire sur ma machine à écrire plutôt que de sans cesse réagir, répondre, être en réaction (au sens le plus Nitzchéen du terme).

Si j’ai replongé dans la connexion intermittente, un progrès énorme s’est fait : la connexion m’ennuie de plus en plus. Le nombre de plateformes sur lesquelles lire du contenu s’est à ce point restreint que j’en fais très vite le tour. J’ai également compris que mon addiction n’est pas uniquement due à la connexion, elle est également technophile. J’aime être sur mon ordinateur, devant mon écran. Je tente de trouver des excuses pour garder les mains sur le clavier, pour mettre à jour un logiciel, configurer mon environnement, améliorer mes processus, découvrir, coder. Bref, « chipoter ».

La découverte de cette composante de mon addiction m’a convaincu de faire entrer ma déconnexion dans une nouvelle phase. Celle de la matérialité.

Recevez les billets par mail ou par RSS. Max 2 billets par semaine, rien d’autre. Adresse email jamais partagée et définitivement effacée lors du désabonnement. Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, lisez, offrez et partagez des livres.

Ce texte est publié sous la licence CC-By BE.

August 02, 2022

Zoals altijd zijn we als EU of Europa achtergesteld omdat we geen enkel militair antwoord hebben op de zaken die geostrategisch gaande zijn.

Voor Ukraine kunnen we weinig tot niets doen omdat we geen enkel antwoord hebben op de vraag ‘vanwaar komt het gas dan wel?’

Deze vraag is te belachelijk simpel en toch kan geen enkel EU politicus ze beantwoorden.

Nochtans was het antwoord hierop haalbaar: men had werk kunnen maken van alternatieve energiebronnen. Maar dat vonden de heren en dames EU-politici niet nodig. Overbodig. En zo verder.

M.a.w. zijn ze volstrekt incompetent. Ik schrijf ze effectief volledig af. Want ze hadden daar wel een antwoord op moeten kunnen formuleren. De idioten die er nu nog zitten kunnen dat niet. Daarom zijn het losers en daarom horen we ze te ontslaan uit hun functie. Helaas zijn het ook populisten en daarom zal hun ontslag vele jaren duren (zie Brexit).

Richting Taiwan doen de EU politici alweer hun belachelijke best om wat dan ook te betekenen. Maar iets betekenen doen ze helemaal niet. Ze doen niets dat er toe doet.

Omdat ze collectief besloten hebben geen EU-leger te hebben.

Daarom zijn ze onbelangrijk. Insignificant. Incompetent. Onbelangrijk.

August 01, 2022

I wanted to outline the development and deployment workflow I use on dri.es, my personal website.

My site uses Drupal (obviously) and runs on Acquia Cloud (of course), but a lot of this is a best practice for any web application.

I manage my website's code and configuration in Git. Each time I commit a change to my Git repository, I go through the following steps:

  1. I create a staging environment to test my code before deploying it to production. It's a complete staging environment: not just PHP, MySQL and Nginx, but also Varnish, Memcache, etc.
  2. I check out my Git repository. My Git repository hosts my custom files only. It's a best practice not to commit Drupal core or third-party Drupal modules to your Git repository.
  3. I run PHP Code Sniffer to make sure my code conforms to my coding style rules. I specify my coding style rules in phpcs.xml and use phpcs to make sure my code adheres to them. If not, phpcbf tries to fix my code automatically. I like my code tidy.
  4. I run PHPStan, a static code analysis tool for PHP, that scans my code base for bugs. It will find dead code, type casting problems, incorrect function arguments, missing type hints, unknown function calls, and much more. PHPStan is a fantastic tool.
  5. I run PHP Unit, a PHP testing framework, to make sure my unit tests pass.
  6. I run phpcs-security-audit, a static code analysis tool for PHP. It scans my PHP code for security vulnerabilities and security weaknesses.
  7. I run ESLint, a static code analysis tool for JavaScript. It scans my JavaScript code for security vulnerabilities and weaknesses.
  8. I run nodejs-scan to find insecure code patterns in my Node.js applications. I don't use Node.js at the moment though.
  9. I also run Semgrep, a static code analysis tool for a variety of programming languages.
  10. I run Rector to make sure I don't use deprecated Drupal code. When I do, Rector will try to programmatically update any deprecated code that it finds.
  11. As my Git repository only has custom files, I use Composer to download and install the latest version of Drupal and all third-party modules and components.
  12. I run drush pm:security. Drush is a Drupal-specific tool, and the pm:security option verifies that I have no insecure dependencies installed.

This all might sound like a lot of work to set up, and it can be. For Acquia customers and partners, Acquia Code Studio automates all the steps above. Acquia Code Studio is a fully managed CI/CD based on Gitlab, with specific steps optimized for Drupal. In 20+ years of working on Drupal, it's my best webops workflow yet. It couldn't be easier.

A screenshot of the Acquia Code Studio UI showing some of the automated tests.A screenshot of Acquia Code Studio showing the automated tests feature.

Acquia Code Studio also takes care of automating dependency updates. Code Studio regularly checks if Drupal or any of its dependencies have a new release available. If there is a new release, it will run all the steps above. When all of the above tools pass, Acquia Code Studio can deploy new code to production with one click of a button.

A screenshot of the Acquia Code Studio UI showing that some Composer packages have been updated.A screenshot of Acquia Code Studio showing the automated update feature.

I love it!

July 30, 2022

Many Bluetooth Low Energy (BLE) devices broadcast data using manufacturer-specific data or service data in their advertisements. The data format is often defined in a specification or should be reverse-engineered.

If you want to decode the binary data format into usable chunks of data from various data types in your own Python program, I find the Construct library quite an accessible solution. And Bleak is my favorite BLE library in Python, so first install Bleak and Construct:

pip3 install bleak construct

As an example, let's see how you could decode iBeacon advertisements with Bleak and Construct in Python.

The iBeacon specification

The iBeacon specification, published by Apple, is officially called Proximity Beacon. The idea is to have Bluetooth beacons advertise their presence in order to calculate their approximate distance. You can find the iBeacon specification online.

The specification lists the format of the iBeacon advertising packet. This always consists of two advertising data structures: flags (of length 2) and manufacturer-specific data (of length 26). That’s why an iBeacon advertising packet is always 30 bytes long (1 + 2 + 1 + 26). Here's the structure of the complete packet:

/images/proximity-beacon-advertising-packet.png

We're specifically interested in the second data structure with type 0xff, which signifies that it's manufacturer-specific data. The first two bytes of these manufacturer-specific data are always the company ID. To know which company ID is linked to which company, consult the list of all registered company identifiers. Normally the company ID is the ID of the manufacturer of the device. However, Apple allows other manufacturers to use its company ID for iBeacon devices if they agree to the license.

Note

The company ID is a field of two bytes that are sent as a little-endian value. If you look at an iBeacon packet capture in Wireshark, the bytes on the air are 4c 00. However, Apple's real company ID is 00 4c, or 76 in decimal.

If you want to know more about the meaning of the iBeacon packet's fields, consult the document Getting Started with iBeacon published by Apple.

Decoding iBeacon advertisements

Now that you know the format, let's see how to scan for iBeacon advertisements and decode them:

ibeacon_scanner/ibeacon_scanner.py (Source)

"""Scan for iBeacons.

Copyright (c) 2022 Koen Vervloesem

SPDX-License-Identifier: MIT
"""
import asyncio
from uuid import UUID

from construct import Array, Byte, Const, Int8sl, Int16ub, Struct
from construct.core import ConstError

from bleak import BleakScanner
from bleak.backends.device import BLEDevice
from bleak.backends.scanner import AdvertisementData

ibeacon_format = Struct(
    "type_length" / Const(b"\x02\x15"),
    "uuid" / Array(16, Byte),
    "major" / Int16ub,
    "minor" / Int16ub,
    "power" / Int8sl,
)


def device_found(
    device: BLEDevice, advertisement_data: AdvertisementData
):
    """Decode iBeacon."""
    try:
        apple_data = advertisement_data.manufacturer_data[0x004C]
        ibeacon = ibeacon_format.parse(apple_data)
        uuid = UUID(bytes=bytes(ibeacon.uuid))
        print(f"UUID     : {uuid}")
        print(f"Major    : {ibeacon.major}")
        print(f"Minor    : {ibeacon.minor}")
        print(f"TX power : {ibeacon.power} dBm")
        print(f"RSSI     : {device.rssi} dBm")
        print(47 * "-")
    except KeyError:
        # Apple company ID (0x004c) not found
        pass
    except ConstError:
        # No iBeacon (type 0x02 and length 0x15)
        pass


async def main():
    """Scan for devices."""
    scanner = BleakScanner()
    scanner.register_detection_callback(device_found)

    while True:
        await scanner.start()
        await asyncio.sleep(1.0)
        await scanner.stop()


asyncio.run(main())

First it defines a Struct object from the Construct library, and calls it ibeacon_format. A Struct is a collection of ordered and usually named fields. 1 Each field in itself is an instance of a Construct class. This is how you define the data type of bytes in an iBeacon data structure. In this case the fields are:

  • Const(b"\x02\x15"): a constant value of two bytes, because these are always fixed for an iBeacon data structure.

  • Array(16, Byte): an array of 16 bytes that define the UUID.

  • Int16ub for both the major and minor numbers, which are both unsigned big-endian 16-bit integers.

  • Int8sl for the measured power, which is a signed 8-bit integer.

Now when the device_found function receives manufacturer-specific data from Apple, it can easily parse it. It just calls the parse function on the ibeacon_format object, with the bytes of the manufacturer-specific data as its argument. The result is an object of the class construct.lib.containers.Container, with the fields that are defined in the ibeacon_format struct. That's why you can just refer to the fields like ibeacon.major, ibeacon.minor and ibeacon.power.

However, ibeacon.uuid returns a construct.lib.containers.ListContainer object, which is printed as a list of separate numbers. To print it like a UUID, first convert it to bytes and then create a UUID object from these bytes.

Note

This Python program doesn't explicitly check for the company ID and the first two bytes in the manufacturer-specific data. The code just assumes it receives iBeacon data and catches exceptions if this assumption proves false: the KeyError exception happens if there's no 0x004c key in the manufacturer_data dictionary and the ConstError exceptions happens if the first two bytes of the data don't equal the constant b"\x02\x15". This common Python coding style is called EAFP (easier to ask for forgiveness than permission), and in many cases it makes the code easier to follow. The other style, testing for all conditions before, is called LBYL (look before you leap).

If you run this program, it will scan continuously for iBeacons and shows their information:

$ python3 ibeacon_scanner.py
UUID     : fda50693-a4e2-4fb1-afcf-c6eb07647825
Major    : 1
Minor    : 2
TX power : -40 dBm
RSSI     : -80 dBm
-----------------------------------------------
UUID     : d1338ace-002d-44af-88d1-e57c12484966
Major    : 1
Minor    : 39904
TX power : -59 dBm
RSSI     : -98 dBm
-----------------------------------------------

This will keep scanning indefinitely. Just press Ctrl+c to stop the program.

1

A Struct object in Construct is comparable to a struct in the C programming language.

Long time no Radiohead here, so let’s fix that shall we? Here’s Thom Yorke solo in Zermatt (Switzerland) playing songs from Radiohead, his solo-records and his new band (The Smile). If anything this is a testament to the great songwriter the man is! Also remarkable; he seems so much more at ease on stage now, maybe having accepted the spotlights which sometimes seemed too much for him to cope with.

Source

July 29, 2022

Setup of a HifiBerry AMP2...on a Rapsberry Pi 2.

First attempt was with Volumio, as advised by a friend. Well that works, but I personally find the interface a horror, and I seem to lose control of the Pi since Volumio is a full OS that seems only accessible by web interface. No thanks.

Using Raspberry Pi OS:

- download Raspberry Pi OS lite (command line is fine)

- extract the image

- dd the image to the sd-card

dd if=/home/paul/2022-04-04-raspios-bullseye-armhf-lite.img of=/dev/sdb bs=1M

- mount it to enable ssh

touch /boot/ssh

- I also had to set a password for the pi user, since 'raspberry' was not accepted?

- Boot the Pi (the HifiBerry is still attached)

- ssh into the Pi 

apt update
apt upgrade
apt install vim
vi /boot/config.txt
#dtparam=audio=on
#dtoverlay=vc4-kms-v3d

# added by paul 2022-07-29 for HifiBerry AMP2
dtoverlay=hifiberry-dacplus
force_eeprom_read=0


Comment out the first two lines, add the last two. Check here for other HifiBerries.

Now, before using mplayer or something, LOWER THE VOLUME! Use a really low value, and gradually go up while playing music since the default is extremely loud.

amixer -c 0 sset Digital "20%"

Thanks for listening :)

July 25, 2022

Een eigen Europees leger starten. Waarbij ieder Europees land haar eigen expertise in de groep werpt.

Afspraken maken met Rusland over de energievoorziening van Europa.

Een nieuw veiligheidspakt met Rusland maken opdat er zo weinig mogelijk conflicten in Europa zullen zijn.

Machtsprojectie doen vanuit Europa, met het Europees leger. We moeten opnieuw leren wat het is om aan geostrategie te doen. We moeten dat Europees leger durven inzetten om onze strategische doelen te behalen. We moeten niet verlegen zijn om de wereld duidelijk te maken dat wij zulke strategische doelen hebben.

Het conflict in Oekraïne beïndigen. Want het dient ons (Europeanen) en Russen niet. We zijn beiden benadeeld door dit conflict. We hebben er beiden baad bij om dit te beïndigen.

Durven praten over Europa en niet enkel over de Europese Unie.