Planet Grep

Planet'ing Belgian FLOSS people

Planet Grep is maintained by Wouter Verhelst. All times are in UTC.

September 25, 2023

Sometimes it’s convenient to retrieve the user creation statement and to copy it to another server.

However, with the new authentication method used as default since MySQL 8.0, caching_sha2_password, this can become a nightmare as the output is binary and some bytes can be hidden or decoded differently depending of the terminal and font used.

Let’s have a look:

If we cut the create user statement and paste it into another server what will happen ?

We can see that we get the following error:

ERROR: 1827 (HY000): The password hash doesn't have the expected format.

How could we deal with that ?

The solution to be able to cut & paste the authentication string without having any issue, is to change it as a binary representation (hexadecimal) like this:

And then replace the value in the user create statement:

But there is an easier way. MySQL Server provides the possibility to display the Create User Statement directly with the hexadecimal representation of the authentication string using the variable print_identified_with_as_hex:

The user creation succeeded, and now let’s test to connect to this second server using the same credentials:

Using MySQL Shell Plugins

I’ve updated the MySQL Shell Plugins available on GitHub to use the same technique to be able to cut & paste the user creation and the grants:

And for MySQL HeatWave on OCI ?

Can we use the generated user creation statement and grants with MySQL HeatWave ?

For the user creation, there is no problem and it will work. However for the grants there is a limitation as some of the grants are not compatible or allowed within MySQL HeatWave.

The list of grants allowed in HeatWave is available on this page.

Let’s try:

As you can see, some of the privileges are not allowed and the GRANT statements fail.

You have the possibility to ask to the MySQL Shell Plugin to strip the incompatible privileges, using the option ocimds=True:

Now you can use the generated SQL statements with a MySQL HeatWave instance:

Conclusion

As you can see, the default authentication plugin for MySQL 8.0 and 8.1 is more secure but can be complicated to cut and paste. But MySQL Server provides the required variable to generate statements that can transferred and pasted easily.

Enjoy MySQL !

September 24, 2023

Op microcontrollerbordjes zoals een Arduino, Raspberry Pi Pico of ESP32 kun je allerlei leds, knoppen en sensoren aansluiten. Veel van die bordjes hebben een ingebouwde wifi-chip, waardoor je ze op afstand kunt aansturen. Maar soms is wifi niet mogelijk, te lastig of gewoon helemaal niet nodig.

Gelukkig zijn de meeste microcontrollerbordjes uitgerust met een usb-aansluiting en die kun je ook gebruiken om vanaf je computer opdrachten naar de microcontroller te sturen of informatie zoals sensordata terug te krijgen. Dat kan eenvoudig via een seriële interface over USB CDC. Onder Windows is het apparaat dan zichtbaar als een COM-poort, onder Linux als een apparaat zoals /dev/ttyACM0 en onder macOS /dev/cu.usbmodem<ennogiets>. Software op je computer kan dan met de microcontroller communiceren via deze COM-poort of het juiste apparaatbestand.

In het artikel Zo breid je met microcontrollers je computer uit met extra functies op id.nl beschrijf ik hoe je dit doet met CircuitPython op een microcontrollerbordje aan de ene kant en de Web Serial API op de computer aan de andere kant (helaas alleen ondersteund onder Chrome). Zo kun je eenvoudig een webinterface maken om een led op je via usb aangesloten microcontrollerbordje aan te sturen:

/images/microcontroller-usb-webpagina.png

Op dezelfde manier kun je in een webinterface een temperatuursensor van een microcontrollerbordje uitlezen:

/images/microcontroller-usb-temperature.png

Zolang je in je seriële communicatie een eenduidig protocol definieert (1 om de led aan te doen, 0 om ze uit te doen, ...), kun je dezelfde aanpak nog op allerlei manieren uitwerken. Zo hoef je aan de kant van de microcontroller geen CircuitPython te gebruiken. Je kunt exact hetzelfde implementeren met eenvoudige Arduino-code. Ik vind de Digispark bijvoorbeeld een handig microcontrollerbordje. Je laat het via de Arduino-bibliotheek DigiCDC met je computer communiceren.

Ook aan de computerkant zijn er talloze alternatieven voor Web Serial in de browser. Zo kun je in Python een programma schrijven dat via de bibliotheek pySerial met de seriële poort communiceert. Zolang je ervoor zorgt dat beide kanten hetzelfde protocol gebruiken, zijn de verschillende alternatieven uitwisselbaar. Je kunt hetzelfde microcontrollerbordje aansturen via zowel je webbrowser als een Python-programma.

Ik vind USB CDC handig omdat alle grote besturingssystemen het ondersteunen zonder dat je nog speciale drivers hoeft te installeren. Ik gebruik het daarom in mijn lessen "Basis programmeren", waarin ik de studenten van het graduaat Internet of Things leer om te programmeren in Python. In de les over seriële communicatie met pySerial programmeer ik dan gewoon enkele microcontrollerbordjes met mijn CircuitPython-code en deel ze uit in de klas, waarna ik me geen zorgen hoef te maken over driverproblemen. [1] Zo kan ik me focussen op het programmeren.

[1]

Oorspronkelijk wilde ik dat doen met de Digispark, maar dit bordje heeft onder Windows drivers nodig, die niet voor recente Windows-versies bestaan.

September 21, 2023

If you are deploying MySQL on containers, one of the first tasks is to find the right image.

There’s a certain amount of confusion, especially when we’re trying to help someone who’s having problems with their deployment.

For example, when people say I’m using the official docker image… what does that really mean?

Docker Hub, provides their official image (https://hub.docker.com/_/mysql), but this is not the official MySQL image that we, the MySQL Team at Oracle, support.

Before the mess with Docker Hub ([1], [2], [3]), the real official images for MySQL were also available here: https://hub.docker.com/r/mysql/mysql-server/.

Has you can see, there is no update and the last available one is 8.0.32 from January 2023.

So if you are looking for the latest official, supported, versions of MySQL, including Innovation Releases, you can find them in the Oracle Container Registry:

And the registry contains all the recent versions including images for ARM:

The Oracle Container Registry includes container images for all Oracle products like Java, Oracle Linux and MySQL.

If you are looking for a MySQL container image, you know where to find it.

Enjoy deploying MySQL in containers !

September 17, 2023

FOSDEM 2024 will take place at the ULB on the 3th and 4th of February 2024. As has become traditional, we offer free and open source projects a stand to display their work "in real life" to the audience. You can share information, demo software, interact with your users and developers, give away goodies, sell merchandise or accept donations. Anything is possible! We offer you: One table (180x80cm) with a set of chairs and a power socket. Fast wireless internet access. A spot on stands.fosdem.org. You can choose whether you want the spot for the entire conference, or simply舰

How to migrate Jails from ezjail to BastilleBSD

daemon_hammer

In my previous blog post, I reviewed BastilleBSD. In this post, we go through the required steps to migrate the Jails from ezjail to BastilleBSD.

ezjail test Jail

To test the Jail migration, we’ll first create a test Jail with ezjail. This test Jail will migrate to a BastilleBSD Jail.

Create the test ezjail Jail

We use the ezjail-admin create staftestje001 'vtnet0|<ip>' command to create the test Jail.

root@pi-rataplan:~ # ezjail-admin create staftestje001 'vtnet0|<ip>'
Warning: Some services already seem to be listening on all IP, (including 192.168.1.51)
  This may cause some confusion, here they are:
root     nfsd       93987 5  tcp4   *:2049                *:*
root     nfsd       93987 6  tcp6   *:2049                *:*
root     mountd     92576 6  udp6   *:1014                *:*
root     mountd     92576 7  tcp6   *:1014                *:*
root     mountd     92576 8  udp4   *:1014                *:*
root     mountd     92576 9  tcp4   *:1014                *:*
root     ntpd       88967 20 udp6   *:123                 *:*
root     ntpd       88967 21 udp4   *:123                 *:*
root     rpc.statd  86127 4  udp6   *:654                 *:*
root     rpc.statd  86127 5  tcp6   *:654                 *:*
root     rpc.statd  86127 6  udp4   *:654                 *:*
root     rpc.statd  86127 7  tcp4   *:654                 *:*
root     rpcbind    85696 6  udp6   *:111                 *:*
root     rpcbind    85696 7  udp6   *:702                 *:*
root     rpcbind    85696 8  tcp6   *:111                 *:*
root     rpcbind    85696 9  udp4   *:111                 *:*
root     rpcbind    85696 10 udp4   *:996                 *:*
root     rpcbind    85696 11 tcp4   *:111                 *:*
root@pi-rataplan:~ # 

Review the created Jail.

root@pi-rataplan:~ # ezjail-admin list
STA JID  IP              Hostname                       Root Directory
--- ---- --------------- ------------------------------ ------------------------
ZS  N/A  192.168.1.51    staftestje001                  /usr/jails/staftestje001
root@pi-rataplan:~ #

Start the Jail with ezjail-admin start staftst1

root@pi-rataplan:~ # ezjail-admin start staftst1 
Starting jails: staftst1.
/etc/rc.d/jail: WARNING: Per-jail configuration via jail_* variables  is obsolete.  Please consider migrating to /etc/jail.conf.
root@pi-rataplan:~ # 

Access the console with ezjail-admin console

root@pi-rataplan:~ # ezjail-admin console staftestje001
FreeBSD 13.2-RELEASE-p2 GENERIC

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List:        https://www.FreeBSD.org/lists/questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

To change this login announcement, see motd(5).
root@staftestje001:~ # 

Add a user.

root@staftestje001:~ # adduser 
Username: staf
Full name: staf
Uid (Leave empty for default): 
Login group [staf]: 
Login group is staf. Invite staf into other groups? []: wheel
Login class [default]: 
Shell (sh csh tcsh nologin) [sh]: 
Home directory [/home/staf]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: 
Use an empty password? (yes/no) [no]: 
Use a random password? (yes/no) [no]: 
Enter password: 
Enter password again: 
Lock out the account after creation? [no]: no
Username   : staf
Password   : *****
Full Name  : staf
Uid        : 1001
Class      : 
Groups     : staf wheel
Home       : /home/staf
Home Mode  : 
Shell      : /bin/sh
Locked     : no
OK? (yes/no): yes
adduser: INFO: Successfully added (staf) to the user database.
Add another user? (yes/no): no
Goodbye!

Become the user test user and create some files.

root@staftestje001:~ # su - staf
You can use aliases to decrease the amount of typing you need to do to get
commands you commonly use.  Examples of fairly popular aliases include (in
Bourne shell style, as in /bin/sh, bash, ksh, and zsh):

	alias lf="ls -FA"
	alias ll="ls -lA"
	alias su="su -m"

In csh or tcsh, these would be

	alias lf ls -FA
	alias ll ls -lA
	alias su su -m

To remove an alias, you can usually use 'unalias aliasname'.  To list all
aliases, you can usually type just 'alias'.
staf@staftestje001:~ $ 
staf@staftestje001:~ $ vi testfile

bastille-icon.png

Migrate the ezjail Jail to BastilleBSD

Stop the ezjail jail

Execute ezjail-admin stop to stop the Jail.

root@pi-rataplan:~ # ezjail-admin stop staftestje001
Stopping jails: staftestje001.
root@pi-rataplan:~ # 

Archive

Use ezjail-admin archive to create a tar dump of the Jail.

root@pi-rataplan:~ # ezjail-admin archive staftestje001
pax: Access/modification time set failed on: ./var/empty <Operation not permitted>
Warning: Archiving jail staftestje001 was not completely successful.\n  Please refer to the output above for problems the archiving tool encountered.\n  You may ignore reports concerning setting access and modification times.\n  You might want to check and remove /usr/jails/ezjail_archives/staftestje001-202308161229.21.tar.gz.Warning: Archiving jail staftestje001 was not completely successful. For a running jail this is not unusual.
root@pi-rataplan:~ # 

The tar file is created at /usr/jails/ezjail_archives

root@pi-rataplan:~ # ls -l  /usr/jails/ezjail_archives
total 267233
-rw-r--r--  1 root  wheel  136712524 Aug 16 12:29 staftestje001-202308161229.21.tar.gz
root@pi-rataplan:~ # 

Import

It’s possible to import the ezjail archive with bastille import.

[root@pi-rataplan ~]# bastille import /usr/jails/ezjail_archives/staftestje001-202308161229.21.tar.gz 
Importing 'staftestje001' from foreign compressed .tar.gz archive.
Preparing ZFS environment...
Extracting files from 'staftestje001-202308161229.21.tar.gz' archive...
tar: Removing leading '/' from member names
Generating jail.conf...
Updating symlinks...
Container 'staftestje001' imported successfully.
[root@pi-rataplan ~]# 

List the Jails.

[root@pi-rataplan ~]# bastille list -a
 JID              State  IP Address           Published Ports  Hostname         Release          Path
 bastille-tst001  Up     192.168.1.50         -                bastille-tst001  13.2-RELEASE-p2  /usr/local/bastille/jails/bastille-tst001/root
 staftestje001    Down   vtnet0|192.168.1.51  -                staftestje001    13.2-RELEASE-p2  /usr/local/bastille/jails/staftestje001/root
[root@pi-rataplan ~]# 

Correct the IP Address

Our archived test Jail is imported.

We defined the interface as part of the ezjail-admin create command. But this ended up in the IP Address configuration.

Let’s see how this is defined in our Jail configuration.

Go to the Jail dataset.

root@pi-rataplan:~ # cd /usr/local/bastille/jails/staftestje001/
root@pi-rataplan:/usr/local/bastille/jails/staftestje001 # 

List the configuration files.

root@pi-rataplan:/usr/local/bastille/jails/staftestje001 # ls
fstab
fstab.ezjail
jail.conf
prop.ezjail-staftestje001-202309032022.27-pi_rataplan-13.2_RELEASE_p2-aarch64
root
root@pi-rataplan:/usr/local/bastille/jails/staftestje001 #

Edit the jail.conf

root@pi-rataplan:/usr/local/bastille/jails/staftestje001 # vi jail.conf
staftestje001 {
  devfs_ruleset = 4;
  enforce_statfs = 2;
  exec.clean;
  exec.consolelog = /var/log/bastille/staftestje001_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = staftestje001;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/staftestje001/fstab;
  path = /usr/local/bastille/jails/staftestje001/root;
  securelevel = 2;

  interface = vtnet0;
  ip4.addr = vtnet0|192.168.1.51;
  ip6 = disable;
}

The interface is defined in the interface config and the ip4.addr. Remove the interface from the ip4.addr.

  ip4.addr = 192.168.1.51;

Execute bastille list -a to verify.

root@pi-rataplan:/usr/local/bastille/jails/staftestje001 # bastille list -a
 JID              State  IP Address           Published Ports  Hostname         Release          Path
 bastille-tst001  Down   192.168.1.50         -                bastille-tst001  13.2-RELEASE-p2  /usr/local/bastille/jails/bastille-tst001/root
 staftestje001    Down   192.168.1.51   

Verify

Start the Jail with bastille start

root@pi-rataplan:/usr/local/bastille/jails/staftestje001 # bastille start staftestje001
[staftestje001]:
staftestje001: created

root@pi-rataplan:/usr/local/bastille/jails/staftestje001 # 

Test that the test user and files are imported correctly.

[staftestje001]:
Last login: Sun Sep  3 18:02:03 on pts/2
FreeBSD 13.2-RELEASE-p2 GENERIC

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List:        https://www.FreeBSD.org/lists/questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

To change this login announcement, see motd(5).
root@staftestje001:~ # su - staf
Need to quickly return to your home directory? Type "cd".
		-- Dru <genesis@istar.ca>
staf@staftestje001:~ $ ls
testfile
staf@staftestje001:~ $ 

Delete the ezjail Jail

The last step is to remove the “old” ezjail.

[root@pi-rataplan ~]# ezjail-admin list
STA JID  IP              Hostname                       Root Directory
--- ---- --------------- ------------------------------ ------------------------
ZS  N/A  192.168.1.51    staftestje001                  /usr/jails/staftestje001
ZR  2    192.168.1.49    stafscm                        /usr/jails/stafscm
ZR  3    192.168.1.45    stafproxy                      /usr/jails/stafproxy
ZR  4    192.168.1.47    stafmail                       /usr/jails/stafmail
ZR  5    192.168.1.41    staffs                         /usr/jails/staffs
ZR  6    192.168.1.85    stafdns                        /usr/jails/stafdns
[root@pi-rataplan ~]# ezjail-admin delete staftestje001
[root@pi-rataplan ~]# 

ezjail delete only removes the Jail configuration. The storage is still there. Might be useful if you want to restore the Jail. And we still have a backup in /usr/local/jails/archives if for some reason we need to restore the old ezjail.

[root@pi-rataplan ~]# zfs list | grep -i testje001
zroot/bastille/jails/staftestje001             219M   153G      144K  /usr/local/bastille/jails/staftestje001
zroot/bastille/jails/staftestje001/root        219M   153G      219M  /usr/local/bastille/jails/staftestje001/root
zroot/usr/jails/staftestje001                  219M   153G      219M  /usr/jails/staftestje001
[root@pi-rataplan ~]# 

As the procedure seems to work, I’ll continue with migration with the ezjail Jails to BastilleBSD :-)

Links

September 16, 2023

Ik zag net The First Turning and the End of Woke en  hoewel ik zelf iemand ben die eerst bewijzen en cijfers wil zien, erken ik dat de sociologie met moeilijkheden zit: het is niet zo gemakkelijk als bv. in de fysica om die cijfers te bekomen. Net zoals de psychologie. Ik erken dat de psychologie het niet gemakkelijk heeft.

Maar de psychologie erkent dat te weinig. Veel psychologen willen vooral middeltjes verkopen en veel geld verdienen. Dat is een aandoening (het zou goed zijn moesten psychologen Ethica van Spinoza lezen) waar de sociologie minder mee kampt. Ik neem de psychologie al maar minder serieus: zij willen geld verdienen aan de onwetendheid van de gewone mensen (dus hun patiënten – ik link niet zomaar wat). Ik maak uitzonderingen: ik denk dat Dirk De Wachter een uitstekend psycholoog / psychiater is. Dat wil niet zeggen dat (ik denk dat) andere psychologen van even goede wil zijn. Psychologie hoort over de patiënt te gaan. Niet over geld. Het zal nog lang duren eer dat terug zo is.

In tegenstelling tot de psychologie, die deze dagen ideologisch losgeslagen blijkt te zijn met bijvoorbeeld genderideologie (want er zijn middeltjes zoals hormoonbehandelingen te verkopen), lijkt mij de socioloog in ‘The First Turning and the End of Woke’ toch een betere benadering van een werkelijkheid te hebben. Ik denk dat vooral psychologen eens achter de oren mogen krabben. Want het gaat van kwaad naar erger.

Ik ben geen expert want ik nuttig mijn expertise en vooral mijn passietijd al in het vak van programmeren. Dus neem mijn mening met een vat zout.

Ik ben blij dat ik geen kind gekregen heb de afgelopen twintig jaar: wat ik heb gezien is walgelijk. De huidige generatie jong volwassenen werd en is extreem overbeschermd; ze heeft geen idee van hoe zelf te leven en ze faalt daar dan ook in.

Met de fietshelm op rijden ze de afgrond in. Ze denkt dat die fietshelm hen zal redden. Ze had beter leren leven, zodat ze de afgrond niet zou ingefiets zijn. Maar het leven was te gevaarlijk voor hun overbeschermende ouders en vooral henzelf om hen dat aan te mogen leren. Dus: helmpje op, tablet aan, een veilig spelletje spelen en zwijgen: dat is de wijze van het oppervlakkige opvoeden en het opgevoed worden geweest de afgelopen jaren.

Die opvoendende ouders en henzelf willen en wilden ook altijd maar meer regels. Meer fietshelmen. Regels voor alles. Inclusief wetten voor voornaamwoorden. Iets wat totaal absurd was nog geen tien jaar eerder. Leren leven doen ze vooral niet. Het oppervlakkige slachtoffer zijn, dat wel. Ze willen regels. Meer regels. Voor anderen. Zodat ze zelf veilig zijn. Zodat ze slachtoffer kunnen blijven. Ze zijn geen echt slachtoffer. Maar hun  oppervlakkige ideologie laat hen het toe om dat toch te zijn.

We leven daardoor nu in de meest debiele vorm van samenleving denkbaar: Een kus van een voetbalcoach die blij was omwille van een overwinning, is nu volledig gecriminaliseerd. Ik heb hier geen enkel begrip voor. Ik haat het. Ik haat niet de kus. Ik haat de samenleving die dit criminaliseert. Ik hoop oprecht dat dit verdwijnt. Het spijt me voor mijn mening (het spijt me helemaal niet), maar die man deed niets verkeerd. Die was blij en die kuste daarom een speelster. Dat is alles.

De huidige generatie van jongvolwassenen die geïndoctrineerd werd om het opervlakkige slachtofferschap na te streven heeft nooit de kans gekregen om te leren leven.

Mede omwille van deze generatie heb ik gewacht met een kind te verwekken. Dit jaar is het zo ver en is mijn vriendin zwanger. Ik kon ook niet langer wachten want ik zou anders te oud worden wanneer die kleine het huis verlaat. Maar het zat echt in mijn hoofd: ik wilde geen kind omdat de samenleving volledig idioot geworden was.

Kan de huidige generatie jongvolwassenen opschuiven? Zodat mijn kind zeker niet in hun leefwereld hoeft te vertoeven.

Ikzelf groeide op in de jaren tachtig en dat was in veel opzichten een veel betere tijd. Voor zowel meisjes als voor jongens. Voor iedereen. Wat we nu meemaken is degressief. Ik ‘klaag’ vandaag wel eens tegen mijn ouders hoe gevaarlijk onze opvoeding wel niet was; maar eigenlijk ben ik ze daar dankbaar voor: ik mocht van hen leren te leven. Ik leerde te overleven. Zij waren er heus wanneer het fout ging. Maar het ging niet vaak fout. Daarom ben ik nu een bekwaam en gezond volwassen individu. Met een vinger die toch wel bleek gebroken te zijn want die duim staat nu een beetje scheef. Maarja. Dat is echt waar niet zo erg.

Ik leerde turnen (en werd daar zelfs goed in) en skateboarden (en werd daar zelfs goed in, en ik brak er mijn duim). Wij groeven ondergrondse tunnels (ja, echt) en wij maakte meerdere boomhutten. Wij gingen koeien omver werpen en toen kwam de boer met zijn traktor achter ons aan. Wij gingen letterlijk wekelijks belleke-trek doen. Op het einde van onze (bijna vaste) toer gingen wij altijd bij een rijkswachter: die kwam dan achter ons aangelopen. Den traagste werd dan gepakt en diene flik ging dan eens babbelen met de ouders om dat kind terug thuis te brengen. Wij vonden het spannend om niet den traagste te zijn. Het was vast ook een goede loopoefening voor de rijkswachter: hoeveel criminelen heeft die wel niet gepakt omdat wij hem wekelijks in vorm hielden?!

De wereld was voor ons gevaarlijk en uitdagend. Leuk. Venieuwend. Leerzaam. We vielen heel de tijd en we kropen altijd weer recht. Niet af en toe maar iedere paar dagen hadden wij wel blauwe plekken en soms een gebroken been of arm. Maar, is dat dan zo erg?

Beste huidige generatie jongvolwassenen: Maak plaats voor meer samenhorigheidsgevoel i.p.v. jullie individualisme. Maak plaats voor vrijheid en ja ook voor meer gevaar. Gevaar dat toch wel opgevangen wordt omdat alle volwassenen samen een beetje meekijken.

Laat ons doen.

Laat ons onszelf leren leven.

ps. Bij de echo zagen we in de buik van mijn vrouwke dat het kindje erg veel bewoog. De echoscopist zei zelfs dat dat haar job wat moeilijker maakt. Maar heel erg vonden we dat niet. Spontaan zei ik tegen de echoscopist en tegen mijn vrouwke: dat wordt een skateboarder.

Zolang het in mijn handen is, zal die kleine stapsgewijs blootgesteld worden aan het gevaar van de wereld. Ik zal er zijn om haar of hem te redden en helpen wanneer nodig.

September 10, 2023

Introduction to BastilleBSD

What are “containers”?

Chroot, Jails, containers, zones, LXC, Docker

I use FreeBSD on my home network to serve services like email, git, fileserver, etc. For some other services, I use k3s with GNU/Linux application containers.

The FreeBSD services run as Jails. For those who aren’t familiar with FreeBSD Jails. Jails started the whole concept of “containers”.

FreeBSD Jails inspired Sun Microsystems to create Solaris zones.

If you want to know more about the history of FreeBSD Jails, Solaris zones and containers on Un!x systems in general and the challenges to run containers securely I recommend the video;

“Papers We Love: Jails and Solaris Zones by Bryan Cantrill”

Papers We Love: Jails and Solaris Zones by Bryan Cantrill

Sun took containers to the next level with Solaris zones , allowing a fine-grade CPU and memory allocation.

On GNU/Linux LXC was the most popular container framework. …Till Docker came along.

Application vs system containers

To the credit of Docker, Docker made the concept of application containers popular.

System containers run the complete operating system and can be used like virtual machines without the overhead.

Application containers run a single application binary inside a container that holds all the dependencies for this application.

FreeBSD Jails

FreeBSD Jails can be used for both “application” containers and “system” containers. FreeBSD Jails is a framework that separates the host and the Jails with security in mind.

My home network setup

To make the management of Jails easier we have management tools. I started to use ezjail in 2013 after my OpenSolaris system died and Oracle killed OpenSolaris.

ezjail isn’t developed that actively.

BastilleBSD is the most the popular Jail management tool for FreeBSD Jails. It supports both application and system containers.

I migrated all the services to Raspberry Pi’s to save electricity. But continued to use ezjail, just to stick to something that I knew and was stable. Migrating to BastileBSD was still on my to-do list and I finally found the time to do it.

My blog posts are mostly my installation notes that I publish in the hope that they are useful to somebody else.

In this blog post, you’ll find my journey to explore BastileBSD, in a next blog post, I’ll go through the migration from ezjail to BastilleBSD.

If you are interested in the concepts behind BastilleBSD, I recommend the video:

FreeBSD Fridays: Introduction to BastilleBSD

FreeBSD Fridays: Introduction to BastilleBSD

BastilleBSD exploration

Start

We’ll execute all actions on a virtual machine running FreeBSD on a Raspberry PI. The Raspberry PI is running Debian GNU/Linux with the KVM/libvirt hypervisor.

The virtual machine is running FreeBSD 13.2.

root@pi-rataplan:~ # freebsd-version 
13.2-RELEASE-p2
root@pi-rataplan:~ # 
root@pi-rataplan:~ # uname -a
FreeBSD pi-rataplan 13.2-RELEASE-p2 FreeBSD 13.2-RELEASE-p2 GENERIC arm64
root@pi-rataplan:~ # 

Install BastilleBSD

Install

The first step is to install bastille.

root@pi-rataplan:~ # pkg install -y bastille
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	bastille: 0.9.20220714

Number of packages to be installed: 1

38 KiB to be downloaded.
[1/1] Fetching bastille-0.9.20220714.pkg: 100%   38 KiB  38.8kB/s    00:01    
Checking integrity... done (0 conflicting)
[1/1] Installing bastille-0.9.20220714...
[1/1] Extracting bastille-0.9.20220714: 100%
root@pi-rataplan:~ # 

Enable BastilleBSD

BastilleBSD is not a daemon that runs in the background, but starts the containers when the bastille_enable system configuration rc.conf variable is set to YES.

FreeBSD has a nice tool to set the system configuration variables; sysrc.

root@pi-rataplan:~ # sysrc bastille_enable=YES
bastille_enable:  -> YES
root@pi-rataplan:~ # 

It’s possible to specify a list of containers in the bastille_list, the containers are started in the order of the list.

Configuration

Update config to use ZFS

BastilleBSD can use OpenZFS, but this isn’t enabled by default. When ZFS support is enabled a new ZFS dataset is created when a container is created.

root@pi-rataplan:/usr/local/etc/bastille # cd /usr/local/etc/bastille/
root@pi-rataplan:/usr/local/etc/bastille # 
root@pi-rataplan:/usr/local/etc/bastille # vi bastille.conf

To enable OpenZFS support you need to set bastille_zfs_enable to YES and specify the zpool to be used. Please note that YES need to be in upper case.

## ZFS options
bastille_zfs_enable="YES"                                                ## default: ""
bastille_zfs_zpool="zroot"                                                 ## default: ""
bastille_zfs_prefix="bastille"                                        ## default: "${bastille_zfs_zpool}/bast
ille"
bastille_zfs_options="-o compress=lz4 -o atime=off"                   ## default: "-o compress=lz4 -o atime=o
ff"

Bootstrap

Bootstrap a release

In order to start a container you need to bootstrap a FreeBSD release first.

root@pi-rataplan:/usr/local/share/bastille # freebsd-version 
13.2-RELEASE-p2

A FreeBSD release is without the patch level.

The command below will bootstrap the 13.2-RELEASE on bastilleBSD. The update option will also include the latest patches.

root@pi-rataplan:/usr/local/share/bastille # bastille bootstrap 13.2-RELEASE update
<snip>
Installing updates...
Restarting sshd after upgrade
Performing sanity check on sshd configuration.
Stopping sshd.
Waiting for PIDS: 93019.
Performing sanity check on sshd configuration.
Starting sshd.
Scanning /usr/local/bastille/releases/13.2-RELEASE/usr/share/certs/blacklisted for certificates...
Scanning /usr/local/bastille/releases/13.2-RELEASE/usr/share/certs/trusted for certificates...
 done.
root@pi-rataplan:/usr/local/share/bastille # 

List

You can use the bastille list release command to list the bootstrapped releases.

root@pi-rataplan:/usr/local/bastille/jails/bastille-tst001 # bastille list release
13.1-RELEASE
13.2-RELEASE
root@pi-rataplan:/usr/local/bastille/jails/bastille-tst001 # 

The releases are stored in the bastilleBSD dataset /usr/local/bastille/releases in our case.

root@pi-rataplan:/usr/local/bastille # cd releases/
root@pi-rataplan:/usr/local/bastille/releases # ls
13.1-RELEASE	13.2-RELEASE
root@pi-rataplan:/usr/local/bastille/releases # 

The downloaded tar archive are stored in the cache ZFS dataset.

root@pi-rataplan:/usr/local/bastille/releases/13.1-RELEASE # zfs list | grep -i bastille | grep -i cache
zroot/bastille/cache                           334M  93.6G      104K  /usr/local/bastille/cache
zroot/bastille/cache/13.1-RELEASE              165M  93.6G      165M  /usr/local/bastille/cache/13.1-RELEASE
zroot/bastille/cache/13.2-RELEASE              169M  93.6G      169M  /usr/local/bastille/cache/13.2-RELEASE
root@pi-rataplan:/usr/local/bastille/releases/13.1-RELEASE # 
root@pi-rataplan:/usr/local/bastille/releases/13.1-RELEASE # ls -l /usr/local/bastille/cache/13.2-RELEASE
total 345866
-rw-r--r--  1 root  wheel        782 Apr  7 07:01 MANIFEST
-rw-r--r--  1 root  wheel  176939748 Apr  7 07:01 base.txz
root@pi-rataplan:/usr/local/bastille/releases/13.1-RELEASE # 

Verify a release

With bastille verify you can verify a release to be sure that no files are altered.

root@pi-rataplan:/usr/local/bastille/releases #  bastille verify 13.2-RELEASE
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.2-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
root@pi-rataplan:/usr/local/bastille/releases # 

Create your first container

“Thin” vs “thick” Jails.

“Thin” Jails are created by default. With a “thin” Jail the operating system is “shared” with the release. This saves a lot of disk space. It’s possible to convert a container to a “thick” Jail after a Jail is created.

Network

BastilleBSD has a lot of network options, it can also create dynamic firewall rules to expose services.

See https://docs.bastillebsd.org/en/latest/chapters/networking.html for more information.

In this example we’ll use a “shared” network interface vtnet0 with the host system.

root@pi-rataplan:/usr/local/bastille/releases # bastille create bastille-tst001 13.2-RELEASE <ip_address> vtnet0
Valid: (<ip_address>).
Valid: (vtnet0).

Creating a thinjail...

[bastille-tst001]:
bastille-tst001: created

[bastille-tst001]:
Applying template: default/thin...
[bastille-tst001]:
Applying template: default/base...
[bastille-tst001]:
[bastille-tst001]: 0

[bastille-tst001]:
syslogd_flags: -s -> -ss

[bastille-tst001]:
sendmail_enable: NO -> NO

[bastille-tst001]:
sendmail_submit_enable: YES -> NO

[bastille-tst001]:
sendmail_outbound_enable: YES -> NO

[bastille-tst001]:
sendmail_msp_queue_enable: YES -> NO

[bastille-tst001]:
cron_flags:  -> -J 60

[bastille-tst001]:
/etc/resolv.conf -> /usr/local/bastille/jails/bastille-tst001/root/etc/resolv.conf

Template applied: default/base

Template applied: default/thin

rdr-anchor not found in pf.conf
[bastille-tst001]:
bastille-tst001: removed

[bastille-tst001]:
bastille-tst001: created

root@pi-rataplan:/usr/local/bastille/releases # 

A new ZFS dataset is created for the Jail.

root@pi-rataplan:/usr/local/bastille/releases/13.1-RELEASE # zfs list | grep -i tst001
zroot/bastille/jails/bastille-tst001          5.61M  93.6G      116K  /usr/local/bastille/jails/bastille-tst001
zroot/bastille/jails/bastille-tst001/root     5.50M  93.6G     5.50M  /usr/local/bastille/jails/bastille-tst001/root
root@pi-rataplan:/usr/local/bastille/releases/13.1-RELEASE # 

The mounted Jail dataset holds the Jail configuration file and fstab that is used by the Jail.

root@pi-rataplan:/usr/local/bastille/jails/bastille-tst001 # cd /usr/local/bastille/jails/bastille-tst001
root@pi-rataplan:/usr/local/bastille/jails/bastille-tst001 # ls
fstab		jail.conf	root
root@pi-rataplan:/usr/local/bastille/jails/bastille-tst001 # 

jail.conf:

root@pi-rataplan:/usr/local/bastille/jails/bastille-tst001 # cat jail.conf 
bastille-tst001 {
  devfs_ruleset = 4;
  enforce_statfs = 2;
  exec.clean;
  exec.consolelog = /var/log/bastille/bastille-tst001_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = bastille-tst001;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/bastille-tst001/fstab;
  path = /usr/local/bastille/jails/bastille-tst001/root;
  securelevel = 2;

  interface = vtnet0;
  ip4.addr = <ip_address>;
  ip6 = disable;
}
root@pi-rataplan:/usr/local/bastille/jails/bastille-tst001

fstab:

root@pi-rataplan:/usr/local/bastille/jails/bastille-tst001 # cat /usr/local/bastille/jails/bastille-tst001/fstab
/usr/local/bastille/releases/13.2-RELEASE /usr/local/bastille/jails/bastille-tst001/root/.bastille nullfs ro 0 0
root@pi-rataplan:/usr/local/bastille/jails/bastille-tst001 # 

Interact with containers

list

With bastille list we can list the containers. When no option is given it’ll list the running containers only. To list all containers (stopped and running) you can use the -a option.

Note that bastille list is a wrapper around the jls command and also lists the running ezjail containers.

root@pi-rataplan:/usr/jails/stafproxy/basejail # bastille list
 JID             IP Address      Hostname                      Path
 stafscm         <ip>            stafscm                       /usr/jails/stafscm
 stafproxy       <ip>            stafproxy                     /usr/jails/stafproxy
 stafmail        <ip>            stafmail                      /usr/jails/stafmail
 staffs          <ip>            staffs                        /usr/jails/staffs
 stafdns         <ip>            stafdns                       /usr/jails/stafdns
 bastille-tst001 <ip>            bastille-tst001               /usr/local/bastille/jails/bastille-tst001/root
root@pi-rataplan:/usr/jails/stafproxy/basejail # 

console access

To gain console access you use the bastille console command.

root@pi-rataplan:/usr/jails/stafproxy/basejail # bastille console bastille-tst001
[bastille-tst001]:
root@bastille-tst001:~ # 

Verify the disk space. We’re using a “thin” Jail only, 5.5M of disk space is used.

root@bastille-tst001:~ # df -h .
Filesystem                                   Size    Used   Avail Capacity  Mounted on
zroot/bastille/jails/bastille-tst001/root    154G    5.5M    154G     0%    /
root@bastille-tst001:~ # 

Readonly file system

We’re using “thin” Jails. The system binaries are read-only.

[root@bastille-tst001 ~]# ls -l /bin
lrwxr-xr-x  1 root  wheel  14 Aug 16 10:25 /bin -> /.bastille/bin
[root@bastille-tst001 ~]# 
[root@bastille-tst001 ~]# touch /bin/ls
touch: /bin/ls: Read-only file system
[root@bastille-tst001 ~]# 

freebsd-version

[root@bastille-tst001 ~]# freebsd-version 
13.2-RELEASE-p2
[root@bastille-tst001 ~]# uname -a
FreeBSD bastille-tst001 13.2-RELEASE-p2 FreeBSD 13.2-RELEASE-p2 GENERIC arm64
[root@bastille-tst001 ~]# 

Install FreeBSD packages

Bootstrap the pkg command to install packages.

root@bastille-tst001:~ # pkg
The package management tool is not yet installed on your system.
Do you want to fetch and install it now? [y/N]: y
Bootstrapping pkg from pkg+http://pkg.FreeBSD.org/FreeBSD:13:aarch64/quarterly, please wait...
Verifying signature with trusted certificate pkg.freebsd.org.2013102301... done
[bastille-tst001] Installing pkg-1.19.2...
[bastille-tst001] Extracting pkg-1.19.2: 100%
pkg: not enough arguments
Usage: pkg [-v] [-d] [-l] [-N] [-j <jail name or id>|-c <chroot path>|-r <rootdir>] [-C <configuration file>] [-R <repo config dir>] [-o var=value] [-4|-6] <command> [<args>]

For more information on available commands and options see 'pkg help'.
root@bastille-tst001:~ #

I use ansible to manage my homelab python3 and sudo are required for this, so these are usually the first packages I install.

root@bastille-tst001:~ # pkg install -y sudo python3 bash
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 9 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	bash: 5.2.15
	gettext-runtime: 0.21.1
	indexinfo: 0.3.1
	libffi: 3.4.4
	mpdecimal: 2.5.1
	python3: 3_3
	python39: 3.9.17
	readline: 8.2.1
	sudo: 1.9.14p3

Number of packages to be installed: 9

The process will require 140 MiB more space.
21 MiB to be downloaded.
[bastille-tst001] [1/9] Fetching indexinfo-0.3.1.pkg: 100%    5 KiB   5.5kB/s    00:01    
[bastille-tst001] [2/9] Fetching mpdecimal-2.5.1.pkg: 100%  292 KiB 299.4kB/s    00:01    
[bastille-tst001] [3/9] Fetching python39-3.9.17.pkg: 100%   17 MiB   2.6MB/s    00:07    
[bastille-tst001] [4/9] Fetching libffi-3.4.4.pkg: 100%   36 KiB  36.6kB/s    00:01    
[bastille-tst001] [5/9] Fetching readline-8.2.1.pkg: 100%  345 KiB 353.1kB/s    00:01    
[bastille-tst001] [6/9] Fetching sudo-1.9.14p3.pkg: 100%    2 MiB   1.6MB/s    00:01    
[bastille-tst001] [7/9] Fetching python3-3_3.pkg: 100%    1 KiB   1.1kB/s    00:01    
[bastille-tst001] [8/9] Fetching bash-5.2.15.pkg: 100%    2 MiB   1.6MB/s    00:01    
[bastille-tst001] [9/9] Fetching gettext-runtime-0.21.1.pkg: 100%  160 KiB 164.0kB/s    00:01    
Checking integrity... done (0 conflicting)
[bastille-tst001] [1/9] Installing indexinfo-0.3.1...
[bastille-tst001] [1/9] Extracting indexinfo-0.3.1: 100%
[bastille-tst001] [2/9] Installing mpdecimal-2.5.1...
[bastille-tst001] [2/9] Extracting mpdecimal-2.5.1: 100%
[bastille-tst001] [3/9] Installing libffi-3.4.4...
[bastille-tst001] [3/9] Extracting libffi-3.4.4: 100%
[bastille-tst001] [4/9] Installing readline-8.2.1...
[bastille-tst001] [4/9] Extracting readline-8.2.1: 100%
[bastille-tst001] [5/9] Installing gettext-runtime-0.21.1...
[bastille-tst001] [5/9] Extracting gettext-runtime-0.21.1: 100%
[bastille-tst001] [6/9] Installing python39-3.9.17...
[bastille-tst001] [6/9] Extracting python39-3.9.17: 100%
[bastille-tst001] [7/9] Installing sudo-1.9.14p3...
[bastille-tst001] [7/9] Extracting sudo-1.9.14p3: 100%
[bastille-tst001] [8/9] Installing python3-3_3...
[bastille-tst001] [8/9] Extracting python3-3_3: 100%
[bastille-tst001] [9/9] Installing bash-5.2.15...
[bastille-tst001] [9/9] Extracting bash-5.2.15: 100%
=====
Message from python39-3.9.17:

--
Note that some standard Python modules are provided as separate ports
as they require additional dependencies. They are available as:

py39-gdbm       databases/py-gdbm@py39
py39-sqlite3    databases/py-sqlite3@py39
py39-tkinter    x11-toolkits/

Let’s install neofetch.

[root@bastille-tst001 ~]# pkg install -y neofetch
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
The following 1 package(s) will be affected (of 0 checked):

New packages to be INSTALLED:
	neofetch: 7.1.0_1

Number of packages to be installed: 1

79 KiB to be downloaded.
[bastille-tst001] [1/1] Fetching neofetch-7.1.0_1.pkg: 100%   79 KiB  81.0kB/s    00:01    
Checking integrity... done (0 conflicting)
[bastille-tst001] [1/1] Installing neofetch-7.1.0_1...
[bastille-tst001] [1/1] Extracting neofetch-7.1.0_1: 100%
[root@bastille-tst001 ~]# 
[root@bastille-tst001 ~]# neofetch 
  ` `.....---.......--.```   -/    -------------------- 
  +o   .--`         /y:`      +.   OS: FreeBSD 13.2-RELEASE-p2 aarch64 
   yo`:.            :o      `+-    Uptime: 3 days, 14 hours, 13 mins 
    y/               -/`   -o/     Packages: 11 (pkg) 
   .-                  ::/sy+:.    Shell: csh tcsh 6.22.04 
   /                     `--  /    Terminal: /dev/pts/1 
  `:                          :`   CPU: ARM Cortex-A72 r0p3 (4) 
  `:                          :`   Memory: 2193MiB / 3039MiB 
   /                          /
   .-                        -.                            
    --                      -.                             
     `:`                  `:`
       .--             `--.
          .---.....----.

[root@bastille-tst001 ~]# 

Logout

[root@bastille-tst001 ~]# 
exit
root@bastille-tst001:~ # logout

update containers

List the releases.

root@pi-rataplan:~ # bastille list release
13.1-RELEASE
13.2-RELEASE
root@pi-rataplan:~ # 

Update a release to the latest patch level.

root@pi-rataplan:/usr/jails/stafproxy/basejail # bastille update 13.2-RELEASE
src component not installed, skipped
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
Fetching metadata signature for 13.2-RELEASE from update2.freebsd.org... done.
Fetching metadata index... done.
Inspecting system... done.
Preparing to download files... done.

No updates needed to update system to 13.2-RELEASE-p2.
No updates are available to install.

Execute commands inside a container.

You can use bastille cmd to execute a command inside a container which is a wrapper around the jexec.

List running processes.

The syntax is

bastille cmd <container|ALL> <command string>

root@pi-rataplan:~ # bastille cmd ALL ps aux
[bastille-tst001]:
grep: /usr/local/bastille/jails/ALL/fstab: No such file or directory
USER   PID %CPU %MEM   VSZ  RSS TT  STAT STARTED    TIME COMMAND
root 35869  0.0  0.1 12876 2496  -  IJ   10:55   0:00.00 cron: running job (cron)
root 46730  0.0  0.1 12704 2696  -  SsJ  10:25   0:00.01 /usr/sbin/syslogd -ss
root 55608  0.0  0.1 12876 2496  -  IJ   10:55   0:00.00 cron: running job (cron)
root 69222  0.0  0.1 12876 2512  -  IsJ  10:25   0:00.01 /usr/sbin/cron -J 60 -s
root 67674  0.0  0.1 13440 2960  1  R+J  10:55   0:00.00 ps aux
[bastille-tst001]: 0

root@pi-rataplan:~ # 

audit packages

FreeBSD has implemented the vulnerability management in the correct way. The vulnerability database is outside of the packages management database as it should be.

Unfortunately the FreeBSD vulnerability database isn’t compatible with the OVAL standard.

This makes auditing the installed FreeBSD packages with the pkg audit command in a way comparable with a security audit with OpenSCAP on a GNU/Linux system.

But this might be a topic for another blog post :-)

root@pi-rataplan:~ # bastille pkg ALL audit -F
[bastille-tst001]:
[bastille-tst001] Fetching vuln.xml.xz: 100%    1 MiB 349.2kB/s    00:03    
0 problem(s) in 0 installed package(s) found.

root@pi-rataplan:~ # 

Have fun!

Links

September 01, 2023

FOSDEM 2024 will take place on Saturday 3rd and Sunday 4th of February 2024. Further details and calls for participation will be announced in the coming days and weeks.

WordPress has made some good progress to speed up site rendering the last couple of years and part of this is thanks to images and iframes having the “loading” attribute with value “lazy”, telling browsers these can be loaded later allowing other (more important) assets to take priority. Especially iframes can consume a lot of bandwidth (think embedded Google Maps or one or more YouTube videos)...

Source

August 31, 2023

Waterloo:

The combined number of men killed or wounded reached nearly 50,000, with close to 25,000 casualties on the French side and approximately 23,000 for the Allied army.

Bakhmut:

Western estimate: 60,000+ casualties (20,000+ killed) Per Ukraine: 100,000+ killed or wounded (20,000–30,000 killed)

August 26, 2023

Today, I published the following diary on isc.sans.edu: “macOS: Who’s Behind This Network Connection?“:

When you must investigate suspicious behavior or work on an actual incident, you could be asked to determine who’s behind a network connection. From a pure network point of view, your firewall or any network security control device/app will tell you that the source is the connection is host « A », « B » or « C ». But investigating further how to discover who or which process is the source of the connection (now, at the operating system level)… [Read more]

The post [SANS ISC] macOS: Who’s Behind This Network Connection? appeared first on /dev/random.

August 25, 2023

Today, I published the following diary on isc.sans.edu: “Python Malware Using Postgresql for C2 Communications“:

For modern malware, having access to its C2 (Command and control) is a crucial point. There are many ways to connect to a C2 server using tons of protocols, but today, HTTP remains very common because HTTP is allowed on most networks… I found a malicious Python script that is pretty well obfuscated. The applied technique reduces its VT  score to 6/60! It’s based on a mix of Based64- and Hex-encoded data… [Read more]

The post [SANS ISC] Python Malware Using Postgresql for C2 Communications appeared first on /dev/random.

August 23, 2023

Today, I published the following diary on isc.sans.edu: “More Exotic Excel Files Dropping AgentTesla”:

Excel is an excellent target for attackers. The Microsoft Office suite is installed on millions of computers, and people trust these files. If we have the classic xls, xls, xlsm file extensions, Excel supports many others! Just check your local registry… [Read morehttps://isc.sans.edu/diary/More%20Exotic%20Excel%20Files%20Dropping%20AgentTesla/30150]

The post [SANS ISC] More Exotic Excel Files Dropping AgentTesla appeared first on /dev/random.

August 22, 2023

Many of us, old MySQL DBAs used Seconds_Behind_Source from SHOW REPLICA STATUS to find out the status and correct execution of (asynchronous) replication.

Please pay attention of the new terminology. I’m sure we’ve all used the old terminology.

However, MySQL replication has evolved a lot and the replication team has worked to include a lot of useful information about all the replication flavors available with MySQL.

For example, we’ve added parallel replication, group replication, … all that information is missing from the the good old SHOW REPLICA STATUS result.

There much better ways to monitoring and observing the replication process(es) using Performance_Schema.

Currently in Performance_Schema, there are 15 tables relating to replication instrumentation:

+------------------------------------------------------+
| Tables_in_performance_schema (replication%)          |
+------------------------------------------------------+
| replication_applier_configuration                    |
| replication_applier_filters                          |
| replication_applier_global_filters                   |
| replication_applier_status                           |
| replication_applier_status_by_coordinator            |
| replication_applier_status_by_worker                 |
| replication_asynchronous_connection_failover         |
| replication_asynchronous_connection_failover_managed |
| replication_connection_configuration                 |
| replication_connection_status                        |
| replication_group_communication_information          |
| replication_group_configuration_version              |
| replication_group_member_actions                     |
| replication_group_member_stats                       |
| replication_group_members                            |
+------------------------------------------------------+
15 rows in set (0.0038 sec)

But it’s true that it’s not always easy to understand these metrics mean and where to look for the useful information that matters for us MySQL DBAs: is my replication lagging behind the source ?

I’ve created a few views that can be installed in sys to use most of these metrics to get something relevant for us DBAs: mysql_8_replication_observability.sql

Let’s take a closer look at these views.

Replication Lag

select * from sys.replication_lag;
+---------------------------+-----------------------+------------------------+
| channel_name              | max_lag_from_original | max_lag_from_immediate |
+---------------------------+-----------------------+------------------------+
| clusterset_replication    | 00:00:04.963223       | 00:00:04.940782        |
| group_replication_applier | 0                     | 0                      |
+---------------------------+-----------------------+------------------------+

From the output above, we can see that the MySQL instance is an asynchronous replica, but it’s also part of a group replication cluster.

In fact, this is the Primary Member of the DR cluster in an InnoDB ClusterSet.

We can also see that this replica is almost 5 seconds late (lag).

We then have the name of the replication channel and the maximum delay/lag (as there may be several workers in the case of parallel replication) with the original committer and the immediate source (in case of cascading replication).

On a secondary member of a group replication cluster (InnoDB Cluster), we can see the following output:

select * from sys.replication_lag;
+----------------------------+-----------------------+------------------------+
| channel_name               | max_lag_from_original | max_lag_from_immediate |
+----------------------------+-----------------------+------------------------+
| group_replication_recovery | null                  | null                   |
| group_replication_applier  | 00:00:02.733008       | 00:00:02.733008        |
+----------------------------+-----------------------+------------------------+

We can see that the channel used for recovery (reading the missing binary log events, transactions, when a node joins the group) is not being used and that the group replication’s applier is lagging a little behind.

Replication status

This view is more complete, with a line for each workers.

Let’s take the example of our Primary member of the DR site of an InnoDB ClusterSet:

select * from replication_status;
+-------------------------------+----------+----------+---------+-------------------+--------------------+
| channel                       | io_state | co_state | w_state | lag_from_original | lag_from_immediate |
+-------------------------------+----------+----------+---------+-------------------+--------------------+
| group_replication_applier (1) | ON       | ON       | ON      | none              | none               |
| group_replication_applier (2) | ON       | ON       | ON      | none              | none               |
| group_replication_applier (3) | ON       | ON       | ON      | none              | none               |
| group_replication_applier (4) | ON       | ON       | ON      | none              | none               |
| clusterset_replication (1)    | ON       | ON       | ON      | 00:00:15.395870   | 00:00:15.380884    |
| clusterset_replication (2)    | ON       | ON       | ON      | 00:00:15.395686   | 00:00:15.380874    |
| clusterset_replication (3)    | ON       | ON       | ON      | 00:00:15.411204   | 00:00:15.388451    |
| clusterset_replication (4)    | ON       | ON       | ON      | 00:00:15.406154   | 00:00:15.388434    |
+-------------------------------+----------+----------+---------+-------------------+--------------------+

We can see that parallel (asynchronous) replication from the Primary cluster uses 4 parallel workers.

We can also see that they are falling behind…

You may have noticed that there are 3 states (all at ON). Using SHOW REPLICA STATUS we can only see:

       Replica_IO_Running: Yes
      Replica_SQL_Running: Yes

With parallel replication, we have another thread involved in replication during application of binlog events: the coordinator thread.

Replication Status Full

Of course, we can also have more details on replication.

Let’s take a look at an example result:

select * from sys.replication_status_full\G
*************************** 1. row ***************************
                 channel: group_replication_applier (1)
                    host: <NULL>
                    port: 0
                    user: 
             source_uuid: 7b6bf13d-40ed-11ee-bfdd-c8cb9e32df8e
              group_name: 7b6bf13d-40ed-11ee-bfdd-c8cb9e32df8e
last_heartbeat_timestamp: 0000-00-00 00:00:00.000000
      heartbeat_interval: 30
                io_state: ON
         io_thread_state: NULL
                io_errno: 0
               io_errmsg: 
              io_errtime: 0000-00-00 00:00:00.000000
                co_state: ON
         co_thread_state: Replica has read all relay log; waiting for more updates
                co_errno: 0
               co_errmsg: 
              co_errtime: 0000-00-00 00:00:00.000000
                 w_state: ON
          w_thread_state: Waiting for an event from Coordinator
                 w_errno: 0
                w_errmsg: 
               w_errtime: 0000-00-00 00:00:00.000000
 time_since_last_message: 03:36:40.474223
      applier_busy_state: IDLE
       lag_from_original: none
      lag_from_immediate: none
          transport_time: 1.80 us
       time_to_relay_log: 12.00 us
              apply_time: 784.00 us
last_applied_transaction: 7b6bf4f0-40ed-11ee-bfdd-c8cb9e32df8e:3
 last_queued_transaction: 7b6bf4f0-40ed-11ee-bfdd-c8cb9e32df8e:3
queued_gtid_set_to_apply: 
*************************** 2. row ***************************
                 channel: group_replication_applier (2)
                    host: <NULL>
                    port: 0
                    user: 
             source_uuid: 7b6bf13d-40ed-11ee-bfdd-c8cb9e32df8e
              group_name: 7b6bf13d-40ed-11ee-bfdd-c8cb9e32df8e
last_heartbeat_timestamp: 0000-00-00 00:00:00.000000
      heartbeat_interval: 30
                io_state: ON
         io_thread_state: NULL
                io_errno: 0
               io_errmsg: 
              io_errtime: 0000-00-00 00:00:00.000000
                co_state: ON
         co_thread_state: Replica has read all relay log; waiting for more updates
                co_errno: 0
               co_errmsg: 
              co_errtime: 0000-00-00 00:00:00.000000
                 w_state: ON
          w_thread_state: Waiting for an event from Coordinator
                 w_errno: 0
                w_errmsg: 
               w_errtime: 0000-00-00 00:00:00.000000
 time_since_last_message: 03:36:40.474223
      applier_busy_state: IDLE
       lag_from_original: none
      lag_from_immediate: none
          transport_time: 1.80 us
       time_to_relay_log: 12.00 us
              apply_time:   0 ps
last_applied_transaction: 
 last_queued_transaction: 7b6bf4f0-40ed-11ee-bfdd-c8cb9e32df8e:3
queued_gtid_set_to_apply: 
*************************** 3. row ***************************
                 channel: group_replication_applier (3)
                    host: <NULL>
                    port: 0
                    user: 
             source_uuid: 7b6bf13d-40ed-11ee-bfdd-c8cb9e32df8e
              group_name: 7b6bf13d-40ed-11ee-bfdd-c8cb9e32df8e
last_heartbeat_timestamp: 0000-00-00 00:00:00.000000
      heartbeat_interval: 30
                io_state: ON
         io_thread_state: NULL
                io_errno: 0
               io_errmsg: 
              io_errtime: 0000-00-00 00:00:00.000000
                co_state: ON
         co_thread_state: Replica has read all relay log; waiting for more updates
                co_errno: 0
               co_errmsg: 
              co_errtime: 0000-00-00 00:00:00.000000
                 w_state: ON
          w_thread_state: Waiting for an event from Coordinator
                 w_errno: 0
                w_errmsg: 
               w_errtime: 0000-00-00 00:00:00.000000
 time_since_last_message: 03:36:40.474223
      applier_busy_state: IDLE
       lag_from_original: none
      lag_from_immediate: none
          transport_time: 1.80 us
       time_to_relay_log: 12.00 us
              apply_time:   0 ps
last_applied_transaction: 
 last_queued_transaction: 7b6bf4f0-40ed-11ee-bfdd-c8cb9e32df8e:3
queued_gtid_set_to_apply: 
*************************** 4. row ***************************
                 channel: group_replication_applier (4)
                    host: <NULL>
                    port: 0
                    user: 
             source_uuid: 7b6bf13d-40ed-11ee-bfdd-c8cb9e32df8e
              group_name: 7b6bf13d-40ed-11ee-bfdd-c8cb9e32df8e
last_heartbeat_timestamp: 0000-00-00 00:00:00.000000
      heartbeat_interval: 30
                io_state: ON
         io_thread_state: NULL
                io_errno: 0
               io_errmsg: 
              io_errtime: 0000-00-00 00:00:00.000000
                co_state: ON
         co_thread_state: Replica has read all relay log; waiting for more updates
                co_errno: 0
               co_errmsg: 
              co_errtime: 0000-00-00 00:00:00.000000
                 w_state: ON
          w_thread_state: Waiting for an event from Coordinator
                 w_errno: 0
                w_errmsg: 
               w_errtime: 0000-00-00 00:00:00.000000
 time_since_last_message: 03:36:40.474223
      applier_busy_state: IDLE
       lag_from_original: none
      lag_from_immediate: none
          transport_time: 1.80 us
       time_to_relay_log: 12.00 us
              apply_time:   0 ps
last_applied_transaction: 
 last_queued_transaction: 7b6bf4f0-40ed-11ee-bfdd-c8cb9e32df8e:3
queued_gtid_set_to_apply: 
*************************** 5. row ***************************
                 channel: clusterset_replication (1)
                    host: 127.0.0.1
                    port: 3310
                    user: mysql_innodb_cs_b0adbc6c
             source_uuid: 2cb77a02-40eb-11ee-83f4-c8cb9e32df8e
              group_name: 
last_heartbeat_timestamp: 2023-08-22 18:48:41.037817
      heartbeat_interval: 30
                io_state: ON
         io_thread_state: Waiting for source to send event
                io_errno: 0
               io_errmsg: 
              io_errtime: 0000-00-00 00:00:00.000000
                co_state: ON
         co_thread_state: Waiting for replica workers to process their queues
                co_errno: 0
               co_errmsg: 
              co_errtime: 0000-00-00 00:00:00.000000
                 w_state: ON
          w_thread_state: waiting for handler commit
                 w_errno: 0
                w_errmsg: 
               w_errtime: 0000-00-00 00:00:00.000000
 time_since_last_message: 00:00:00.001134
      applier_busy_state: APPLYING
       lag_from_original: 00:00:01.799071
      lag_from_immediate: 00:00:01.783404
          transport_time: 2.26 ms
       time_to_relay_log: 19.00 us
              apply_time: 14.63 ms
last_applied_transaction: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105180
 last_queued_transaction: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105547
queued_gtid_set_to_apply: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105184-105547
*************************** 6. row ***************************
                 channel: clusterset_replication (2)
                    host: 127.0.0.1
                    port: 3310
                    user: mysql_innodb_cs_b0adbc6c
             source_uuid: 2cb77a02-40eb-11ee-83f4-c8cb9e32df8e
              group_name: 
last_heartbeat_timestamp: 2023-08-22 18:48:41.037817
      heartbeat_interval: 30
                io_state: ON
         io_thread_state: Waiting for source to send event
                io_errno: 0
               io_errmsg: 
              io_errtime: 0000-00-00 00:00:00.000000
                co_state: ON
         co_thread_state: Waiting for replica workers to process their queues
                co_errno: 0
               co_errmsg: 
              co_errtime: 0000-00-00 00:00:00.000000
                 w_state: ON
          w_thread_state: waiting for handler commit
                 w_errno: 0
                w_errmsg: 
               w_errtime: 0000-00-00 00:00:00.000000
 time_since_last_message: 00:00:00.001134
      applier_busy_state: APPLYING
       lag_from_original: 00:00:01.797743
      lag_from_immediate: 00:00:01.783390
          transport_time: 2.26 ms
       time_to_relay_log: 19.00 us
              apply_time: 21.47 ms
last_applied_transaction: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105181
 last_queued_transaction: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105547
queued_gtid_set_to_apply: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105184-105547
*************************** 7. row ***************************
                 channel: clusterset_replication (3)
                    host: 127.0.0.1
                    port: 3310
                    user: mysql_innodb_cs_b0adbc6c
             source_uuid: 2cb77a02-40eb-11ee-83f4-c8cb9e32df8e
              group_name: 
last_heartbeat_timestamp: 2023-08-22 18:48:41.037817
      heartbeat_interval: 30
                io_state: ON
         io_thread_state: Waiting for source to send event
                io_errno: 0
               io_errmsg: 
              io_errtime: 0000-00-00 00:00:00.000000
                co_state: ON
         co_thread_state: Waiting for replica workers to process their queues
                co_errno: 0
               co_errmsg: 
              co_errtime: 0000-00-00 00:00:00.000000
                 w_state: ON
          w_thread_state: waiting for handler commit
                 w_errno: 0
                w_errmsg: 
               w_errtime: 0000-00-00 00:00:00.000000
 time_since_last_message: 00:00:00.001134
      applier_busy_state: APPLYING
       lag_from_original: 00:00:01.786087
      lag_from_immediate: 00:00:01.767563
          transport_time: 2.26 ms
       time_to_relay_log: 19.00 us
              apply_time: 21.58 ms
last_applied_transaction: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105182
 last_queued_transaction: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105547
queued_gtid_set_to_apply: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105184-105547
*************************** 8. row ***************************
                 channel: clusterset_replication (4)
                    host: 127.0.0.1
                    port: 3310
                    user: mysql_innodb_cs_b0adbc6c
             source_uuid: 2cb77a02-40eb-11ee-83f4-c8cb9e32df8e
              group_name: 
last_heartbeat_timestamp: 2023-08-22 18:48:41.037817
      heartbeat_interval: 30
                io_state: ON
         io_thread_state: Waiting for source to send event
                io_errno: 0
               io_errmsg: 
              io_errtime: 0000-00-00 00:00:00.000000
                co_state: ON
         co_thread_state: Waiting for replica workers to process their queues
                co_errno: 0
               co_errmsg: 
              co_errtime: 0000-00-00 00:00:00.000000
                 w_state: ON
          w_thread_state: waiting for handler commit
                 w_errno: 0
                w_errmsg: 
               w_errtime: 0000-00-00 00:00:00.000000
 time_since_last_message: 00:00:00.001134
      applier_busy_state: APPLYING
       lag_from_original: 00:00:01.785881
      lag_from_immediate: 00:00:01.767550
          transport_time: 2.26 ms
       time_to_relay_log: 19.00 us
              apply_time: 29.59 ms
last_applied_transaction: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105183
 last_queued_transaction: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105547
queued_gtid_set_to_apply: 54d83026-40eb-11ee-a5d3-c8cb9e32df8e:105184-105547

With this view we have much more details, including the replication heartbeat, for example. We also have a an overview of GTIDs (queued, applied, …).

We also see information on transport time (the network), time to be write to the relay log and finally time to apply.

Of course, you can use the view to display whatever you need, like this for example:

MySQL InnoDB Cluster, ClusterSet, Read Replicas

If you use the nice integrated solutions managed by the Admin API with MySQL Shell, all that information is already available using the status() method.

The status() method can be extended with 3 values:

  • 1: includes information about the Metadata Version, Group Protocol Version, Group name, cluster member UUIDs, cluster member roles and states as reported byGroup Replication and the list of fenced system variables;
  • 2: includes information about transactions processed by connection and
    applier;
  • 3: includes more detailed stats about the replication machinery of each
    cluster member;

Let’s have a look at a ClusterSet example with extended option 3:

JS> cs.status({extended:3})
{
    "clusters": {
        "cluster2": {
            "clusterRole": "REPLICA", 
            "clusterSetReplication": {
                "applierQueuedTransactionSet": "54d83026-40eb-11ee-a5d3-c8cb9e32df8e:137385-138500", 
                "applierQueuedTransactionSetSize": 1116, 
                "applierState": "ON", 
                "applierStatus": "APPLYING", 
                "applierThreadState": "waiting for handler commit", 
                "applierWorkerThreads": 4, 
                "coordinatorState": "ON", 
                "coordinatorThreadState": "Waiting for replica workers to process their queues", 
                "options": {
                    "connectRetry": 3, 
                    "delay": 0, 
                    "heartbeatPeriod": 30, 
                    "retryCount": 10
                }, 
                "receiver": "127.0.0.1:4420", 
                "receiverStatus": "ON", 
                "receiverThreadState": "Waiting for source to send event", 
                "receiverTimeSinceLastMessage": "00:00:00.002737", 
                "replicationSsl": null, 
                "source": "127.0.0.1:3310"
            }, 
            "clusterSetReplicationStatus": "OK", 
            "communicationStack": "MYSQL", 
            "globalStatus": "OK", 
            "groupName": "7b6bf13d-40ed-11ee-bfdd-c8cb9e32df8e", 
            "groupViewChangeUuid": "7b6bf4f0-40ed-11ee-bfdd-c8cb9e32df8e", 
            "paxosSingleLeader": "OFF", 
            "receivedTransactionSet": "54d83026-40eb-11ee-a5d3-c8cb9e32df8e:129-138500", 
            "ssl": "REQUIRED", 
            "status": "OK_NO_TOLERANCE", 
            "statusText": "Cluster is NOT tolerant to any failures.", 
            "topology": {
                "127.0.0.1:4420": {
                    "address": "127.0.0.1:4420", 
                    "applierWorkerThreads": 4, 
                    "fenceSysVars": [
                        "read_only", 
                        "super_read_only"
                    ], 
                    "memberId": "c3d726ac-40ec-11ee-ab38-c8cb9e32df8e", 
                    "memberRole": "PRIMARY", 
                    "memberState": "ONLINE", 
                    "mode": "R/O", 
                    "readReplicas": {}, 
                    "replicationLagFromImmediateSource": "00:00:05.420247", 
                    "replicationLagFromOriginalSource": "00:00:05.433548", 
                    "role": "HA", 
                    "status": "ONLINE", 
                    "version": "8.1.0"
                }, 
                "127.0.0.1:4430": {
                    "address": "127.0.0.1:4430", 
                    "applierWorkerThreads": 4, 
                    "fenceSysVars": [
                        "read_only", 
                        "super_read_only"
                    ], 
                    "memberId": "709b15ea-40ed-11ee-a9b3-c8cb9e32df8e", 
                    "memberRole": "SECONDARY", 
                    "memberState": "ONLINE", 
                    "mode": "R/O", 
                    "readReplicas": {}, 
                    "replicationLagFromImmediateSource": "00:00:00.038075", 
                    "replicationLagFromOriginalSource": "00:00:05.432536", 
                    "role": "HA", 
                    "status": "ONLINE", 
                    "version": "8.1.0"
                }
            }, 
            "transactionSet": "2cb77a02-40eb-11ee-83f4-c8cb9e32df8e:1-4,54d83026-40eb-11ee-a5d3-c8cb9e32df8e:1-137384,54d8329c-40eb-11ee-a5d3-c8cb9e32df8e:1-5,7b6bf4f0-40ed-11ee-bfdd-c8cb9e32df8e:1-3", 
            "transactionSetConsistencyStatus": "OK", 
            "transactionSetErrantGtidSet": "", 
            "transactionSetMissingGtidSet": "54d83026-40eb-11ee-a5d3-c8cb9e32df8e:137385-138552"
        }, 
        "myCluster": {
            "clusterRole": "PRIMARY", 
            "communicationStack": "MYSQL", 
            "globalStatus": "OK", 
            "groupName": "54d83026-40eb-11ee-a5d3-c8cb9e32df8e", 
            "groupViewChangeUuid": "54d8329c-40eb-11ee-a5d3-c8cb9e32df8e", 
            "paxosSingleLeader": "OFF", 
            "primary": "127.0.0.1:3310", 
            "ssl": "REQUIRED", 
            "status": "OK", 
            "statusText": "Cluster is ONLINE and can tolerate up to ONE failure.", 
            "topology": {
                "127.0.0.1:3310": {
                    "address": "127.0.0.1:3310", 
                    "applierWorkerThreads": 4, 
                    "fenceSysVars": [], 
                    "memberId": "2cb77a02-40eb-11ee-83f4-c8cb9e32df8e", 
                    "memberRole": "PRIMARY", 
                    "memberState": "ONLINE", 
                    "mode": "R/W", 
                    "readReplicas": {
                        "127.0.0.1:4410": {
                            "address": "127.0.0.1:4410", 
                            "applierStatus": "APPLYING", 
                            "applierThreadState": "waiting for handler commit", 
                            "applierWorkerThreads": 4, 
                            "receiverStatus": "ON", 
                            "receiverThreadState": "Waiting for source to send event", 
                            "replicationSources": [
                                "PRIMARY"
                            ], 
                            "replicationSsl": "TLS_AES_256_GCM_SHA384 TLSv1.3", 
                            "role": "READ_REPLICA", 
                            "status": "ONLINE", 
                            "version": "8.1.0"
                        }
                    }, 
                    "role": "HA", 
                    "status": "ONLINE", 
                    "version": "8.1.0"
                }, 
                "127.0.0.1:3320": {
                    "address": "127.0.0.1:3320", 
                    "applierWorkerThreads": 4, 
                    "fenceSysVars": [
                        "read_only", 
                        "super_read_only"
                    ], 
                    "memberId": "327cb102-40eb-11ee-9904-c8cb9e32df8e", 
                    "memberRole": "SECONDARY", 
                    "memberState": "ONLINE", 
                    "mode": "R/O", 
                    "readReplicas": {}, 
                    "replicationLagFromImmediateSource": "00:00:04.536190", 
                    "replicationLagFromOriginalSource": "00:00:04.536190", 
                    "role": "HA", 
                    "status": "ONLINE", 
                    "version": "8.1.0"
                }, 
                "127.0.0.1:3330": {
                    "address": "127.0.0.1:3330", 
                    "applierWorkerThreads": 4, 
                    "fenceSysVars": [
                        "read_only", 
                        "super_read_only"
                    ], 
                    "memberId": "3d141d7e-40eb-11ee-933b-c8cb9e32df8e", 
                    "memberRole": "SECONDARY", 
                    "memberState": "ONLINE", 
                    "mode": "R/O", 
                    "readReplicas": {}, 
                    "replicationLagFromImmediateSource": "00:00:04.652745", 
                    "replicationLagFromOriginalSource": "00:00:04.652745", 
                    "role": "HA", 
                    "status": "ONLINE", 
                    "version": "8.1.0"
                }
            }, 
            "transactionSet": "2cb77a02-40eb-11ee-83f4-c8cb9e32df8e:1-4,54d83026-40eb-11ee-a5d3-c8cb9e32df8e:1-138552,54d8329c-40eb-11ee-a5d3-c8cb9e32df8e:1-5"
        }
    }, 
    "domainName": "myClusterSet", 
    "globalPrimaryInstance": "127.0.0.1:3310", 
    "metadataServer": "127.0.0.1:3310", 
    "primaryCluster": "myCluster", 
    "status": "HEALTHY", 
    "statusText": "All Clusters available."
}

MySQL HeatWave

If you’re using any type of replication in MySQL HeatWave on OCI, you can use the same views, but you need to create them on a different database as sys is write-protected.

So if you’re using HA, Read Replicas or manual replication channels, you can also use the same views to get an accurate overview of replication.

Conclusion

Replication observability is very detailed and provides a lot of information with MySQL 8. Maybe now is a good time to change the way you view or monitor replication.

Enjoy MySQL !

Today, I published the following diary on isc.sans.edu: “Have You Ever Heard of the Fernet Encryption Algorithm?“:

In cryptography, there is a gold rule that states to not develop your own algorithm because… it will be probably weak and broken! They are strong algorithms (like AES) that do a great job so why reinvent the wheel? However, there are projects that try to develop new algorithms. One of them is Fernet, described like this… [Read more]

The post [SANS ISC] Have You Ever Heard of the Fernet Encryption Algorithm? appeared first on /dev/random.

August 21, 2023

When working with I²C devices in the Zephyr real-time operating system, a lot is configured behind the scenes using a devicetree, a hierarchical data structure that describes available hardware. It took me some time to figure out how to change the default I²C pins (SDA and SCL). Here are my notes.

I tried this with Zephyr's BME280 Humidity and Pressure Sensor sample. My goal was to flash the sample firmware to Nordic Semiconductor's nRF52840 Dongle with a BME280 breakout board.

What are the default I²C pins?

First I needed to find out the default I²C pins. To check this, I looked at the devicetree source include file for the nRF52840 Dongle's Pin Control:

koan@tux:~$ less ~/zephyrproject/zephyr/boards/arm/nrf52840dongle_nrf52840/nrf52840dongle_nrf52840-pinctrl.dtsi

The relevant configuration for the i2c0 peripheral is as follows:

i2c0_default: i2c0_default {
        group1 {
                psels = <NRF_PSEL(TWIM_SDA, 0, 26)>,
                        <NRF_PSEL(TWIM_SCL, 0, 27)>;
        };
};

i2c0_sleep: i2c0_sleep {
        group1 {
                psels = <NRF_PSEL(TWIM_SDA, 0, 26)>,
                        <NRF_PSEL(TWIM_SCL, 0, 27)>;
                low-power-enable;
        };
};

This configures pin 0.26 for SDA and pin 0.27 for SCL. My first thought was to simply connect the BME280 to these pins on the nRF52840 Dongle. However, upon examining the board's pinout, I discovered that pin 0.26 is only exposed as a pad on the bottom, and pin 0.27 isn't accessible anywhere:

/images/nrf52840-dongle-pinout.png

How to change the default I²C pins?

Since the default I²C pins weren't suitable on this board, I needed to change them. I decided to connect the BME280 sensor board as follows:

BME280

nRF52840 Dongle

SDA

0.31

SCL

0.29

GND

GND

VCC

VDD

This looks like this on a breadboard:

/images/nrf52840-dongle-bme280.jpg

Next, I copied the sample code so I could change it:

koan@tux:~$ cp -r ~/zephyrproject/zephyr/samples/sensor/bme280/ .
koan@tux:~$ cd bme280/

Then, I added a devicetree overlay file called nrf52840dongle_nrf52840.overlay to the project's boards directory. The content of the overlay file is as follows:

/*
 * Configuration of a BME280 device on an I2C bus.
 *
 * Device address 0x76 is assumed. Your device may have a different
 * address; check your device documentation if unsure.
 */
&pinctrl {
        i2c0_default: i2c0_default {
                group1 {
                        psels = <NRF_PSEL(TWIM_SDA, 0, 31)>,
                                <NRF_PSEL(TWIM_SCL, 0, 29)>;
                };
        };

        i2c0_sleep: i2c0_sleep {
                group1 {
                        psels = <NRF_PSEL(TWIM_SDA, 0, 31)>,
                                <NRF_PSEL(TWIM_SCL, 0, 29)>;
                        low-power-enable;
                };
        };
};

&i2c0 {
    status = "okay";
    bme280@76 {
        compatible = "bosch,bme280";
        reg = <0x76>;
    };
};

The &pinctrl section was copied from the devicetree source include file for the nRF52840 Dongle, with pins 26 and 27 changed to 31 and 29, respectively.

Note

Pin 31 is also assigned as the default SCL pin for the i2c1 peripheral. However, this doesn't matter if you're not using i2c1.

The &i2c0 part defines the bme280 sensor with the I²C address 0x76. We need to add this sensor definition here because the nRF52840 Dongle doesn't have it built in.

Note

If you have an Adafruit BME280 breakout board, the address should be 0x77.

So, with this overlay, you can use the BME280 sensor connected to pins 0.31 (SDA) and 0.29 (SCL).

Because the file name of the devicetree overlay is the board name nrf52840dongle_nrf52840 with the extension .overlay, Zephyr's build system picks it up automatically if you're building the project for the nRF52840 Dongle and merges it with the default devicetree of the board. If you're building the project for another board, the overlay will be ignored.

Building the code

Let's build the sample code with this devicetree overlay, adding configuration options to initialize USB at boot and enable Zephyr's shell:

koan@tux:~/bme280$ source ~/zephyrproject/zephyr/zephyr-env.sh
koan@tux:~/bme280$ west build -b nrf52840dongle_nrf52840 . -- -DCONFIG_USB_DEVICE_INITIALIZE_AT_BOOT=y -DCONFIG_SHELL=y

Now let's examine the generated devicetree:

koan@tux:~/bme280$ less build/zephyr/zephyr.dts

This includes the bme280 sensor on the i2c0 bus:

i2c0: i2c@40003000 {
        compatible = "nordic,nrf-twi";
        #address-cells = < 0x1 >;
        #size-cells = < 0x0 >;
        reg = < 0x40003000 0x1000 >;
        clock-frequency = < 0x186a0 >;
        interrupts = < 0x3 0x1 >;
        status = "okay";
        pinctrl-0 = < &i2c0_default >;
        pinctrl-1 = < &i2c0_sleep >;
        pinctrl-names = "default", "sleep";
        bme280@76 {
                compatible = "bosch,bme280";
                reg = < 0x76 >;
        };
};

The Pin Control section shows the I²C pin defaults:

i2c0_default: i2c0_default {
        phandle = < 0x4 >;
        group1 {
                psels = < 0xc001f >, < 0xb001d >;
        };
};
i2c0_sleep: i2c0_sleep {
        phandle = < 0x5 >;
        group1 {
                psels = < 0xc001f >, < 0xb001d >;
                low-power-enable;
        };
};

Here, 1f is of course the hexadecimal representation of 31 and 1d corresponds to 29.

Running the sample

Now that we're confident that the devicetree is correct, let's create a firmware package and flash it to the nRF52840 Dongle:

koan@tux:~/bme280$ nrfutil pkg generate --hw-version 52 --sd-req=0x00 --application build/zephyr/zephyr.hex --application-version 1 bme280.zip

|===============================================================|
|##      ##    ###    ########  ##    ## #### ##    ##  ######  |
|##  ##  ##   ## ##   ##     ## ###   ##  ##  ###   ## ##    ## |
|##  ##  ##  ##   ##  ##     ## ####  ##  ##  ####  ## ##       |
|##  ##  ## ##     ## ########  ## ## ##  ##  ## ## ## ##   ####|
|##  ##  ## ######### ##   ##   ##  ####  ##  ##  #### ##    ## |
|##  ##  ## ##     ## ##    ##  ##   ###  ##  ##   ### ##    ## |
| ###  ###  ##     ## ##     ## ##    ## #### ##    ##  ######  |
|===============================================================|
|You are not providing a signature key, which means the DFU     |
|files will not be signed, and are vulnerable to tampering.     |
|This is only compatible with a signature-less bootloader and is|
|not suitable for production environments.                      |
|===============================================================|

Zip created at bme280.zip
koan@tux:~/bme280$ nrfutil dfu usb-serial -pkg bme280.zip -p /dev/ttyACM0
  [####################################]  100%
Device programmed.

After this, connect to the UART interface over USB:

koan@tux:~$ screen /dev/ttyACM0

You should see that the I²C device has been detected, and the sensor values are being displayed:

[00:00:00.317,932] <dbg> BME280: bme280_chip_init: ID OK
[00:00:00.328,582] <dbg> BME280: bme280_chip_init: "bme280@76" OK
*** Booting Zephyr OS build zephyr-v3.4.0-837-gb4ed6c4300a2 ***
Found device "bme280@76", getting sensor data
temp: 25.850000; press: 101.987800; humidity: 59.248046
temp: 25.840000; press: 101.986144; humidity: 59.187500
temp: 25.850000; press: 101.985195; humidity: 59.176757

Today, I published the following diary on isc.sans.edu: “Quick Malware Triage With Inotify Tools“:

When you handle a lot of malicious files, you must have a process and tools in place to speedup the analysis. It’s impossible to investigate all files and a key point is to find interesting files that deserve more attention. In my malware analysis lab, I use a repository called my “Malware Zoo” where I put all the files. This repository is shared across different hosts (my computer, REMnux and Windows virtual machines). This helps me to keep all the “dangerous files” in a central location and avoid spreading dangerous stuff everywhere. When you analyze a malware, you’ll quickly generate more files: You extract shellcodes, configurations, DLLs, more executables and those files should also be analyzed. To perform a quick triage with basic operations, I rely on the Inotify suite… [Read more]

The post [SANS ISC] Quick Malware Triage With Inotify Tools appeared first on /dev/random.

August 20, 2023

After having explained how to build MySQL 8 (MySQL 8.0 and MySQL 8.1) on OL9 and OL7, this episode of the series will cover how to build MySQL 8 on Oracle Linux 8 (OL8) and compatible (EL8, CentOS 8, …).

My build machine is a VM.Standard.E4.Flex instance on OCI having 8 OCPUs and 128GB of ram with Oracle Linux 8.8. The machine has also a block volume of 50GB attached and mounted in /home/opc/rpmbuild:

Getting the source RPM

To get the source RPM, you need first to install the MySQL Community’s repo:

$ sudo dnf install -y https://dev.mysql.com/get/mysql80-community-release-el8-7.noarch.rpm

But comparing to OL7 and OL9, this is maybe the most complicate process if you are not aware of how to proceed.

To be able to use the MySQL Community Repo, we need first to remove the one provided by the system (see this article):

$ sudo dnf module disable mysql -y

And now we can download the src rpm for the version we want to recompile.

For MySQL 8.0:

$ dnf download --source mysql-community-server

And for the latest MySQL Innovation Release (8.1):

$ dnf download --source mysql-community-server --enablerepo=mysql-innovation-community

On my system, both files are present:

$ ls -lh *src.rpm
-rw-rw-r--. 1 opc opc 514M Aug 19 11:12 mysql-community-8.0.34-1.el8.src.rpm
-rw-rw-r--. 1 opc opc 515M Aug 19 11:11 mysql-community-8.1.0-1.el8.src.rpm

Dependencies

To build RPMs, you need at least the following package:

$ sudo dnf install -y rpm-build

Of course MySQL also need several packages including libraries and compiler required to compile it. To know which one we need, we can already try to rebuild the rpm:

$ rpmbuild  --rebuild mysql-community-8.1.0-1.el8.src.rpm 
[...]
error: Failed build dependencies:
	cmake >= 3.6.1 is needed by mysql-community-8.1.0-1.el8.x86_64
	cyrus-sasl-devel is needed by mysql-community-8.1.0-1.el8.x86_64
	gcc-toolset-12-annobin-annocheck is needed by mysql-community-8.1.0-1.el8.x86_64
	gcc-toolset-12-annobin-plugin-gcc is needed by mysql-community-8.1.0-1.el8.x86_64
	libaio-devel is needed by mysql-community-8.1.0-1.el8.x86_64
	libtirpc-devel is needed by mysql-community-8.1.0-1.el8.x86_64
	ncurses-devel is needed by mysql-community-8.1.0-1.el8.x86_64
	numactl-devel is needed by mysql-community-8.1.0-1.el8.x86_64
	openldap-devel is needed by mysql-community-8.1.0-1.el8.x86_64
	perl is needed by mysql-community-8.1.0-1.el8.x86_64
	perl(Env) is needed by mysql-community-8.1.0-1.el8.x86_64
	perl(JSON) is needed by mysql-community-8.1.0-1.el8.x86_64
	perl(Memoize) is needed by mysql-community-8.1.0-1.el8.x86_64
	perl(Time::HiRes) is needed by mysql-community-8.1.0-1.el8.x86_64
	rpcgen is needed by mysql-community-8.1.0-1.el8.x86_64

So let’s start by installing those packages:

$ sudo dnf install -y cmake cyrus-sasl-devel gcc-toolset-12-annobin-annocheck \                      
                gcc-toolset-12-annobin-plugin-gcc libaio-devel libtirpc-devel \
                ncurses-devel numactl-devel openldap-devel  perl perl-JSON \
                perl-Env perl-Memoize 
$ sudo dnf install -y https://yum.oracle.com/repo/OracleLinux/OL8/codeready/builder/x86_64/getPackage/rpcgen-1.3.1-4.el8.x86_64.rpm

We also need to install the following two packages:

$ sudo dnf install -y systemd-devel curl-devel

Building RPMs

Now we are ready to build all the rpms from the source package. This is the command to rebuild everything for the version you want:

$ rpmbuild --rebuild mysql-community-8.1.0-1.el8.src.rpm

When done, all the new rpms can be found in ~/rpmbuild/RPMS/x86_64/:

$ ls -lh ~opc/rpmbuild/RPMS/x86_64/
total 966M
-rw-rw-r--. 1 opc opc  17M Aug 19 12:40 mysql-community-client-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc  36M Aug 19 12:44 mysql-community-client-debuginfo-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 3.6M Aug 19 12:41 mysql-community-client-plugins-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 4.3M Aug 19 12:44 mysql-community-client-plugins-debuginfo-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 671K Aug 19 12:40 mysql-community-common-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 3.3M Aug 19 12:42 mysql-community-debuginfo-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc  25M Aug 19 12:42 mysql-community-debugsource-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 2.3M Aug 19 12:41 mysql-community-devel-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 2.3M Aug 19 12:40 mysql-community-icu-data-files-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 1.6M Aug 19 12:41 mysql-community-libs-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 3.0M Aug 19 12:44 mysql-community-libs-debuginfo-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc  36M Aug 19 12:39 mysql-community-server-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc  27M Aug 19 12:40 mysql-community-server-debug-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 126M Aug 19 12:44 mysql-community-server-debug-debuginfo-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 242M Aug 19 12:43 mysql-community-server-debuginfo-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 363M Aug 19 12:41 mysql-community-test-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc  28M Aug 19 12:44 mysql-community-test-debuginfo-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc 5.2M Aug 19 12:41 mysql-router-community-8.1.0-1.el8.x86_64.rpm
-rw-rw-r--. 1 opc opc  45M Aug 19 12:44 mysql-router-community-debuginfo-8.1.0-1.el8.x86_64.rpm

And voilà !

Once again, this is not too trivial when we know how it works and what are the right dependencies.

As usual, enjoy MySQL !

August 19, 2023

Following our discussion with Simon about compiling MySQL 8.0.34 and 8.1.0 for OL9 (RedHat and CentOS Stream as well), see this blog post, we realized that compiling the latest MySQL 8.x on OL7 (EL7) wasn’t very simple either.

After soliciting the MySQL release engineer team, I was able to compile MySQL correctly from its src rpm.

Let me share the various steps with you.

My build machine is a VM.Standard.E4.Flex instance on OCI having 4 OCPUs and 64GB of ram with Oracle Linux 7.9. The machine has also a block volume of 50GB attached and mounted in /home/opc/rpmbuild.

Getting the source RPM

To get the source RPM, you need first to install the MySQL Community’s repo:

$ sudo yum install -y https://dev.mysql.com/get/mysql80-community-release-el7-9.noarch.rpm

Depending on which version you want to compile, you can use one of the following commands:

For latest 8.0:

$ yumdownloader --source mysql-community-server

For latest Innovation Release (8.1 at the moment):

$ yumdownloader --source mysql-community-server --enablerepo=mysql-innovation-community

Now we have both files we can use:

$ ls -lh *src.rpm
-rw-rw-r--. 1 opc opc 544M Jun 25 03:05 mysql-community-8.0.34-1.el7.src.rpm
-rw-rw-r--. 1 opc opc 544M Jun 23 08:23 mysql-community-8.1.0-1.el7.src.rpm

Removing Problematic Packages

I learned from the MySQL Release Engineering Team, that ksplice packages will break any rpm build.

This mean that on my build machine, I will have to remove all ksplice packages.

$ rpm -qa | grep ksplice
ksplice-release-el7-1.0-7.el7.x86_64
ksplice-core0-1.0.59-1.el7.x86_64
ksplice-1.0.59-1.el7.x86_64
ksplice-tools-1.0.59-1.el7.x86_64

This is not much… let’s do it now:

$ sudo yum remove -y ksplice ksplice-core0 ksplice-release-el7 ksplice-tools

Dependencies

To build RPMs, we need to start by installing the following package:

$ sudo yum install rpm-build

Of course MySQL also need several packages including libraries and compiler required to compile it.

To know which one we need, we can already try to rebuild the rpm:

$ rpmbuild --rebuild mysql-community-8.1.0-1.el7.src.rpm
[...]
error: Failed build dependencies:
	cmake3 >= 3.6.1 is needed by mysql-community-8.1.0-1.el7.x86_64
	devtoolset-11-gcc is needed by mysql-community-8.1.0-1.el7.x86_64
	devtoolset-11-gcc-c++ is needed by mysql-community-8.1.0-1.el7.x86_64
	devtoolset-11-binutils is needed by mysql-community-8.1.0-1.el7.x86_64
	bison >= 2.1 is needed by mysql-community-8.1.0-1.el7.x86_64
	perl(Env) is needed by mysql-community-8.1.0-1.el7.x86_64
	perl(Data::Dumper) is needed by mysql-community-8.1.0-1.el7.x86_64
	perl(JSON) is needed by mysql-community-8.1.0-1.el7.x86_64
	libaio-devel is needed by mysql-community-8.1.0-1.el7.x86_64
	ncurses-devel is needed by mysql-community-8.1.0-1.el7.x86_64
	numactl-devel is needed by mysql-community-8.1.0-1.el7.x86_64
	openssl-devel is needed by mysql-community-8.1.0-1.el7.x86_64
	zlib-devel is needed by mysql-community-8.1.0-1.el7.x86_64
	cyrus-sasl-devel is needed by mysql-community-8.1.0-1.el7.x86_64
	openldap-devel is needed by mysql-community-8.1.0-1.el7.x86_64

So let’s start by installing those packages:

$ sudo yum install -y cmake devtoolset-11-gcc devtoolset-11-gcc-c++ \
                      devtoolset-11-binutils bison perl perl-Data-Dumper \
                      perl-JSON libaio-devel ncurses-devel numactl-devel \
                      openssl-devel zlib-devel cyrus-sasl-devel openldap-devel \ 
                      perl-Env
$ sudo yum install -y cmake3 --enablerepo=ol7_developer_EPEL

Then they are other packages that are required but not listed (we also need to install the system gcc-c++ to build MySQL 5.6 for the shared compat libraries):

$ sudo yum install -y systemd-devel curl-devel gcc-c++

Building RPMs

Now we are ready to build all the rpms from the source package.

This is the command to rebuild everything for the version you want:

$ rpmbuild --rebuild mysql-community-8.1.0-1.el7.src.rpm

When done, all the new rpms can be found in ~/rpmbuild/RPMS/x86_64/:

$ ls -lh ~/rpmbuild/RPMS/x86_64/
total 946M
-rw-rw-r--. 1 opc opc  17M Aug 19 00:07 mysql-community-client-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc 3.6M Aug 19 00:08 mysql-community-client-plugins-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc 667K Aug 19 00:07 mysql-community-common-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc 486M Aug 19 00:10 mysql-community-debuginfo-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc 1.9M Aug 19 00:08 mysql-community-devel-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc 4.1M Aug 19 00:08 mysql-community-embedded-compat-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc 2.3M Aug 19 00:07 mysql-community-icu-data-files-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc 1.6M Aug 19 00:08 mysql-community-libs-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc 669K Aug 19 00:08 mysql-community-libs-compat-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc  36M Aug 19 00:06 mysql-community-server-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc  27M Aug 19 00:07 mysql-community-server-debug-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc 363M Aug 19 00:08 mysql-community-test-8.1.0-1.el7.x86_64.rpm
-rw-rw-r--. 1 opc opc 5.2M Aug 19 00:08 mysql-router-community-8.1.0-1.el7.x86_64.rpm

Conclusion

Building an RPM for MySQL on Oracle Linux 7 using the source RPM is an involved but rewarding process and you can also use the same process to build extra RPMs for your own components.

You need then to install the src rpm, modify the mysql.spec file (or work on a copy). This process is similar to what I explained in this article.

As we wrap up, it’s always a good practice to test the custom RPMs in a controlled environment before deploying it into production. This ensures that everything runs as expected and reduces the potential for unforeseen issues.

Thank you for journeying through this guide. And as usual, enjoy MySQL !

August 18, 2023

There are plenty GUI and Web application used to monitor a MySQL server. But if you are long time MySQL DBA, you might have used (and abused) Innotop !

I loved it ! And I even became maintainer of it. This particular task became more and more complicated with the different forks and their differences. Also, let’s be honest, Perl saved my life so many times in the past… but this was in the past. These days, having Perl on a system is more complicated.

But Innotop is still very popular in the MySQL world and to help me maintaining it, I would like to welcome a new member in the maintainer group: yoku0825. Tsubasa Tanaka has been a long time user and contributor of Innotop and I’m sure will keep to good work.

I’ve tried to find an alternative to Innotop, and I even wrote my own clone in Go for MySQL 8.0: innotopgo. But some limitations of the framework I used affected my motivation…

But some time ago, Charles Thompson contacted me about a new tool he was writing. He was looking for feedback.

The tool was very promising and finally this week he released it !

The tool is written in Python 3 and it’s very easy to modify it to contribute code.

Dolphie, the name of the tool, is available on GitHub and can easily be installed using pip:

$ pip install dolphie

Dolphie is already very complete and supports several new features available in MySQL 8.0.

For example I do like the Transaction History, that display the statement that were done inside a running transaction:

Initial Dashboard

Dolphie also integrates the error log from Performance_Schema:

And it also allows searches:

Trending

Dolphie also provides some very interesting trending graphs that can be used to look at performance issues.

This is an example:

The best way to discover all its possibilities is to install and test it.

Conclusion

Dolphie is a brand new Open Source (GPLv3) tool for MySQL DBAs, made for the Community by the Community. It’s very easy to get involved, as Dolphie is written in Python, and Charles, its author, is very responsive in implementing features and solving problems.

I really encourage you to test it, submit bugs, feature requests and, of course, contributions !

Welcome Dolphie and long life !

Today, I published the following diary on isc.sans.edu: “From a Zalando Phishing to a RAT“:

Phishing remains a lucrative threat. We get daily emails from well-known brands (like DHL, PayPal, Netflix, Microsoft, Dropbox, Apple, etc). Recently, I received a bunch of phishing emails targeting Zalando customers. Zalando is a German retailer of shoes, fashion across Europe. It was the first time I saw them used in a phishing campaign… [Read more]

The post [SANS ISC] From a Zalando Phishing to a RAT appeared first on /dev/random.

August 16, 2023

I've been maintaining a number of Perl software packages recently. There's SReview, my video review and transcoding system of which I split off Media::Convert a while back; and as of about a year ago, I've also added PtLink, an RSS aggregator (with future plans for more than just that).

All these come with extensive test suites which can help me ensure that things continue to work properly when I play with things; and all of these are hosted on salsa.debian.org, Debian's gitlab instance. Since we're there anyway, I configured GitLab CI/CD to run a full test suite of all the software, so that I can't forget, and also so that I know sooner rather than later when things start breaking.

GitLab has extensive support for various test-related reports, and while it took a while to be able to enable all of them, I'm happy to report that today, my perl test suites generate all three possible reports. They are:

  • The coverage regex, which captures the total reported coverage for all modules of the software; it will show the test coverage on the right-hand side of the job page (as in this example), and it will show what the delta in that number is in merge request summaries (as in this example
  • The JUnit report, which tells GitLab in detail which tests were run, what their result was, and how long the test took (as in this example)
  • The cobertura report, which tells GitLab which lines in the software were ran in the test suite; it will show up coverage of affected lines in merge requests, but nothing more. Unfortunately, I can't show an example here, as the information seems to be no longer available once the merge request has been merged.

Additionally, I also store the native perl Devel::Cover report as job artifacts, as they show some information that GitLab does not.

It's important to recognize that not all data is useful. For instance, the JUnit report allows for a test name and for details of the test. However, the module that generates the JUnit report from TAP test suites does not make a distinction here; both the test name and the test details are reported as the same. Additionally, the time a test took is measured as the time between the end of the previous test and the end of the current one; there is no "start" marker in the TAP protocol.

That being said, it's still useful to see all the available information in GitLab. And it's not even all that hard to do:

test:
  stage: test
  image: perl:latest
  coverage: '/^Total.* (\d+.\d+)$/'
  before_script:
    - cpanm ExtUtils::Depends Devel::Cover TAP::Harness::JUnit Devel::Cover::Report::Cobertura
    - cpanm --notest --installdeps .
    - perl Makefile.PL
  script:
    - cover -delete
    - HARNESS_PERL_SWITCHES='-MDevel::Cover' prove -v -l -s --harness TAP::Harness::JUnit
    - cover
    - cover -report cobertura
  artifacts:
    paths:
    - cover_db
    reports:
      junit: junit_output.xml
      coverage_report:
        path: cover_db/cobertura.xml
        coverage_format: cobertura

Let's expand on that a bit.

The first three lines should be clear for anyone who's used GitLab CI/CD in the past. We create a job called test; we start it in the test stage, and we run it in the perl:latest docker image. Nothing spectacular here.

The coverage line contains a regular expression. This is applied by GitLab to the output of the job; if it matches, then the first bracket match is extracted, and whatever that contains is assumed to contain the code coverage percentage for the code; it will be reported as such in the GitLab UI for the job that was ran, and graphs may be drawn to show how the coverage changes over time. Additionally, merge requests will show the delta in the code coverage, which may help deciding whether to accept a merge request. This regular expression will match on a line of that the cover program will generate on standard output.

The before_script section installs various perl modules we'll need later on. First, we intall ExtUtils::Depends. My code uses ExtUtils::MakeMaker, which ExtUtils::Depends depends on (no pun intended); obviously, if your perl code doesn't use that, then you don't need to install it. The next three modules -- Devel::Cover, TAP::Harness::JUnit and Devel::Cover::Report::Cobertura are necessary for the reports, and you should include them if you want to copy what I'm doing.

Next, we install declared dependencies, which is probably a good idea for you as well, and then we run perl Makefile.PL, which will generate the Makefile. If you don't use ExtUtils::MakeMaker, update that part to do what your build system uses. That should be fairly straightforward.

You'll notice that we don't actually use the Makefile. This is because we only want to run the test suite, which in our case (since these are PurePerl modules) doesn't require us to build the software first. One might consider that this makes the call of perl Makefile.PL useless, but I think it's a useful test regardless; if that fails, then obviously we did something wrong and shouldn't even try to go further.

The actual tests are run inside a script snippet, as is usual for GitLab. However we do a bit more than you would normally expect; this is required for the reports that we want to generate. Let's unpack what we do there:

cover -delete

This deletes any coverage database that might exist (e.g., due to caching or some such). We don't actually expect any coverage database, but it doesn't hurt.

HARNESS_PERL_SWITCHES='-MDevel::Cover'

This tells the TAP harness that we want it to load the Devel::Cover addon, which can generate code coverage statistics. It stores that in the cover_db directory, and allows you to generate all kinds of reports on the code coverage later (but we don't do that here, yet).

prove -v -l -s

Runs the actual test suite, with verbose output, shuffling (aka, randomizing) the test suite, and adding the lib directory to perl's include path. This works for us, again, because we don't actually need to compile anything; if you do, then -b (for blib) may be required.

ExtUtils::MakeMaker creates a test target in its Makefile, and usually this is how you invoke the test suite. However, it's not the only way to do so, and indeed if you want to generate a JUnit XML report then you can't do that. Instead, in that case, you need to use the prove, so that you can tell it to load the TAP::Harness::JUnit module by way of the --harness option, which will then generate the JUnit XML report. By default, the JUnit XML report is generated in a file junit_output.xml. It's possible to customize the filename for this report, but GitLab doesn't care and neither do I, so I don't. Uploading the JUnit XML format tells GitLab which tests were run and

Finally, we invoke the cover script twice to generate two coverage reports; once we generate the default report (which generates HTML files with detailed information on all the code that was triggered in your test suite), and once with the -report cobertura parameter, which generates the cobertura XML format.

Once we've generated all our reports, we then need to upload them to GitLab in the right way. The native perl report, which is in the cover_db directory, is uploaded as a regular job artifact, which we can then look at through a web browser, and the two XML reports are uploaded in the correct way for their respective formats.

All in all, I find that doing this makes it easier to understand how my code is tested, and why things go wrong when they do.

August 11, 2023

Today, I published the following diary on isc.sans.edu: “Show me All Your Windows!“:

It’s a key point for attackers to implement anti-debugging and anti-analysis techniques. Anti-debugging means the malware will try to detect if it’s being debugged (executed in a debugger or its execution is slower than expected). Anti-analysis refers to techniques to detect if the malware is detonated in a sandbox or by a malware analyst. In such cases, tools run in parallel with the malware to collect live data (packets, API calls, files, or registry activity)… [Read more]

The post [SANS ISC] Show me All Your Windows! appeared first on /dev/random.

August 10, 2023

Wilco is in town tonight, playing the “Lokerse Feesten” ; an open air 9 day festival here, a mere 5 minutes bicycle ride from home. I’m not counting on it, but it would be great if I could see, hear and feel “Impossible Germany” live … Update the day after; they did play Impossible Germany, as always impressed by Cline’s (improvised) solo. Watch this video on YouTube.

Source

After discussing with Simon about some issues when trying to recompile MySQL 8.0.34 on CentOS 9 (see #111159), I also tried it and indeed some dependencies are not listed when compiling via the source RPM.

Let’s see how to recompile the two latest versions of MySQL (8.0.34 and 8.1.0) using the source RPMs.

I use Oracle Linux 9 as build machine.

Getting the source RPM

To get the source RPM, you need first to install the MySQL Community’s repo:

$ sudo dnf install https://dev.mysql.com/get/mysql80-community-release-el9-3.noarch.rpm

Then depending on which version you want to compile, you can use one of the following commands:

For latest 8.0:

$ dnf download --source mysql-community-server

For latest Innovation Release (8.1 at the moment):

$ dnf download --source mysql-community-server --enablerepo mysql-innovation-community

Now you have the src RPM files you can use to rebuild all the RPMS:

$ ls -lh *src.rpm
-rw-r--r--. 1 root root 514M Aug  7 09:30 mysql-community-8.0.34-1.el9.src.rpm
-rw-r--r--. 1 root root 515M Aug  8 21:27 mysql-community-8.1.0-1.el9.src.rpm

Dependencies

To build RPMs, you need at least the following package:

$ sudo dnf install -y rpm-build

Then, if you try to build the downloaded src rpm, some dependencies will be required.

Let’s see how to install them:

$ sudo dnf install -y cmake cyrus-sasl-devel gcc-toolset-12-annobin-annocheck \
    gcc-toolset-12-annobin-plugin-gcc libaio-devel ncurses-devel numactl-devel \ 
    openldap-devel rpcgen perl perl-JSON perl-Time
$ sudo dnf install -y https://yum.oracle.com/repo/OracleLinux/OL9/codeready/builder/x86_64/getPackage/libtirpc-devel-1.3.3-1.el9.x86_64.rpm

There are also some other packages that are required but not listed as BuildRequires in the spec file:

$ sudo dnf install -y krb5-devel libudev-devel libcurl-devel
$ sudo dnf install -y https://yum.oracle.com/repo/OracleLinux/OL9/codeready/builder/x86_64/getPackage/libfido2-devel-1.6.0-7.el9.x86_64.rpm

Building RPMs

Now we are ready to build all the rpms from source.

This is the command to rebuild everything for the version you want:

$ rpmbuild --rebuild mysql-community-8.1.0-1.el9.src.rpm

When done (this takes some time), you will find all the new rpms in ~/rpmbuild/RPMS/x86_64/:

$ ls -lh 
-rw-r--r--. 1 root root 3.5M Aug  8 23:24 mysql-community-client-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  23M Aug  8 23:24 mysql-community-client-debuginfo-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 1.4M Aug  8 23:25 mysql-community-client-plugins-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 1.9M Aug  8 23:24 mysql-community-client-plugins-debuginfo-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 559K Aug  8 23:24 mysql-community-common-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 3.9M Aug  8 23:24 mysql-community-debuginfo-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  21M Aug  8 23:25 mysql-community-debugsource-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 2.1M Aug  8 23:24 mysql-community-devel-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 2.3M Aug  8 23:25 mysql-community-icu-data-files-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 1.5M Aug  8 23:25 mysql-community-libs-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 2.2M Aug  8 23:24 mysql-community-libs-debuginfo-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  20M Aug  8 23:24 mysql-community-server-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  25M Aug  8 23:25 mysql-community-server-debug-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 127M Aug  8 23:28 mysql-community-server-debug-debuginfo-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 181M Aug  8 23:28 mysql-community-server-debuginfo-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 349M Aug  8 23:27 mysql-community-test-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  18M Aug  8 23:25 mysql-community-test-debuginfo-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 3.7M Aug  8 23:24 mysql-router-community-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root  32M Aug  8 23:24 mysql-router-community-debuginfo-8.1.0-1.el9.x86_64.rpm

Patching MySQL and build the RPMs

From the same source RPM it’s also possible to modify MySQL and create RPMS.

To illustrate this, we will create a RPM for a component of ours.

The first step is to install the RPM package with the source (the one we used to rebuild the packages):

$ rpm -ivh mysql-community-8.1.0-1.el9.src.rpm

The component we will use is uuid_v7 available on GitHub.

The recommended way is to create a patch for the component and then include it in the spec file.

There are plenty methods to create a patch, this is how I personally did:

$ cd /tmp
$ wget https://github.com/lefred/mysql-component-uuid_v7/archive/refs/heads/main.zip
$ unzip main.zip
$ mkdir components components_new
$ mv mysql-component-uuid_v7-main components_new/uuid_v7
$ diff -urN components components_new > mysql-component-uuid_v7.patch

We need to add the patch ~/rpmbuild/SOURCES (where boost and MySQL source tarballs should be already):

$ cp mysql-component-uuid_v7.patch ~/rpmbuild/SOURCES

Now we need to modify the mysql.spec file in ~/rpmbuild/SPECS to include the patch and the new package.

After the last source:

[...]
Source91:     filter-requires.sh

We add the patch:

Patch0:       mysql-component-uuid_v7.patch

Then after the definition and description of the client-plugins package:

[...]
%description    client-plugins
This package contains the client-plugins libraries used by MySQL client applications.

We add the 2 new packages (the normal one and the debug one):

%package        component-uuidv7
Summary:        Adding a UUID v7 generator user function
Group:          Applications/Databases
Provides:       mysql-component-uuidv7 = %{version}-%{release}
Provides:       MySQL-component-uuidv7 = %{version}-%{release}
Requires:       mysql-server = %{version}-%{release}

%description    component-uuidv7
This package contains the component to generate uuid v7 within MySQL Server.

%if 0%{?with_debuginfo}
%package        component-uuidv7-debug
Summary:        Adding a UUID v7 generator user function
Group:          Applications/Databases

%description    component-uuidv7-debug
This package contains the component to generate uuid v7 within MySQL Server with debug info.
%endif # with_debuginfo

We need now to also use our patch, this is done at the end of the %prep section:

%prep
%if 0%{?compatlib}
%setup -q -T -a 0 -a 7 -a 10 -c -n %{src_dir}
%else
%setup -q -T -a 0 -a 10 -c -n %{src_dir}
%endif # 0%{?compatlib}

We need to add:

%patch0 -p1
mv uuid_v7 mysql-%{version}/components

And finally, after the %files libs section:

%files libs
%defattr(-, root, root, -)
%doc %{?license_files_server}
%dir %attr(755, root, root) %{_libdir}/mysql
%attr(644, root, root) %{_sysconfdir}/ld.so.conf.d/mysql-%{_arch}.conf
%{_libdir}/mysql/libmysqlclient.so.22*
%if 0%{?ssl_bundled}
%attr(755, root, root) %{_libdir}/mysql/private/libssl.so
%attr(755, root, root) %{_libdir}/mysql/private/libssl.so.1.1
%attr(755, root, root) %{_libdir}/mysql/private/libcrypto.so
%attr(755, root, root) %{_libdir}/mysql/private/libcrypto.so.1.1
%endif # ssl_bundled

We add the files that must be included in each packages:

%files component-uuidv7
%defattr(-, root, root, -)
%doc %{?license_files_server}
%attr(755, root, root) %{_libdir}/mysql/plugin/component_uuid_v7.so

%if 0%{?with_debuginfo}
%files component-uuidv7-debug
%defattr(-, root, root, -)
%doc %{?license_files_server}
%attr(755, root, root) %{_libdir}/mysql/plugin/component_uuid_v7.so
%attr(755, root, root) %{_libdir}/mysql/plugin/debug/component_uuid_v7.so
#%attr(755, root, root) %{_libdir}/mysql/plugin/component_uuid_v7.so-8.1.0-1.el9.x86_64.debug
%endif # with_debuginfo

We can now compile all the packages using the spec file:

$ rpmbuild -ba mysql.spec

You can also use macro to no redo everything, for example, to not rebuild MySQL Router your can use:

$ rpmbuild -ba mysql.spec --define 'with_router 0'

When finished, you will have a new rpm to install your own component:

$ cd ~/rpmbuild/RPMS/x86_64/
$ ls -lh *uuid*
-rw-r--r--. 1 root root 114K Aug  9 20:00 mysql-community-component-uuidv7-8.1.0-1.el9.x86_64.rpm
-rw-r--r--. 1 root root 169K Aug  9 20:00 mysql-community-component-uuidv7-debug-8.1.0-1.el9.x86_64.rpm

And we can check the info provided by the package:

$ rpm -qi  mysql-community-component-uuidv7-8.1.0-1.el9.x86_64.rpm 
Name        : mysql-community-component-uuidv7
Version     : 8.1.0
Release     : 1.el9
Architecture: x86_64
Install Date: (not installed)
Group       : Applications/Databases
Size        : 520488
License     : Copyright (c) 2000, 2023, Oracle and/or its affiliates. Under GPLv2 license as shown in the Description field.
Signature   : (none)
Source RPM  : mysql-community-8.1.0-1.el9.src.rpm
Build Date  : Wed 09 Aug 2023 06:13:23 PM GMT
Build Host  : build-ol9-x86-64.mysqlpub.mysqlvcn.oraclevcn.com
Packager    : MySQL Release Engineering <mysql-build@oss.oracle.com>
Vendor      : Oracle and/or its affiliates
URL         : http://www.mysql.com/
Summary     : Adding a UUID v7 generator user function
Description :
This package contains the component to generate uuid v7 within MySQL Server.

And the files included in it:

$ rpm -ql mysql-community-component-uuidv7-8.1.0-1.el9.x86_64.rpm 
/usr/lib/.build-id
/usr/lib/.build-id/8e/eb48888cfd2f338d4b11148cd515b9d3ab92ee
/usr/lib64/mysql/plugin/component_uuid_v7.so
/usr/share/doc/mysql-community-component-uuidv7
/usr/share/doc/mysql-community-component-uuidv7/LICENSE
/usr/share/doc/mysql-community-component-uuidv7/README

Conclusion

You have seen how you can build or rebuild MySQL from the source RPM on OL9 (and similar) and you are now also able to patch MySQL and include your patch as part of the rpm or even better if you use a component on it’s own rpm that you can distribute or install easily on multiple instances.

Enjoy compiling MySQL and building RPMs !

In case you are using MySQL 8.1 on OL9 (or similar), here is the RPM of the component if you want to test it:

August 09, 2023

Het dondert ten huize de voorzitster van de lokale Chiro van België. Haar voorgangster zat in de put toen hij voor een paar Euros de zeepkisten van de organisatie verkochte aan Flurk. Flurk heeft een zaak in schroot.

Flurk heeft nu aan de huisfabrikant van de Chiro van België die zeepkisten voor tien keer meer verkocht.

Over een paar maanden is het weer internationale zeepkistenrace aan het front. Flurk is na de race van plan daar nog meer schroot op te kopen.

“Op papier is alles schroot”, reageerde Flurk aan onze redactie. “Met wa Cola kunde de roest verwijderen. Daarna bolt da trug ze!” Zei Flurk nog.

Misschien moet ik maar eens solliciteren bij De Rechtzetting? De werkelijkheid wordt steeds onnauwkeuriger.

August 07, 2023

I eat (breakfast/lunch/supper) at a table while sitting on a chair.

Most of the time, while eating, I can see a rather busy sidewalk where a lot of people are rushing by, hurrying, while eating junk food from a plastic bowl or some kind of (junk food) sandwich from a paper bag. Is this a new thing? Do they not have time to go sit at a table for lunch? Or do they just don't care?

I was taught to never walk while eating or drinking...

Or is this just part of the rat-race where people don't have a choice anymore and need to force feed themselves between two work sessions?


(Also, when did people stop cooking for themselves? Cause I see way too many pizza's being delivered.)

August 06, 2023

ESPHome is an open-source program that allows you to create your own home-automation devices using an ESP32, ESP8266, or RP2040 microcontroller board that you connect to LEDs, sensors, or switches. What sets ESPHome apart from other solutions like Arduino or MicroPython is that you don't need to program. Instead, you define your components and their respective pin connections in a YAML configuration file. ESPHome then generates the necessary C++ code and compiles it into firmware that you can install on the device. [1]

ESPHome is often used with ESP32 development boards. Support for the RP2040 platform, the chip in the popular Raspberry Pi Pico W, is still relatively new (introduced in ESPHome 2022.11). As a result, you may encounter some issues, some things are not clearly documented, and there aren't that many ESPHome example configurations using the RP2040. In this article, I'll share my findings after exploring ESPHome's RP2040 support.

Use the dashboard, not the wizard

A first issue you may encounter is that ESPHome's command-line wizard doesn't support the RP2040 platform. The wizard is typically used to create a new ESPHome project by guiding you through a series of steps and generating a default YAML file. However, as of ESPHome 2023.7.0, the wizard only accepts ESP32 or ESP8266 as a platform:

$ esphome wizard test.yaml
Hi there!
I'm the wizard of ESPHome :)
And I'm here to help you get started with ESPHome.
In 4 steps I'm going to guide you through creating a basic configuration file for your custom ESP8266/ESP32 firmware. Yay!



============= STEP 1 =============
    _____ ____  _____  ______
   / ____/ __ \|  __ \|  ____|
  | |   | |  | | |__) | |__
  | |   | |  | |  _  /|  __|
  | |___| |__| | | \ \| |____
   \_____\____/|_|  \_\______|

===================================
First up, please choose a name for your node.
It should be a unique name that can be used to identify the device later.
For example, I like calling the node in my living room livingroom.

(name): test
Great! Your node is now called "test".


============= STEP 2 =============
      ______  _____ _____
     |  ____|/ ____|  __ \\
     | |__  | (___ | |__) |
     |  __|  \___ \|  ___/
     | |____ ____) | |
     |______|_____/|_|

===================================
Now I'd like to know what microcontroller you're using so that I can compile firmwares for it.
Are you using an ESP32 or ESP8266 platform? (Choose ESP8266 for Sonoff devices)

Please enter either ESP32 or ESP8266.
(ESP32/ESP8266): RP2040
Unfortunately, I can't find an espressif microcontroller called "RP2040". Please try again.

Please enter either ESP32 or ESP8266.
(ESP32/ESP8266):

Fortunately, this limitation doesn't mean that you have to configure your Raspberry Pi Pico W from scratch. Instead, you can use the ESPHome dashboard.

To start the ESPHome dashboard and specify the directory where your ESPHome configuration files are located, run the following command: [2]

$ esphome dashboard config/

This will start a web server on http://0.0.0.0:6052. You can access this URL from your web browser. If you already have ESPHome devices on your network, the dashboard will automatically discover them.

Next, click on New device at the bottom right corner, and then on Continue. Give your device a name and enter the SSID and password for the Wi-Fi network that you want your device to connect to. Click on Next and choose your device type. For an ESP32 or ESP8266, you first need to choose the platform and then the specific board. For the RP2040 platform, the dashboard (as of ESPHome 2023.7.0) only allows you to choose Raspberry Pi Pico W, and also Raspberry Pi Pico if you uncheck Use recommended settings. You can always manually change the board later. For now, let's assume you want to install ESPHome on a Raspberry Pi Pico W. After choosing the platform, the dashboard creates a minimal configuration and shows an encryption key that you can use to allow the ESPHome device to communicate with Home Assistant, the popular open-source home automation gateway developed by the same team behind ESPHome. Finally, click on Install.

ESPHome offers several installation methods, but not all of them are supported by every device. Since there's no ESPHome firmware running on your device yet, the first method (over Wi-Fi) is not yet possible. Plug into the computer running ESPHome Dashboard isn't available either, for the same reason. However, you can always choose Manual download. This gives you instructions on how to accomplish the installation. For the Raspberry Pi Pico W, you need to disconnect the board from USB, hold down the BOOTSEL button while reconnecting the board, and then release the button. This will cause a USB drive named RPI-RP2 to appear in your file manager. In the ESPHome dashboard, click on Download project and then drag the .uf2 file to the USB drive. Once the drive disappears, the board runs your ESPHome firmware, and you can click on Close.

Blinking the built-in LED on the Raspberry Pi Pico W

On the Raspberry Pi Pico, the built-in LED is connected to GPIO25. However, on the Raspberry Pi Pico W, the built-in LED is connected to the Wi-Fi chip, the Infineon CYW43439. To blink the LED on the Raspberry Pi Pico W, you need to use GPIO32. [3] For example, edit your configuration file by clicking on Edit in the box representing your device, and add this YAML configuration:

output:
  - platform: gpio
    pin: 32
    id: led

interval:
  - interval: 1000ms
    then:
      - output.turn_on: led
      - delay: 500ms
      - output.turn_off: led

This configuration adds an output component with the gpio platform, assigning it to GPIO pin 32, which corresponds to the built-in LED of the Raspberry Pi Pico W. Additionally, an interval component is defined to trigger the LED to turn on, wait for 500 ms, and then turn off every 1000ms.

After saving the file (in the web editor at the top right), click on Install. Choose your installation method. Since your Raspberry Pi Pico W is already running ESPHome and connected to your Wi-Fi network, you can choose Wirelessly as the installation method. The board doesn't even need to be connected to your computer's USB port anymore. Your YAML configuration is now transformed into C++ code and compiled. If you see the message INFO Successfully compiled program., the dashboard will upload the new firmware. Once the board reboots, the LED starts blinking.

Note

You can also use the LED pin number. ESPHome automatically translates it to pin 25 for the Raspberry Pi Pico and to pin 32 for the Raspberry Pi Pico W. [4]

Using PWM output

If you want to send a PWM (pulse-width modulation) signal to a GPIO pin on the RP2040, you can use the output component with the rp2040_pwm platform (not yet documented on ESPHome's web site). Here's an example how you can make a dimmable LED connected to GPIO15: [5]

output:
  - platform: rp2040_pwm
    pin: 15
    id: led

light:
  - platform: monochromatic
    name: "Dimmable LED"
    output: led

On a breadboard, connect the anode (the longer leg) of an LED to the Pico W's GPIO15 pin. Then connect the cathode (the shorter leg), via a 220 Ω resistor, to GND. The circuit should look like this:

/images/picow-led-gpio15_bb.png

After uploading this firmware to your board, you can control the LED's brightness using Home Assistant's dashboard:

/images/ha-dimmable-led.png

You can also use a PWM output to create various light effects. This is how you define a slow pulse effect that continuously pulses the LED:

output:
  - platform: rp2040_pwm
    pin: 15
    id: led

light:
  - platform: monochromatic
    output: led
    id: pulsating_led
    effects:
      - pulse:
          name: "Slow pulse"
          transition_length: 2s
          update_interval: 2s

To start this effect, add the following automation to the on_boot section of the esphome core configuration:

esphome:
  name: raspberry-pi-pico-w
  friendly_name: Raspberry Pi Pico W
  on_boot:
    then:
      - light.turn_on:
          id: pulsating_led
          effect: "Slow pulse"

After uploading the firmware to the board, the LED will start slowly pulsing.

Only use I2C0

When using I²C devices with the RP2040, it's important to note that ESPHome's I²C component for the RP2040 only works for the i2c0 bus. To determine which pins to use for I²C, always refer to the Raspberry Pi Pico W's pinout:

/images/picow-pinout.svg

Only the pins defined as I2C0 SDA and I2C0 SCL can be used for I²C communication. This means you can use the following pin combinations for SDA/SCL: GP0/GP1, GP4/GP5, GP8/GP9, GP12/GP13, GP16/GP17, or GP20/GP21.

Warning

If you accidentally use the I2C1 bus, the board won't boot. However, you can recover from this by disconnecting the board from USB, holding down the BOOTSEL button while reconnecting the board, and then releasing the button. Then move the .uf2 file with the correct I²C configuration to the RPI-RP2 drive that appears in your file manager.

Since the GP20/GP21 pins on the Raspberry Pi Pico W are dedicated to I2C0 SDA/I2C0 SCL and do not have any alternative functions, I like to use these pins for the I²C bus. Here's an example configuration to read the temperature, pressure, and humidity from a BME280 sensor board:

i2c:
  sda: 20
  scl: 21

sensor:
  - platform: bme280
    temperature:
      name: "BME280 Temperature"
    pressure:
      name: "BME280 Pressure"
    humidity:
      name: "BME280 Humidity"
    address: 0x77

For some BME280 boards, you need to specify an alternative address, 0x76.

Connect the pins as shown in the following circuit diagram for the BME280:

/images/picow-bme280_bb.png

Note

On Adafruit's version of the sensor board, SDA is called SDI and SCL is referred to as SCK.

Driving addressable LED strips

Traditionally, ESPHome supported adressable LED strips using the NeoPixelBus and FastLED platforms for the Light component. However, these platforms don't work with the RP2040. Instead, you can use the RP2040 PIO LED Strip platform, which uses the RP2040 PIO (Programmable Input Output) peripheral to control various addressable LED strips.

For example, an ESPHome configuration to drive a LED strip of eight WS2812B LEDs looks like this:

light:
  - platform: rp2040_pio_led_strip
    name: led_strip
    id: led_strip
    pin: 13
    num_leds: 8
    pio: 0
    rgb_order: GRB
    chipset: WS2812B

Connect the LED strip's DIN to the Pico W's GPIO13, GND to GND, and the LED strip's power pin to VBUS (3V3 isn't enough voltage, while on VBUS there's 5V from the USB connector).

You can now add various light effects to your configuration.

Other RP2040 microcontroller boards

If you look at the configuration file generated by the ESPHome dashboard, by clicking on Edit in the box representing your device, you'll notice the following section for the platform configuration:

rp2040:
  board: rpipicow
  framework:
    # Required until https://github.com/platformio/platform-raspberrypi/pull/36 is merged
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

Since ESPHome uses PlatformIO under the hood, it depends on PlatformIO's support for microcontroller platforms. However, this doesn't support the Raspberry Pi Pico W yet. Max Gerhardt forked PlatformIO's repository and added support for the Raspberry Pi Pico W, as well as other RP2040-based boards.

Initially, I thought that I could just run ESPHome on RP2040 microcontroller boards other than the Raspberry Pi Pico W by specifying a different board name. To test this, I created a new device in the ESPHome dashboard, chose Raspberry Pi Pico W as the device type, and then pressed Skip to not install the firmware immediately to the board. I clicked on Edit in the box representing my device, and changed the board definition rpipicow to another supported board in Max Gerhardt's fork of the RP2040 development platform for PlatformIO, the Arduino Nano RP2040 Connect (arduino_nano_connect). After saving the configuration, I clicked on Install at the top right corner.

However, when I chose Manual download to download the .uf2 file, the image preparation failed, and I encountered several "undefined reference" errors related to the Wi-Fi chip. I should've expected this actually, as the Arduino Nano RP2040 Connect uses the u-blox NINA-W102 instead of the Raspberry Pi Pico W's Infineon CYW43439 for Wi-Fi. The same holds for the Challenger RP2040 WiFi (challenger_2040_wifi), which uses the ESP8285 for Wi-Fi. It seems that additional work on the ESPHome side is required to support these other boards. I don't know any RP2040 boards from other manufacturers than Raspberry Pi using the CYW43439. If they exist, I suspect they should work with ESPHome.

You can still use non-Wi-Fi RP2040 boards this way. For example, here's a configuration for the Seeed Studio XIAO RP2040 (seeed_xiao_rp2040):

esphome:
  name: xiao-rp2040
  friendly_name: Xiao RP2040
  on_boot:
    then:
      - output.turn_on:
          id: led_rgb_enable
      - light.turn_on:
          id: led_rgb
          effect: "Random colors"

rp2040:
  board: seeed_xiao_rp2040
  framework:
    # Required until https://github.com/platformio/platform-raspberrypi/pull/36 is merged
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

# Enable logging
logger:

output:
  - platform: gpio
    pin: 11
    id: led_rgb_enable

light:
  - platform: rp2040_pio_led_strip
    id: led_rgb
    pin: 12
    num_leds: 1
    pio: 0
    rgb_order: GRB
    chipset: WS2812
    effects:
      - random:
          name: "Random colors"
          transition_length: 1s
          update_interval: 1s

The board has a WS2812 RGB LED connected to GPIO12, so you can address it using the rp2040_pio_led_strip platform with num_leds set to 1. The light component also defines a light effect with random colors. But before you can use the RGB LED on this board, you need to set GPIO11 high, so that's what the output component is for. On boot, we first enable the LED, and then start the light effect.

Even if a board isn't supported yet by Max Gerhardt's repository, you can still use it with ESPHome by specifying the board as rpipico (or rpipicow if you find a board with the CYW43439 Wi-Fi chip). This will probably work if you're not doing anything exotic. As an example, this is a configuration for Pimoroni's Tiny 2040 board:

esphome:
  name: tiny-2040
  friendly_name: Tiny 2040
  on_boot:
    then:
      - light.turn_on:
          id: led_rgb
          effect: "Random colors"

rp2040:
  board: rpipico
  framework:
    # Required until https://github.com/platformio/platform-raspberrypi/pull/36 is merged
    platform_version: https://github.com/maxgerhardt/platform-raspberrypi.git

# Enable logging
logger:

output:
  - platform: rp2040_pwm
    id: led_red
    pin: 18
    inverted: True
  - platform: rp2040_pwm
    id: led_green
    pin: 19
    inverted: True
  - platform: rp2040_pwm
    id: led_blue
    pin: 20
    inverted: True

light:
  - platform: rgb
    id: led_rgb
    red: led_red
    green: led_green
    blue: led_blue
    effects:
      - random:
          name: "Random colors"
          transition_length: 1s
          update_interval: 1s

The board has an RGB LED, which the R, G, and B components connected active low (that's why each output definition has inverted: True) to GPIO18, GPIO19, and GPIO20, respectively. This configuration creates a PWM output for each color component, combines them into one RGB light, and starts a light effect with random colors on boot.

Conclusion

In conclusion, the support for the RP2040 platform in ESPHome is not as mature as the support for ESP32 or ESP8266. It lacks the same level of testing and comprehensive documentation. There are several issues reported, and certain functionalities such as MQTT and Bluetooth Low Energy are not yet supported on the RP2040.

However, for a lot of tasks, ESPHome on the Raspberry Pi Pico W is still quite usable. If you have a spare Raspberry Pi Pico W lying around, give it a try! I'm also hopeful that support for additional RP2040 boards with Wi-Fi will be added in the near future.

[1]

If you're interested in exploring more ESPHome projects with various sensors and electronic components, you may find my book Getting Started with ESPHome: Develop your own custom home automation devices helpful.

[2]

It is recommended to create a directory where you store all your ESPHome projects.

[3]

This doesn't seem to be documented, but you can find this in the esphome/components/rp2040/gpio.py file of ESPHome's source code.

[4]

This also doesn't seem to be documented, but the LED pin is defined in esphome/components/rp2040/boards.py for the Raspberry Pi Pico and Pico W.

[5]

The built-in LED doesn't seem to support PWM.

August 04, 2023

Today, I published the following diary on isc.sans.edu: “Are Leaked Credentials Dumps Used by Attackers?“:

Leaked credentials are a common thread for a while. Popular services like “Have I Been Pwned” help everyone know if some emails and passwords have been leaked. This is a classic problem: One day, you create an account on a website (ex: an online shop), and later, this website is compromised. All credentials are collected and shared by the attacker. To reduce this risk, a best practice is to avoid password re-use (as well as to not use your corporate email address for non-business-related stuff)… [Read more]

The post [SANS ISC] Are Leaked Credentials Dumps Used by Attackers? appeared first on /dev/random.

August 03, 2023

Beste Belgische militaire planner die planet.grep leest louter voor het plezier. Hier zijn de mogelijke scenarios voor uw Wereld Oorlog 3.

Allereerst, den component Belgischen landmacht:

  • De nek van het volledige Belgische leger is gebroken want het batalion dat U had gestationeerd in Oekraïne en dat onder uw commando was, werd verdampt door een nucleaire detonatie. Het Belgische leger heeft in het totaal zo’n vier strijdende batalions. Samen met alle éénheden van zo’n vier a vijf landen is één Belgisch batalion volledig verdwenen. Hun metalen tags zijn samen met hun lichamen verdampt in een nucleaire explosie. Er zijn nu nog drie batalions in het Belgische leger over. Twee zijn gestationeerd in België en één in het buitenland.
  • De twee batalions die nog over zijn in eigen land weigeren hun dienst nadat het eerste Batalion volledig verdampt en verdwenen is. Hoe ga jij als militaire planner hen toch overtuigen naar het front te gaan?
  • Het batalion in het buitenland deserteert massaal en van de 1000tal legeréénheden zijn er nog welgeteld 15 over die fanatiek genoeg zijn om uw commando toch te blijven uitvoeren.

Den component Belgischen luchtmacht:

  • Onze land’s-eigen patriotistische Belgische F-16 piloten zijn onder de indruk nadat een conventioneel bombardement op Kleine Brogel is gebeurd
  • Hun landingsbaan werd volledig vernield maar de N74 is zoals we weten geschikt gemaakt voor straaljagers om op te stijgen.
    • Beste Belgische leger. Ik moet ongeveer één maal per week op die weg tot in Eindhoven geraken. Kunnen jullie niet te veel in onze weg zitten? De show is leuk en zo. Maar het was handiger wanneer jullie dat in de lucht deden. Zolang Eindhoven en Veldhoven de aanval op Volkel overleven, moeten wij daar zijn. Als Rusland en China een paar ASML machines van Veldhoven krijgen dan zullen ze dat daar vast niet bombarderen.
  • Gelukkig heeft de vijand geen nucleaire wapens op België gebruikt. Maar op RT staat er vandaag een nieuwsartikel dat de Russische president dreigt om Kleine Brogel en de gehele N74 nucleair te bombarderen indien om het even welke F-16 of gelijk welk militair vliegtuig vanaf de militaire luchthaven of in de buurt van opstijgt.
  • Volkel is overigens wel reeds nucleair gebombardeerd. Want in tegenstelling tot de Belgische militaire incompetentie schatte Rusland Nederland als competent in. Dus daarom is de Nederlandse luchtcomponent alvast volledig uitgeschakeld.

Den Amerikaan eist dat we van alles en nog wat doen. Maar ons gans Belgischen leger is letterlijk op de vlucht geslagen.

  • Zoals te verwachten (en zoals in Oekraïne vandaag) doet den Amerikaan ook helemaal niks om ons te helpen. Dus dat was dat dan.

Tot dus ver uw (realistische) carriere als Belgisch militaire planner in een Wereld Oorlog 3 scenario.

Dé show is afgelopen de klus is hier geklaard. En weet ge wat den commendant tegen de soldaat (die nog leefde) zei: Ge moogt naar huis gaan, vaarwel goodbye (als ge nog leeft). Toch bedankt voor het leven, in deze brouwerij. Ga maar naar huis. En tot in den draai. (dat is als ge geluk hebt, en nog leeft)

ps. Hoewel het allemaal grappig klinkt, is het scenario heel erg realitisch. Men heeft hier toch een antwoord op? Want ik denk het eerlijk gezegd niet.

ps. Enkelen van jullie zullen denken: het Belgische leger is toch niet zo incompetent? Maar dan moet ik U het verhaal van Fort Ében-Émael vertellen. Waar enkele Duitse zweefvliegers de gehele Belgische defensie wisten te omzeilen, en zo’n 800 Belgische soldaten in hun ondergrondse tunnels vasthielden tot ze zichzelf overgaven (de incompetentie van het Belgische leger was veel erger dan je op dit moment denkt dat maar mogelijk is – kinderen jonger dan twaalf jaar hadden het beter kunnen beveiligen)

Het commando van het Belgisch leger is al zo’n eeuw niet veel beter dan de huidige leiding van de Chiro. Ik denk zelf dat de Chiroleiding beter is.

Men beweert vandaag de dag wel eens dat de Westerse wereld het beu aan het geraken is dat Oekraïne geen vooruitgang maakt in hun zogenaamde ‘counteroffensive’ (die wij militair op allerhande manieren steunen).

Maar toen we de planning begonnen om Oekraïne aan te zetten om lid te worden van NAVO, wisten we dat dit tot oorlog met Rusland zou leiden: dat is namelijk wat Rusland ons meermaals letterlijk heeft gezegd. Tal van Ruslandkenners zeggen dit nu eindelijk in onze Westerse media. Dat werd ook wel eens tijd. Men had dat beter gedaan al ergens in 2007 of 2008. Dat wisten we toen immers ook al. We wisten dat zelfs al decennia eerder.

We wisten dat die oorlog, in dat geval, in de achtertuin van Rusland zou zijn. We wisten ook hoe laf we zelf zouden zijn want we hadden het al eens meegemaakt tijdens het korte conflict tussen Rusland en Georgië. Het gehele Westen piste toen in haar broek en heeft letterlijk niets gedaan: Het was overal op onze TV hoe een Russische straaljager een Amerikaanse drone uit de lucht haalde. We deden daar totaal niets tegen. De Russische tanks reden Georgië binnen; daar hebben we ook totaal niets tegen gedaan. We pisten in ons broek. De broek stonk maanden later nog naar de angsturine. Vooral de broek van onze Westerse soldaten en hun legers: hadden zij even geluk zeg, dat ze niet tegen Rusland moesten gaan strijden. Dat had hun verzekerde dood geweest.

M.a.w. we wisten dat we laf zouden zijn. Dat was al eens bewezen. We waren al eens laf geweest. We weten dat we laf zijn, en zullen blijven. We wisten dat een conflict met Rusland in haar eigen achtertuin niet iets is waar we ons echt voor zouden inzetten. We zetten ons (gelukkig) niet echt in. Behalve dan dat we wat oud Westers afgeschreven oorlogsmateriaal opsturen.

Dat is maar goed ook: moesten we Westerse soldaten aan de grens met Rusland plaatsen, of bv. piloten: dan zullen die absoluut zeker en gegarandeerd vernietigd worden met Rusland’s nucleaire wapens. Dit zou vrijwel zeker Wereld Oorlog 3 starten. Met miljoenen slachtoffers als gevolg. Vooral de vaders en partners van de vrouwen en kinderen die een soldaat in het gezin hebben. Zij eerst.

Rusland heeft ook al aangekondigd dat Rheinmetall’s aangekondige fabriek in Oekraïne om de Duitse Leopards te herstellen, een target zal zijn. En zal vernietigd worden. Ga er maar van uit dat dat ook zo zal gebeuren. Meermaals. Opnieuw en opnieuw. Desnoods ook met nucleaire wapens (ik zeg dit er maar bij omdat nog steeds veel mensen niet door hebben hoe serieus dit voor Rusland is: het is zo serieus als dat).

Dus eigenlijk begrijp ik de schrik wel. Je wil dat helemaal niet riskeren tenzij het absoluut echt nodig is.

Rusland zal in dat geval, zonder aarzelen, alle gestationeerde Westerse soldaten letterlijk verdampen met enkele honderden nucleaire detonaties. Net zoals wij dat zouden doen indien zij Russische soldaten zouden stationeren aan onze Westerse grenzen. In geen enkel geval zullen beide kampen dat toelaten.

Dus waarom net waren we aan het dreigen tegen Rusland dat Oekraïne lid zou worden van de NAVO? We doen dat namelijk al sinds 2008 of zelfs eerder.

Was dit absoluut echt nodig?

We kunnen het helemaal niet waarmaken. Dat is totaal onmogelijk. Ook niet wanneer alle Instagrammers en TikTokkers gezamelijk HO roepen. Zelfs niet wanneer ze met miljoenen zijn. En zelfs niet wanneer onze gehele Westerse propaganda-machine hen vraagt om ‘HO’ te roepen. Zelfs niet wanneer ons totale Westerse oorlogsapparaat dat wil. Zelfs niet wanneer onze propaganda-mensen dat willen. Zelfs niet wanner onze leiders en onze elite dat willen.

Het kan eenvoudigweg niet. Helemaal niet. Totaal niet. Echt niet.

We wisten dat ook. We weten dat ook. Dus waarom slachtofferen we Oekraïne hiervoor?

Het spijt me, maar ik begrijp het niet.

ps. Hetzelfde zal gelden voor China en Taiwan. Maar dat zien we later weer wel zeker? Ook dat conflict zal de Westerse wereld verliezen. Omwille van dezelfde reden: de aanvoerlijn is voor China veel korter dan die voor de Westerse wereld (het is China’s achtertuin). Bovendien zal tegen dan onze invloed wereldwijd nihil zijn geworden. De dedollarizatie bijvoorbeeld is al volop bezig.

Dit gaat nl. dieper dan wat Vietnam was: de rest van de wereld is ons grondig beu. Als we niet willen luisteren, zullen we het voelen.

ps. Men kan aantonen dat ook Vietnam verloren werd door de VS omdat de aanvoerlijn langer was dan die was voor China en Rusland die Noord-Vietnam steunden.

ps. Ik denk ook aan onze soldaten. Moeten die massaal sneuvelen gewoon omdat wij een bende idioten zijn die niet meer kunnen nadenken over de geostrategische gevolgen van onze waanzin?

ps. Moralistisch doen is helaas naast de kwestie praten: het moraal of wie er gelijk heeft of zou moeten hebben doet er niet meer toe. Het gaat over geostrategie en militaire macht. Berg uw moralisme dus maar op. Het is irrelevant. Want militaire macht. Het geweer en de kogel legt het zwijgen op aan de moraliteit. Nadien wordt die toch herschreven..

The latest release of MySQL (July 18th, 2023) marks a new era for the most popular Open Source database, with the very first Innovation Release (8.1.0). More will follow before MySQL’s very first LTS Release sees the light of day.

This new Innovation Release already contains contributions from our great Community. MySQL 8.1.0 contains patches from Meta, Allen Long, Daniël van Eeden, Brent Gardner, Yura Sorokin (Percona) and Kento Takeuchi.

Let’s have a look at all these contributions:

MySQL Connector Python

  • #93643 – Contribution: fix recv compress bug – Allen Long
  • #110469 – Check for identifier quotes (backticks) in database name and add if necessary – Brent Gardner

Clients

  • #109972 – Contribution: client/mysql: enable comments by default – Daniël van Eeden
  • #110950 – Contribution: mysql client: more details about compression setting – Daniël van Eeden

C API

  • #110658 – mysql_binlog_xxx() symbols are not exported in lybmysqlclient.so – Yura Sorokin
  • #110879 – Compression doesn’t work with C extension API – Daniël van Eeden

MySQL Server

  • #110939 – Contribution: Protocol docs: zstd level is missing an if statement – Daniël van Eeden
  • #111190 – Build fails with LANG=ja_JP.UTF-8 – Kento Takeuchi
  • and several contributions from Meta (Facebook)

We can notice Daniël is again very active , that’s a great news !

If you have patches and you also want to be part of the MySQL Contributors, it’s easy, you can send Pull Requests from MySQL’s GitHub repositories or send your patches on Bugs MySQL (signing the Oracle Contributor Agreement is required).

Thanks again to all our contributors !

August 01, 2023

Splitting the Web

There’s an increasing chasm dividing the modern web. On one side, the commercial, monopolies-riddled, media-adored web. A web which has only one objective: making us click. It measures clicks, optimises clicks, generates clicks. It gathers as much information as it could about us and spams every second of our life with ads, beep, notifications, vibrations, blinking LEDs, background music and fluorescent titles.

A web which boils down to Idiocracy in a Blade Runner landscape, a complete cyberpunk dystopia.

Then there’s the tech-savvy web. People who install adblockers or alternative browsers. People who try alternative networks such as Mastodon or, God forbid, Gemini. People who poke fun at the modern web by building true HTML and JavaScript-less pages.

Between those two extremes, the gap is widening. You have to choose your camp. When browsing on the "normal web", it is increasingly required to disable at least part of your antifeatures-blockers to access content.

Most of the time, I don’t bother anymore. The link I clicked doesn’t open or is wrangled? Yep, I’m probably blocking some important third-party JavaScript. No, I don’t care. I’ve too much to read on a day anyway. More time for something else. I’m currently using kagi.com as my main search engine on the web. And kagi.com comes with a nice feature, a "non-commercial lens" (which is somewhat ironic given the fact that Kagi is, itself, a commercial search engine). It means it will try to deprioritize highly commercial contents. I can also deprioritize manually some domains. Like facebook.com or linkedin.com. If you post there, I’m less likely to read you. I’ve not even talked about the few times I use marginalia.nu.

Something strange is happening: it’s not only a part of the web which is disappearing for me. As I’m blocking completely google analytics, every Facebook domain and any analytics I can, I’m also disappearing for them. I don’t see them and they don’t see me!

Think about it! That whole "MBA, designers and marketers web" is now optimised thanks to analytics describing people who don’t block analytics (and bots pretending to be those people). Each day, I feel more disconnected from that part of the web.

When really needed, dealing with those websites is so nerve breaking that I often resort to… a phone call or a simple email. I signed my mobile phone contract by exchanging emails with a real person because the signup was not working. I phone to book hotels when it is not straightforward to do it in the web interface or if creating an account is required. I hate talking on the phone but it saves me a lot of time and stress. I also walk or cycle to stores instead of ordering online. Which allows me to get advice and to exchange defective items without dealing with the post office.

Despite breaking up with what seems to be "The Web", I’ve never received so many emails commenting my blog posts. I rarely had as many interesting online conversations as I have on Mastodon. I’ve tens of really insightful contents to read every day in my RSS feeds, on Gemini, on Hacker News, on Mastodon. And, incredibly, a lot of them are on very minimalists and usable blogs. The funny thing is that when non-tech users see my blog or those I’m reading, they spontaneously tell me how beautiful and usable they are. It’s a bit like all those layers of JavaScript and flashy css have been used against usability, against them. Against us. It’s a bit like real users never cared about "cool designs" and only wanted something simple.

It feels like everyone is now choosing its side. You can’t stay in the middle anymore. You are either dedicating all your CPU cycles to run JavaScript tracking you or walking away from the big monopolies. You are either being paid to build huge advertising billboards on top of yet another framework or you are handcrafting HTML.

Maybe the web is not dying. Maybe the web is only splitting itself in two.

You know that famous "dark web" that journalists crave to write about? (at my request, one journalist once told me what "dark web" meant to him and it was "websites not easily accessible through a Google search".) Well, sometimes I feel like I’m part of that "dark web". Not to buy drugs or hire hitmen. No! It’s only to have a place where I can have discussions without being spied and interrupted by ads.

But, increasingly, I feel less and less like an outsider.

It’s not me. It’s people living for and by advertising who are the outsiders. They are the one destroying everything they touch, including the planet. They are the sick psychos and I don’t want them in my life anymore. Are we splitting from those click-conversion-funnel-obsessed weirdos? Good riddance! Have fun with them.

But if you want to jump ship, now is the time to get back to the simple web. Welcome back aboard!

As a writer and an engineer, I like to explore how technology impacts society. You can subscribe by email or by rss. I value privacy and never share your adress.

If you read French, you can support me by buying/sharing/reading my books and subscribing to my newsletter in French or RSS. I also develop Free Software.

July 29, 2023

Today, I published the following diary on isc.sans.edu: “Do Attackers Pay More Attention to IPv6?“:

IPv6 has always been a hot topic! Available for years, many ISP’s deployed IPv6 up to their residential customers. In Belgium, we were for a long time, the top-one country with IPv6 deployment because all big players provided IPv6 connectivity. In today’s operating systems, IPv6 will be used first if your computer sees “RA” packets (for “router advertisement”) and can get an IPv6 address. This will be totally transparent. That’s why many people think that they don’t use IPv6 but they do… [Read more]

The post [SANS ISC] Do Attackers Pay More Attention to IPv6? appeared first on /dev/random.

July 28, 2023

Today, I published the following diary on isc.sans.edu: “ShellCode Hidden with Steganography“:

When hunting, I’m often surprised by the interesting pieces of code that you may discover… Attackers (or pentesters/redteamers) like to share scripts on VT to evaluate the detection rates against many antivirus products. Sometimes, you find something cool stuffs.

Yesterday, I found a small Python script that inject a shellcode into memory but, this time, the payload is hidden in a PNG picture using a well-known technique: steganography. The technique used in the sample, is to use the LSB (least significant bit) of each pixel with a bit of the payload. On the Internet, you can find a lot of free services to hide a text message into a picture (and vice-versa) but you can absolutely store any type of data, like in this case, executable code (the shellcode)… [Read more]

The post [SANS ISC] ShellCode Hidden with Steganography appeared first on /dev/random.

July 27, 2023

Bypassing paywalls is a sport for some. And it ain’t hard for Desktop Browsers. Just install a addon in your favorite Desktop Browser.

Unfortunately this didn’t work on a Android or iPhone phone. Nor on Sailfish OS with its Android emulation. Because over there browsers like Chrome and Chromium don’t allow extensions to be installed. Firefox does have some limited support for addons, but it can’t open local XPI files. Its addon menu doesn’t contain the addon and the addon website for it sees the running browser as incompatible.

Luckily you have Kiwi Browser, which is a Chrome based browser that did not disable extensions to be installed.

Once Kiwi is installed you can go to either chrome://extensions or kiwi://extensions, enable Developer mode and then open the zip file as explained in the Readme.md.

ps. For Sailfish I had to install an older version of Kiwi Browser, as the most recent version doesn’t seem to work.

July 26, 2023

The title of this article may seem trivial, but it represents the conclusion I reached after being astonished for a long time while debugging some issues related to decoding Bluetooth Low Energy (BLE) advertisements that rely on the device name. Like many solutions, the actual solution seemed simple in hindsight.

For Theengs Decoder I wrote decoders for two of Shelly's devices: the ShellyBLU Button1 and the ShellyBLU Door/Window sensor. Both devices broadcast BLE advertisements using the BTHome v2 format. Before we can decode these advertisements, we first need to identify the device that is sending a specific advertisement.

In Theengs Decoder, device detection is achieved using a model condition. The Button1 model condition is as follows:

"condition":["servicedata", "=", 14, "index", 0, "40", "|", "servicedata", "=", 14, "index", 0, "44", "&", "uuid", "index", 0, "fcd2", "&", "name", "index", 0, "SBBT-002C"],

This means: the service data (an array of bytes) consists of 14 hexadecimal characters and begins with the byte 0x40 (the first firmware versions) or 0x44 (the latest firmware). Additionally, the service data UUID is fcd2 and the name starts with SBBT-002C.

The Door/Window model condition is similar:

"condition":["servicedata", "=", 28, "index", 0, "44", "&", "uuid", "index", 0, "fcd2", "&", "name", "index", 0, "SBDW-002C"],

Here, the service data consists of 28 hexadecimal characters and starts with the byte 0x44. The service data UUID is fcd2 and the name starts with SBDW-002C.

Therefore, when decoding BLE advertisements, Theengs Decoder tries all model conditions for the supported devices. If any of the conditions is true, the properties defined in the associated decoder are extracted from the advertised data.

After writing these decoders, I tested them with Theengs Gateway on my Linux laptop. Theengs Gateway uses the cross-platform Python library Bleak, which on Linux uses BlueZ, for receiving BLE advertisements. It then decodes them using Theengs Decoder and publishes the decoded properties (in this case a button push or the state of the contact sensor) to an MQTT broker. The decoders worked.

It's important to know that Theengs Decoder purely works on BLE advertisements, which are broadcasted. This is the most basic way to communicate between BLE devices. All information is extracted from these advertisements: manufacturer-specific data, service data, and the local name. [1]

However, when I later experimented with additional functionality of both devices, I connected to them. A couple of days later, I noticed that the decoders no longer worked. When I looked into it, I saw that Theengs Gateway had detected the devices with different names: Shelly Blue Button 1 and Shelly BLE DW. Naturally, the previous model conditions no longer applied, because they were checking for the names SBBT-002C and SBDW-002C, respectively. Interestingly, the same version of Theengs Gateway running on a Raspberry Pi still managed to detect and decode both devices correctly, because it recognized them by their original names. I didn't understand this difference.

It took me a while before I remembered that I had previously connected to both devices on my laptop. So then I started exploring their GATT characteristics. Each BLE device that allows connections has a Generic Access Profile (GAP) service with mandatory Device Name and Appearance characteristics. I read the Device Name characteristic from both devices, and to my surprise they were identified as Shelly Blue Button 1 and Shelly BLE DW.

So that's when I started to connect the dots. Once I had connected to both devices from my laptop, BlueZ seemed to remember their names based on the Device Name characteristic, even after disconnecting. Apparently, when BlueZ later encounters advertisements from the same Bluetooth address, it doesn't use the advertised name (Complete Local Name or Shortened Local Name data type) to identify this device. Instead, it relies on the name it had previously stored from the Device Name characteristic. As a result, when Bleak requests the local name from BlueZ, it receives the device's Device Name characteristic instead of the advertised local name. Consequently, Theengs Gateway sent the 'wrong' name to Theengs Decoder, which caused the devices to remain undetected.

You can check this with bluetoothctl:

$ bluetoothctl
[bluetooth]# info 5C:C7:C1:XX:XX:XX
Device 5C:C7:C1:XX:XX:XX (public)
        Name: Shelly Blue Button 1
        Alias: Shelly Blue Button 1
        Appearance: 0x8001
        Paired: no
        Trusted: yes
        Blocked: no
        Connected: no
        LegacyPairing: no
        UUID: Generic Access Profile    (00001800-0000-1000-8000-00805f9b34fb)
        UUID: Generic Attribute Profile (00001801-0000-1000-8000-00805f9b34fb)
        UUID: Device Information        (0000180a-0000-1000-8000-00805f9b34fb)
        UUID: Vendor specific           (1d14d6ee-fd63-4fa1-bfa4-8f47b42119f0)
        UUID: Vendor specific           (de8a5aac-a99b-c315-0c80-60d4cbb51225)
        ManufacturerData Key: 0x0ba9
        ManufacturerData Value:
  01 09 00 0b 01 00 0a aa bb cc c1 c7 5c           ............\
        ServiceData Key: 0000fcd2-0000-1000-8000-00805f9b34fb
        ServiceData Value:
  40 00 19 01 5a 3a 01                             @...Z:.
        RSSI: -68
        AdvertisingFlags:
  06                                               .

Fortunately, once I had figured this out, the solution was quite simple: BlueZ needed to forget the devices. You can remove a device from BlueZ's cache with the following command:

$ bluetoothctl remove 3C:2E:F5:XX:XX:XX

Alternatively, you can manually delete the directory where BlueZ stores device information. First, find your Bluetooth adapter's address:

$ hciconfig
hci0: Type: Primary  Bus: USB
      BD Address: 9C:FC:E8:XX:XX:XX  ACL MTU: 1021:4  SCO MTU: 96:6
      UP RUNNING
      RX bytes:1203288 acl:31 sco:0 events:34298 errors:0
      TX bytes:898242 acl:29 sco:0 commands:11700 errors:0

Then delete the following directory, based on the adapter address and the device address:

$ sudo rm -rf /var/lib/bluetooth/9C\:FC\:E8\:XX\:XX\:XX/3C:2E:F5:XX:XX:XX

If you're curious about the cached information, first have a look at the culprit before removing the directory:

$ sudo cat /var/lib/bluetooth/9C\:FC\:E8\:XX\:XX\:XX/3C:2E:F5:XX:XX:XX/info
[General]
Name=Shelly BLE DW
AddressType=public
SupportedTechnologies=LE;
Trusted=true
Blocked=false
Services=00001800-0000-1000-8000-00805f9b34fb;00001801-0000-1000-8000-00805f9b34fb;0000180a-0000-1000-8000-00805f9b34fb;1d14d6ee-fd63-4fa1-bfa4-8f47b42119f0;de8a5aac-a99b-c315-0c80-60d4cbb51225;
Appearance=0x8001

It's this value of the Name field that BlueZ returns to Bleak, and subsequently to Theengs Gateway and Theengs Decoder, leading to the wrong detection. After removing the device from BlueZ's cache, the advertised local name was detected correctly, and the decoders worked again.

So now I'm wondering: is it possible to get the real local name advertised by a device? Bleak uses the Name property of a device in its BlueZ backend for scanning. Does BlueZ expose the local name, and could Bleak be adapted to use this instead? I couldn't find the answer to this question.

[1]

If you want to know more about BLE advertisements and connections, read the introductory chapter of my book about developing Bluetooth Low Energy applications.

July 23, 2023

The DebConf video team has been sprinting in preparation for DebConf 23 which will happen in Kochi, India, in September of this year.

Video team sprint

Present were Nicolas "olasd" Dandrimont, Stefano "tumbleweed" Rivera, and yours truly. Additionally, Louis-Philippe "pollo" Véronneau and Carl "CarlFK" Karsten joined the sprint remotely from across the pond.

Thank you to the DPL for agreeing to fund flights, food, and accomodation for the team members. We would also like to extend a special thanks to the Association April for hosting our sprint at their offices.

We made a lot of progress:

  • Now that Debian Bookworm has been released, we updated our ansible repository to work with Debian Bookworm. This encountered some issues, but nothing earth-shattering, and almost all of them are handled. The one thing that is still outstanding is that jibri requires OpenJDK 11, which is no longer in bookworm; a solution for that will need to be found in the longer term, but as jibri is only needed for online conferences, it is not quite as urgent (Stefano, Louis-Philippe).
  • In past years, we used open "opsis" hardware to do screen grabbing. While these work, upstream development has stalled, and their intended functionality is also somewhat more limited than we would like. As such, we experimented with a USB-based HDMI capture device, and after playing with it for some time, decided that it is a good option and that we would like to switch to it. Support for the specific capture device that we played with has now also been added to all the relevant places. (Stefano, Carl)
  • Another open tool that we have been using is voctomix, a software video mixer. Its upstream development has also stalled somewhat . While we managed to make it work correctly on Bookworm, we decided that to ensure long-term viability for the team, it would be preferable if we had an alternative. As such, we quickly investigated Nageru, Sesse's software video mixer, and decided that it can everything we need (and, probably, more). As such, we worked on implementing a user interface theme that would work with our specific requirements. Work on this is still ongoing, and we may decide that we are not ready yet for the switch by the time DebConf23 comes along, but we do believe that the switch is at least feasible. While working on the theme, we found a bug which Sesse quickly fixed for us after a short amount of remote debugging, so, thanks for that! (Stefano, Nicolas, Sesse)
  • Our current streaming architecture uses HLS, which requires MPEG-4-based codecs. While fully functional, MPEG-4 is not the most modern of codecs anymore, not to mention the fact that it is somewhat patent-encumbered (even though some of these patents are expired by now). As such, we investigated switching to the AV1 codec for live streaming. Our ansible repository has been updated to support live streaming using that codec; the post-event transcoding part will follow soon enough. Special thanks, again, to Sesse, for pointing out a few months ago on Planet Debian that this is, in fact, possible to do. (Wouter)
  • Apart from these big-ticket items, we also worked on various small maintenance things: upgrading, fixing, and reinstalling hosts and services, filing budget requests, and requesting role emails. (all of the team, really).

It is now Sunday the 23rd at 14:15, and while the sprint is coming to an end, we haven't quite finished yet, so some more progress can still be made. Let's see what happens by tonight.

All in all, though, we believe that the progress we made will make the DebConf Videoteam's work a bit easier in some areas, and will make things work better in the future.

See you in Kochi!

Pour une poignée de followers

Pour une raison que j’ignore, mon compteur d’abonnés sur Mastodon s’est emballé et vient de franchir le cap de 6700. Ce chiffre porte une petite symbolique pour moi, car je ne pense pas l’avoir jamais franchi sur Twitter.

Si mes souvenirs sont bons, j’ai quitté Twitter avec environ 6600 abonnés, Google+ avec 3000 abonnés, Facebook avec 2500, LinkedIn et Medium avec 1500. Mastodon serait donc le réseau où j’ai historiquement le plus de succès (si l’on excepte l’éphémère compte Twitter du « Blog d’un condamné » qui avait attiré plus de 9000 personnes en quelques jours).

Faut-il être heureux que mon compte Mastodon fasse mieux en six ans que mon compte Twitter entre 2007 et 2021, date de sa suppression définitive ?

Où peut-être est-ce l’occasion de rappeler que, tout comme le like, dont j’ai précédemment détaillé l’inanité, le nombre de followers est une métrique absurde. Fausse. Et qui devrait être cachée.

Où l’on sépare les comptes qui comptent de ceux qui ne comptent pas

Les réseaux sociaux commerciaux vous vendent littéralement l’impression d’être suivis. Il n’y a aucun incitant à offrir un compte correct. Au contraire, tout est fait pour exagérer, gonfler.

Vos followers sont donc composés de comptes de robots, de comptes de sociétés qui suivent, mais ne lisent de toute façon pas les contenus, de comptes générés automatiquement et de toute cette panoplie de comptes inactifs, car la personne est passée à autre chose, a oublié son mot de passe ou, tout simplement, est décédée.

Sur Mastodon, mon intuition me dit que c’est « moins pire » grâce à la jeunesse du réseau. J’y ai déjà néanmoins vu des comptes de robots, des comptes de personnes qui ont testé et n’utilisent plus Mastodon ainsi que des comptes doublons, la personne ayant plusieurs comptes et me suivant sur chacun.

Au final, il y’a beaucoup moins d’humains que le compteur ne veut bien nous le laisser croire.

Où l’on se pose la question de l’utilité de tout cela

Mais même lorsqu’un compte représente un humain réel, un humain intéressé par ce que vous postez, encore faut-il qu’il vous lise lorsque votre contenu est noyé dans les 100, 200 ou 1000 autres comptes qu’il suit. Ou, tout simplement, n’est-il pas sur les réseaux sociaux ce jour-là ? Peut-être vous a-t-il vu et lu, entre deux autres messages.

Et alors ?

Je répète en anglais parce que ça donne un style plus théâtral.

So what ?

So feukinne watte ?

Vous êtes-vous déjà demandé à quoi pouvaient bien servir les followers ?

Tous ces autocollants vous invitant à suivre sur Facebook et Instagram la page de votre fleuriste, de votre plombier ou de votre boulangerie ? Sérieusement, qui s’est un jour dit en voyant un de ces autocollants « Cool, je vais suivre mon fleuriste, mon plombier et ma boulangère sur Facebook et Instagram » ?

Et quand bien même certains le font, certainement tonton Albert et cousine Géraldine qui n’habitent pas la ville, mais soutiennent la boulangère de la famille, pensez-vous que ça ait le moindre impact sur le business ?

À l’opposé, je suis avec assiduité une centaine de blogs par RSS. Je lis tout ce que ces personnes écrivent. Je réagis par mail. Je les partage en privé. J’achète également tous les livres de certains de mes auteurs favoris. Pourtant, je ne suis compté nulle part comme un follower.

Où l’on a la réponse à la question précédente

Militant pour le logiciel libre, le respect de la vie privée et le web non commercial, on pourrait arguer que mon public se trouve, par essence, sur Mastodon. (et me demander pourquoi je suis resté si longtemps sur les réseaux propriétaires. Je n’ai en effet aucune excuse).

Prenons un cas différent.

L’écrivain Henri Lœvenbruck a fermé ses comptes Facebook (29.000 followers), Twitter (10.000 followers) et Instagram (8.000 followers). Son dernier livre, « Les disparus de Blackmore », promu uniquement auprès des 5000 comptes qui le suivent sur Mastodon (et un peu LinkedIn, mais qu’est-ce qu’il fout encore là-bas ?) s’est pourtant beaucoup mieux vendu que le précédent.

Faut-il en déduire que les followers ne sont pas la recette miracle tant louée par… les sociétés publicitaires dont le business model repose à vouloir nous faire avoir à tout prix des followers ? D’ailleurs, entre nous, préférez-vous passer quelques heures à vous engueuler sur Twitter ou à flâner dans un univers typiquement Lœvenbruckien ? (Mystères lovercraftiens, grosses motos qui pétaradent, vieux whiskies qui se dégustent et quelques francs-maçons pour la figuration, on sent que l’auteur de « Nous rêvions juste de liberté » s’est fait plaisir, plaisir partagé avec les lecteurs et après on s’étonne que le bouquin se vende)

Si Lœvenbruck a pris un risque dans sa carrière pour des raisons éthiques et morales, force est de constater que le risque n’en était finalement pas un. Ses comptes Facebook/Instagram/Twitter ne vendaient pas de livres. Ce serait plutôt même le contraire.

Dans son livre "Digital Minimalism" et sur son blog, l’auteur Cal Newport s’est fait une spécialité d’illustrer le fait que beaucoup de succès modernes, qu’ils soient artistiques, entrepreneuriaux ou sportifs, se construisent non pas avec les réseaux sociaux, mais en arrivant à les mettre de côté. Une réflexion que j’ai moi-même esquissée alors que je tentais de me déconnecter.

La conclusion de tout cela est effrayante : nous nous sommes fait complètement avoir. Vraiment. La quête de followers est une arnaque totale qui, loin de nous apporter des bénéfices, nous coûte du temps, de l’énergie mentale, parfois de l’argent voire, dans certains cas, détruit notre business ou notre œuvre en nous forçant à modifier nos produits, nos créations pour attirer des followers.

Où l’on se rend compte des méfaits d’un simple chiffre

Car, pour certains créateurs, le nombre de followers est devenu une telle obsession qu’elle emprisonne. J’ai eu des discussions avec plusieurs personnes très influentes sur Twitter en leur demandant si elles comptaient ouvrir un compte sur Mastodon. Dans la plupart des cas, la réponse a été qu’elles restaient sur Twitter pour garder « leur communauté ». Leur "communauté" ? Quel bel euphémisme pour nommer un chiffre artificiellement gonflé qui les rend littéralement prisonnières. Et peut-être est-ce même une opportunité manquée.

Car un réseau n’est pas l’autre. Le bien connu blogueur-à-la-retraite-fourgeur-de-liens Sebsauvage a 4000 abonnés sur Twitter. Mais plus de 13000 sur Mastodon.

Est-ce que cela veut dire quelque chose ? Je ne le sais pas moi-même. Je rêve d’un Mastodon où le nombre de followers serait caché. Même de moi-même. Surtout de moi-même.

Avant de transformer nos lecteurs en numéros, peut-être est-il bon de se rappeler que nous sommes nous-mêmes des numéros. Que le simple fait d’avoir un compte Twitter ou Facebook, même non utilisé, permet d’augmenter de quelques dollars chaque année la fortune d’un Elon Musk ou d’un Mark Zuckerberg.

En ayant un compte sur une plateforme, nous la validons implicitement. Avoir un compte sur toutes les plateformes, comme Cory Doctorrow, revient à un vote nul. À dire « Moi je ne préfère rien, je m’adapte ».

Si nous voulons défendre certaines valeurs, la moindre des choses n’est-elle pas de ne pas soutenir les promoteurs des valeurs adverses ? De supprimer les comptes des plateformes avec lesquelles nous ne sommes pas moralement alignés ? Si nous ne sommes même pas capables de ce petit geste, avons-nous le moindre espoir de mettre en œuvre des causes plus importantes comme sauver la planète ?

Où l’on relativise et relativise la relativisation

Encore faut-il avoir le choix. Je discutais récemment avec un indépendant qui me disait que, dans son business, les clients envoient un message Whatsapp pour lui proposer une mission. S’il met plus de quelques dizaines de minutes à répondre, il reçoit généralement un « c’est bon, on a trouvé quelqu’un d’autre ». Il est donc obligé d’être sur Whatsapp en permanence. C’est peut-être vrai pour certaines professions et certains réseaux sociaux.

Mais combien se persuadent que LinkedIn, Facebook ou Instagram sont indispensables à leur business ? Qu’ils ne peuvent quitter Twitter sous peine de mettre à mal leur procrastin… leur veille technologique ?

Combien d’entre nous ne font que se donner des excuses, des justifications par simple angoisse d’avoir un jour à renoncer à ce chiffre qui scintille, qui augment lentement, trop lentement, mais assez pour que l’on ait envie de le consulter tous les jours, toutes les heures, toutes les minutes.

Que sommes-nous prêts à sacrifier de notre temps, de nos valeurs, de notre créativité simplement pour l’admirer ?

Notre nombre de followers.

Ingénieur et écrivain, j’explore l’impact des technologies sur l’humain. Abonnez-vous à mes écrits en français par mail ou par rss. Pour mes écrits en anglais, abonnez-vous à la newsletter anglophone ou au flux RSS complet. Votre adresse n’est jamais partagée et effacée au désabonnement.

Pour me soutenir, achetez mes livres (si possible chez votre libraire) ! Je viens justement de publier un recueil de nouvelles qui devrait vous faire rire et réfléchir.

July 21, 2023

I use the lightweight Kubernetes K3s on a 3-node Raspberry Pi 4 cluster.

And created a few ansible to provision the virtual machines with cloud image with cloud-init and deploy k3s on it.

I updated the roles below to be compatible with the latest Debian release: Debian 12 bookworm.

I created a movie to demonstrate how you can setup a kubernetes homelab in few minutes.

Deploy k3s on vms

The latest version 1.1.0 is available at: https://github.com/stafwag/ansible-k3s-on-vms


Have fun!

cloud_localds 2.1.2

stafwag.cloud_localds 2.1.2 is available at: https://github.com/stafwag/ansible-role-cloud_localds

Changelog

  • Generate netconfig when network config template is used
    • bugfix: Generate network config when network_config_template is used
      • Generate network config when network_config_template is used
      • Align on double quote style
    • docs/examples added


virt_install_vm 1.1.0

stafwag.virt_install_vm 1.1.0 is available at: https://github.com/stafwag/ansible-role-virt_install_vm

Changelog

  • Debian GNU/Linux 12 templates

  • Debian GNU/Linux cloud-init v2 templates added
  • Use sudo group in the Debian 12 template
    • use the sudo group in the Debian 12 templates; this is more inline how Debian systems are configured.
  • docs/examples added
  • Documentation updated


delegated_vm_install 2.0.0

stafwag.delegated_vm_install 2.0.0 is available at: https://github.com/stafwag/ansible-role-delegated_vm_install

Changelog

  • Debian GNU/Linux 12 template Latest

    • Introduced templates/vms//path
    • mv Debian 11 template to templates/vms/debian/11
    • Created Debian 12 vm template
    • Set default to Debian 12
    • Documentation updated
    • This release might break existing playbooks; bumped version to 2.0.0

July 16, 2023

High quality font rendering for WebGPU

Cover Image - Live effect run-time inspector

This page includes diagrams in WebGPU, which has limited browser support. For the full experience, use Chrome on Windows or Mac, or a developer build on other platforms.

In this post I will describe Use.GPU's text rendering, which uses a bespoke approach to Signed Distance Fields (SDFs). This was borne out of necessity: while SDF text is pretty common on GPUs, some of the established practice on generating SDFs from masks is incorrect, and some libraries get it right only by accident. So this will be a deep dive from first principles, about the nuances of subpixels.

Sample of Use.GPU text

SDFs

The idea behind SDFs is quite simple. To draw a crisp, anti-aliased shape at any size, you start from a field or image that records the distance to the shape's edge at every point, as a gradient. Lighter grays are inside, darker grays are outside. This can be a lower resolution than the target.

SDF for @ character

Then you increase the contrast until the gradient is exactly 1 pixel wide at the target size. You can sample it to get a perfectly anti-aliased opacity mask:

This works fine for text at typical sizes, and handles fractional shifts and scales perfectly with zero shimmering. It's also reasonably correct from a signal processing math point-of-view: it closely approximates averaging over a pixel-sized circular window, i.e. a low-pass convolution.

Crucially, it takes a rendered glyph as input, which means I can remain blissfully unaware of TrueType font specifics, and bezier rasterization, and just offload that to an existing library.

To generate an SDF, I started with MapBox's TinySDF library. Except, what comes out of it is wrong:

SDF for @ character

The contours are noticeably wobbly and pixelated. The only reason the glyph itself looks okay is because the errors around the zero-level are symmetrical and cancel out. If you try to dilate or contract the outline, which is supposed to be one of SDF's killer features, you get ugly gunk.

Compare to:

The original Valve paper glosses over this aspect and uses high resolution inputs (4k) for a highly downscaled result (64). That is not an option for me because it's too slow. But I did get it to work. As a result Use.GPU has a novel subpixel-accurate distance transform (ESDT), which even does emoji. It's a combination CPU/GPU approach, with the CPU generating SDFs and the GPU rendering them, including all the debug viz.

The Classic EDT

The common solution is a Euclidean Distance Transform. Given a binary mask, it will produce an unsigned distance field. This holds the squared distance for either the inside or outside area, which you can sqrt.

EDT X pass
EDT Y pass

Like a Fourier Transform, you can apply it to 2D images by applying it horizontally on each row X, then vertically on each column Y (or vice versa). To make a signed distance field, you do this for both the inside and outside separately, and then combine the two as inside – outside or vice versa.

The algorithm is one of those clever bits of 80s-style C code which is O(N), has lots of 1-letter variable names, and is very CPU cache friendly. Often copy/pasted, but rarely understood. In TypeScript it looks like this, where array is modified in-place and f, v and z are temporary buffers up to 1 row/column long. The arguments offset and stride allow the code to be used in either the X or Y direction in a flattened 2D array.

for (let q = 1, k = 0, s = 0; q < length; q++) {
  f[q] = array[offset + q * stride];

  do {
    let r = v[k];
    s = (f[q] - f[r] + q * q - r * r) / (q - r) / 2;
  } while (s <= z[k] && --k > -1);

  k++;
  v[k] = q;
  z[k] = s;
  z[k + 1] = INF;
}

for (let q = 0, k = 0; q < length; q++) {
  while (z[k + 1] < q) k++;
  let r = v[k];
  let d = q - r;
  array[offset + q * stride] = f[r] + d * d;
}

To explain what this code does, let's start with a naive version instead.

row of black and white pixels

Given a 1D input array of zeroes (filled), with an area masked out with infinity (empty):

O = [·, ·, ·, 0, 0, 0, 0, 0, ·, 0, 0, 0, ·, ·, ·]

Make a matching sequence … 3 2 1 0 1 2 3 … for each element, centering the 0 at each index:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9,10,11,12,13,14] + ∞
[1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,10,11,12,13] + ∞
[2, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,10,11,12] + ∞
[3, 2, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,10,11] + 0
[4, 3, 2, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,10] + 0
[5, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9] + 0
[6, 5, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8] + 0
[7, 6, 5, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5, 6, 7] + 0
[8, 7, 6, 5, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5, 6] + ∞
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5] + 0
[10,9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 1, 2, 3, 4] + 0
[11,10,9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 1, 2, 3] + 0
[12,11,10,9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 1, 2] + ∞
[13,12,11,10,9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 1] + ∞
[14,13,12,11,10,9, 8, 7, 6, 5, 4, 3, 2, 1, 0] + ∞

You then add the value from the array to each element in the row:

[∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞]
[∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞]
[∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞]
[3, 2, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,10,11]
[4, 3, 2, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9,10]
[5, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
[6, 5, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5, 6, 7, 8]
[7, 6, 5, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5, 6, 7]
[∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞]
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 1, 2, 3, 4, 5]
[10,9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 1, 2, 3, 4]
[11,10,9, 8, 7, 6, 5, 4, 3, 2, 1, 0, 1, 2, 3]
[∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞]
[∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞]
[∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞, ∞]

And then take the minimum of each column:

P = [3, 2, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 2, 3]

This sequence counts up inside the masked out area, away from the zeroes. This is the positive distance field P.

You can do the same for the inverted mask:

I = [0, 0, 0, ·, ·, ·, ·, ·, 0, ·, ·, ·, 0, 0, 0]

to get the complementary area, i.e. the negative distance field N:

N = [0, 0, 0, 1, 2, 3, 2, 1, 0, 1, 2, 1, 0, 0, 0]

That's what the EDT does, except it uses square distance … 9 4 1 0 1 4 9 …:

Countour of parabolas

When you apply it a second time in the second dimension, these outputs are the new input, i.e. values other than 0 or . It still works because of Pythagoras' rule: d² = x² + y². This wouldn't be true if it used linear distance instead. The net effect is that you end up intersecting a series of parabolas, somewhat like a 1D slice of a Voronoi diagram:

I' = [0, 0, 1, 4, 9, 4, 4, 4, 1, 1, 4, 9, 4, 9, 9]
Countour of parabolas in second pass

Each parabola sitting above zero is the 'shadow' of a zero-level paraboloid located some distance in a perpendicular dimension:

The code is just a more clever way to do that, without generating the entire grid per row/column. It instead scans through the array left to right, building up a list v[k] of significant minima, with thresholds s[k] where two parabolas intersect. It adds them as candidates (k++) and discards them (--k) if they are eclipsed by a newer value. This is the first for/while loop:

for (let q = 1, k = 0, s = 0; q < length; q++) {
  f[q] = array[offset + q * stride];

  do {
    let r = v[k];
    s = (f[q] - f[r] + q * q - r * r) / (q - r) / 2;
  } while (s <= z[k] && --k > -1);

  k++;
  v[k] = q;
  z[k] = s;
  z[k + 1] = INF;
}

Then it goes left to right again (for), and fills out the values, skipping ahead to the right minimum (k++). This is the squared distance from the current index q to the nearest minimum r, plus the minimum's value f[r] itself. The paper has more details:

for (let q = 0, k = 0; q < length; q++) {
  while (z[k + 1] < q) k++;
  let r = v[k];
  let d = q - r;
  array[offset + q * stride] = f[r] + d * d;
}

The Broken EDT

So what's the catch? The above assumes a binary mask.

row of black and white pixels

As it happens, if you try to subtract a binary N from P, you have an off-by-one error:

    O = [·, ·, ·, 0, 0, 0, 0, 0, ·, 0, 0, 0, ·, ·, ·]
    I = [0, 0, 0, ·, ·, ·, ·, ·, 0, ·, ·, ·, 0, 0, 0]

    P = [3, 2, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 2, 3]
    N = [0, 0, 0, 1, 2, 3, 2, 1, 0, 1, 2, 1, 0, 0, 0]

P - N = [3, 2, 1,-1,-2,-3,-2,-1, 1,-1,-2,-1, 1, 2, 3]

It goes directly from 1 to -1 and back. You could add +/- 0.5 to fix that.

row of anti-aliased pixels

But if there is a gray pixel in between each white and black, which we classify as both inside (0) and outside (0), it seems to work out just fine:

    O = [·, ·, ·, 0, 0, 0, 0, 0, ·, 0, 0, 0, ·, ·, ·]
    I = [0, 0, 0, 0, ·, ·, ·, 0, 0, 0, ·, 0, 0, 0, 0]

    P = [3, 2, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 2, 3]
    N = [0, 0, 0, 0, 1, 2, 1, 0, 0, 0, 1, 0, 0, 0, 0]

P - N = [3, 2, 1, 0,-1,-2,-1, 0, 1, 0,-1, 0, 1, 2, 3]

This is a realization that somebody must've had, and they reasoned on: "The above is correct for a 50% opaque pixel, where the edge between inside and outside falls exactly in the middle of a pixel."

"Lighter grays are more inside, and darker grays are more outside. So all we need to do is treat l = level - 0.5 as a signed distance, and use for the initial inside or outside value for gray pixels. This will cause either the positive or negative distance field to shift by a subpixel amount l. And then the EDT will propagate this in both X and Y directions."

The initial idea is correct, because this is just running SDF rendering in reverse. A gray pixel in an opacity mask is what you get when you contrast adjust an SDF and do not blow it out into pure black or white. The information inside the gray pixels is "correct", up to rounding.

But there are two mistakes here.

The first is that even in an anti-aliased image, you can have white pixels right next to black ones. Especially with fonts, which are pixel-hinted. So the SDF is wrong there, because it goes directly from -1 to 1. This causes the contours to double up, e.g. around this bottom edge:

Doubled up contour in EDT due to bad edge handling

To solve this, you can eliminate the crisp case by deliberately making those edges very dark or light gray.

But the second mistake is more subtle. The EDT works in 2D because you can feed the output of X in as the input of Y. But that means that any non-zero input to X represents another dimension Z, separate from X and Y. The resulting squared distance will be x² + y² + z². This is a 3D distance, not 2D.

If an edge is shifted by 0.5 pixels in X, you would expect a 1D SDF like:

  […, 0.5, 1.5, 2.5, 3.5, …]
= […, 0.5, 1 + 0.5, 2 + 0.5, 3 + 0.5, …]

Instead, because of the squaring + square root, you will get:

  […, 0.5, 1.12, 2.06, 3.04, …]
= […, sqrt(0.25), sqrt(1 + 0.25), sqrt(4 + 0.25), sqrt(9 + 0.25), …]

The effect of l² = 0.25 rapidly diminishes as you get away from the edge, and is significantly wrong even just one pixel over.

The correct shift would need to be folded into (x + …)² + (y + …)² and depends on the direction. e.g. If an edge is shifted horizontally, it ought to be (x + l)² + y², which means there is a term of 2*x*l missing. If the shift is vertical, it's 2*y*l instead. This is also a signed value, not positive/unsigned.

Given all this, it's a miracle this worked at all. The only reason this isn't more visible in the final glyph is because the positive and negative fields contains the same but opposite errors around their respective gray pixels.

The Not-Subpixel EDT

As mentioned before, the EDT algorithm is essentially making a 1D Voronoi diagram every time. It finds the distance to the nearest minimum for every array index. But there is no reason for those minima themselves to lie at integer offsets, because the second for loop effectively resamples the data.

So you can take an input mask, and tag each index with a horizontal offset Δ:

O = [·, ·, ·, 0, 0, 0, 0, 0, ·, ·, ·]
Δ = [A, B, C, D, E, F, G, H, I, J, K]

As long as the offsets are small, no two indices will swap order, and the code still works. You then build the Voronoi diagram out of the shifted parabolas, but sample the result at unshifted indices.

Problem 1 - Opposing Shifts

This lead me down the first rabbit hole, which was an attempt to make the EDT subpixel capable without losing its appealing simplicity. I started by investigating the nuances of subpixel EDT in 1D. This was a bit of a mirage, because most real problems only pop up in 2D. Though there was one important insight here.

O = [·, ·, ·, 0, 0, 0, 0, 0, ·, ·, ·]
Δ = [·, ·, ·, A, ·, ·, ·, B, ·, ·, ·]

Given a mask of zeroes and infinities, you can only shift the first and last point of each segment. Infinities don't do anything, while middle zeroes should remain zero.

Using an offset A works sort of as expected: this will increase or decrease the values filled in by a fractional pixel, calculating a squared distance (d + A)² where A can be positive or negative. But the value at the shifted index itself is always (0 + A)² (positive). This means it is always outside, regardless of whether it is moving left or right.

If A is moving left (–), the point is inside, and the (unsigned) distance should be 0. At B the situation is reversed: the distance should be 0 if B is moving right (+). It might seem like this is annoying but fixable, because the zeroes can be filled in by the opposite signed field. But this is only looking at the binary 1D case, where there are only zeroes and infinities.

In 2D, a second pass has non-zero distances, so every index can be shifted:

O = [a, b, c, d, e, f, g, h, i, j, k]
Δ = [A, B, C, D, E, F, G, H, I, J, K]

Now, resolving every subpixel unambiguously is harder than you might think:

It's important to notice that the function being sampled by an EDT is not actually smooth: it is the minimum of a discrete set of parabolas, which cross at an angle. The square root of the output only produces a smooth linear gradient because it samples each parabola at integer offsets. Each center only shifts upward by the square of an integer in every pass, so the crossings are predictable. You never sample the 'wrong' side of (d + ...)². A subpixel EDT does not have this luxury.

Subpixel EDTs are not irreparably broken though. Rather, they are only valid if they cause the unsigned distance field to increase, i.e. if they dilate the empty space. This is a problem: any shift that dilates the positive field contracts the negative, and vice versa.

To fix this, you need to get out of the handwaving stage and actually understand P and N as continuous 2D fields.

Problem 2 - Diagonals

Consider an aliased, sloped edge. To understand how the classic EDT resolves it, we can turn it into a voronoi diagram for all the white pixel centers:

Voronoi diagram for slope

Near the bottom, the field is dominated by the white pixels on the corners: they form diagonal sections downwards. Near the edge itself, the field runs perfectly vertical inside a roughly triangular section. In both cases, an arrow pointing back towards the cell center is only vaguely perpendicular to the true diagonal edge.

Voronoi diagram for diagonal slope

Near perfect diagonals, the edge distances are just wrong. The distance of edge pixels goes up or right (1), rather than the more logical diagonal 0.707…. The true closest point on the edge is not part of the grid.

These fields don't really resolve properly until 6-7 pixels out. You could hide these flaws with e.g. an 8x downscale, but that's 64x more pixels. Either way, you shouldn't expect perfect numerical accuracy from an EDT. Just because it's mathematically separable doesn't mean it's particularly good.

In fact, it's only separable because it isn't very good at all.

Problem 3 - Gradients

In 2D, there is also only one correct answer to the gray case. Consider a diagonal edge, anti-aliased:

anti-aliased slope

Thresholding it into black, grey or white, you get:

Voronoi diagram for slope - thresholded

If you now classify the grays as both inside and outside, then the highlighted pixels will be part of both masks. Both the positive and negative field will be exactly zero there, and so will the SDF (P - N):

This creates a phantom vertical edge that pushes apart P and N, and causes the average slope to be less than 45º. The field simply has the wrong shape, because gray pixels can be surrounded by other gray pixels.

This also explains why TinySDF magically seemed to work despite being so pixelized. The gray correction fills in exactly the gaps in the bad (P - N) field where it is zero, and it interpolates towards a symmetrically wrong P and N field on each side.

If we instead classify grays as neither inside nor outside, then P and N overlap in the boundary, and it is possible to resolve them into a coherent SDF with a clean 45 degree slope, if you do it right:

What seemed like an off-by-one error is actually the right approach in 2D or higher. The subpixel SDF will then be a modified version of this field, where the P and N sides are changed in lock-step to remain mutually consistent.

Though we will get there in a roundabout way.

Problem 4 - Commuting

It's worth pointing out: a subpixel EDT simply cannot commute in 2D.

First, consider the data flow of an ordinary EDT:

Voronoi diagram for commute EDT

Information from a corner pixel can flow through empty space both when doing X-then-Y and Y-then-X. But information from the horizontal edge pixels can only flow vertically then horizontally. This is okay because the separating lines between adjacent pixels are purely vertical too: the red arrows never 'win'.

But if you introduce subpixel shifts, the separating lines can turn:

Voronoi diagram for commute ESDT

The data flow is still limited to the original EDT pattern, so the edge pixels at the top can only propagate by starting downward. They can only influence adjacent columns if the order is Y-then-X. For vertical edges it's the opposite.

That said, this is only a problem on shallow concave curves, where there aren't any corner pixels nearby. The error is that it 'snaps' to the wrong edge point, but only when it is already several pixels away from the edge. In that case, the smaller term is dwarfed by the much larger term, so the absolute error is small after sqrt.

The ESDT

Knowing all this, here's how I assembled a "true" Euclidean Subpixel Distance Transform.

Subpixel offsets

To start we need to determine the subpixel offsets. We can still treat level - 0.5 as the signed distance for any gray pixel, and ignore all white and black for now.

The tricky part is determining the exact direction of that distance. As an approximation, we can examine the 3x3 neighborhood around each gray pixel and do a least-squares fit of a plane. As long as there is at least one white and one black pixel in this neighborhood, we get a vector pointing towards where the actual edge is. In practice I apply some horizontal/vertical smoothing here using a simple [1 2 1] kernel.

The result is numerically very stable, because the originally rasterized image is visually consistent.

This logic is disabled for thin creases and spikes, where it doesn't work. Such points are treated as fully masked out, so that neighboring distances propagate there instead. This is needed e.g. for the pointy negative space of a W to come out right.

I also implemented a relaxation step that will smooth neighboring vectors if they point in similar directions. However, the effect is quite minimal, and it rounds very sharp corners, so I ended up disabling it by default.

The goal is then to do an ESDT that uses these shifted positions for the minima, to get a subpixel accurate distance field.

P and N junction

We saw earlier that only non-masked pixels can have offsets that influence the output (#1). We only have offsets for gray pixels, yet we concluded that gray pixels should be masked out, to form a connected SDF with the right shape (#3). This can't work.

SDFs are both the problem and the solution here. Dilating and contracting SDFs is easy: add or subtract a constant. So you can expand both P and N fields ahead of time geometrically, and then undo it numerically. This is done by pushing their respective gray pixel centers in opposite directions, by half a pixel, on top of the originally calculated offset:

This way, they can remain masked in in both fields, but are always pushed between 0 and 1 pixel inwards. The distance between the P and N gray pixel offsets is always exactly 1, so the non-zero overlap between P and N is guaranteed to be exactly 1 pixel wide everywhere. It's a perfect join anywhere we sample it, because the line between the two ends crosses through a pixel center.

When we then calculate the final SDF, we do the opposite, shifting each by half a pixel and trimming it off with a max:

SDF = max(0, P - 0.5) - max(0, N - 0.5)

Only one of P or N will be > 0.5 at a time, so this is exact.

To deal with pure black/white edges, I treat any black neighbor of a white pixel (horizontal or vertical only) as gray with a 0.5 pixel offset (before P/N dilation). No actual blurring needs to happen, and the result is numerically exact minus epsilon, which is nice.

ESDT state

The state for the ESDT then consists of remembering a signed X and Y offset for every pixel, rather than the squared distance. These are factored into the distance and threshold calculations, separated into its proper parallel and orthogonal components, i.e. X/Y or Y/X. Unlike an EDT, each X or Y pass has to be aware of both axes. But the algorithm is mostly unchanged otherwise, here X-then-Y.

The X pass:

At the start, only gray pixels have offsets and they are all in the range -1…1 (exclusive). With each pass of ESDT, a winning minima's offsets propagate to its affecting range, tracking the total distance (Δx, Δy) (> 1). At the end, each pixel's offset points to the nearest edge, so the squared distance can be derived as Δx² + Δy².

The Y pass:

You can see that the vertical distances in the top-left are practically vertical, and not oriented perpendicular to the contour on average: they have not had a chance to propagate horizontally. But they do factor in the vertical subpixel offset, and this is the dominant component. So even without correction it still creates a smooth SDF with a surprisingly small error.

Fix ups

The commutativity errors are all biased positively, meaning we get an upper bound of the true distance field.

You could take the min of X then Y and Y then X. This would re-use all the same prep and would restore rotation-independence at the cost of 2x as many ESDTs. You could try X then Y then X at 1.5x cost with some hacks. But neither would improve diagonal areas, which were still busted in the original EDT.

Instead I implemented an additional relaxation pass. It visits every pixel's target, and double checks whether one of the 4 immediate neighbors (with subpixel offset) isn't a better solution:

It's a good heuristic because if the target is >1px off there is either a viable commutative propagation path, or you're so far away the error is negligible. It fixes up the diagonals, creating tidy lines when the resolution allows for it:

You could take this even further, given that you know the offsets are supposed to be perpendicular to the glyph contour. You could add reprojection with a few dot products here, but getting it to not misfire on edge cases would be tricky.

While you can tell the unrelaxed offsets are wrong when visualized, and the fixed ones are better, the visual difference in the output glyphs is tiny. You need to blow glyphs up to enormous size to see the difference side by side. So it too is disabled by default. The diagonals in the original EDT were wrong too and you could barely tell.

Emoji

An emoji is generally stored as a full color transparent PNG or SVG. The ESDT can be applied directly to its opacity mask to get an SDF, so no problem there.

fondue emoji
fondue emoji sdf

There are an extremely rare handful of emoji with semi-transparent areas, but you can get away with making those solid. For this I just use a filter that detects '+' shaped arrangements of pixels that have (almost) the same transparency level. Then I dilate those by 3x3 to get the average transparency level in each area. Then I divide by it to only keep the anti-aliased edges transparent.

The real issue is blending the colors at the edges, when the emoji is being rendered and scaled. The RGB color of transparent pixels is undefined, so whatever values are there will blend into the surrounding pixels, e.g. creating a subtle black halo:

Not Premultiplied

Premultiplied

A common solution is premultiplied alpha. The opacity is baked into the RGB channels as (R * A, G * A, B * A, A), and transparent areas must be fully transparent black. This allows you to use a premultiplied blend mode where the RGB channels are added directly without further scaling, to cancel out the error.

But the alpha channel of an SDF glyph is dynamic, and is independent of the colors, so it cannot be premultiplied. We need valid color values even for the fully transparent areas, so that up- or downscaling is still clean.

Luckily the ESDT calculates X and Y offsets which point from each pixel directly to the nearest edge. We can use them to propagate the colors outward in a single pass, filling in the entire background. It doesn't need to be very accurate, so no filtering is required.

fondue emoji - rgb

RGB channel

fondue emoji - rendered via sdf

Output

The result looks pretty great. At normal sizes, the crisp edge hides the fact that the interior is somewhat blurry. Emoji fonts are supported via the underlying ab_glyph library, but are too big for the web (10MB+). So you can just load .PNGs on demand instead, at whatever resolution you need. Hooking it up to the 2D canvas to render native system emoji is left as an exercise for the reader.

Use.GPU does not support complex Unicode scripts or RTL text yet—both are a can of worms I wish to offload too—but it does support composite emoji like "pirate flag" (white flag + skull and crossbones) or "male astronaut" (astronaut + man) when formatted using the usual Zero-Width Joiners (U+200D) or modifiers.

Shading

Finally, a note on how to actually render with SDFs, which is more nuanced than you might think.

I pack all the SDF glyphs into an atlas on-demand, the same I use elsewhere in Use.GPU. This has a custom layout algorithm that doesn't backtrack, optimized for filling out a layout at run-time with pieces of a similar size. Glyphs are rasterized at 1.5x their normal font size, after rounding up to the nearest power of two. The extra 50% ensures small fonts on low-DPI displays still use a higher quality SDF, while high-DPI displays just upscale that SDF without noticeable quality loss. The rounding ensures similar font sizes reuse the same SDFs. You can also override the detail independent of font size.

To determine the contrast factor to draw an SDF, you generally use screen-space derivatives. There are good and bad ways of doing this. Your goal is to get a ratio of SDF pixels to screen pixels, so the best thing to do is give the GPU the coordinates of the SDF texture pixels, and ask it to calculate the difference for that between neighboring screen pixels. This works for surfaces in 3D at an angle too. Bad ways of doing this will instead work off relative texture coordinates, and introduce additional scaling factors based on the view or atlas size, when they are all just supposed to cancel out.

As you then adjust the contrast of an SDF to render it, it's important to do so around the zero-level. The glyph's ideal vector shape should not expand or contract as you scale it. Like TinySDF, I use 75% gray as the zero level, so that more SDF range is allocated to the outside than the inside, as dilating glyphs is much more common than contraction.

At the same time, a pixel whose center sits exactly on the zero level edge is actually half inside, half outside, i.e. 50% opaque. So, after scaling the SDF, you need to add 0.5 to the value to get the correct opacity for a blend. This gives you a mathematically accurate font rendering that approximates convolution with a pixel-sized circle or box.

But I go further. Fonts were not invented for screens, they were designed for paper, with ink that bleeds. Certain renderers, e.g. MacOS, replicate this effect. The physical bleed distance is relatively constant, so the larger the font, the smaller the effect of the bleed proportionally. I got the best results with a 0.25 pixel bleed at 32px or more. For smaller sizes, it tapers off linearly. When you zoom out blocks of text, they get subtly fatter instead of thinning out, and this is actually a great effect when viewing document thumbnails, where lines of text become a solid mass at the point where the SDF resolution fails anyway.

Sample of Use.GPU text at various scales

In Use.GPU I prefer to use gamma correct, linear RGB color, even for 2D. What surprised me the most is just how unquestionably superior this looks. Text looks rock solid and readable even at small sizes on low-DPI. Because the SDF scales, there is no true font hinting, but it really doesn't need it, it would just be a nice extra.

Presumably you could track hinted points or edges inside SDF glyphs and then do a dynamic distortion somehow, but this is an order of magnitude more complex than what it is doing now, which is splat a contrasted texture on screen. It does have snapping you can turn on, which avoids jiggling of individual letters. But if you turn it off, you get smooth subpixel everything:

I was always a big fan of the 3x1 subpixel rendering used on color LCDs (i.e. ClearType and the like), and I was sad when it was phased out due to the popularity of high-DPI displays. But it turns out the 3x res only offered marginal benefits... the real improvement was always that it had a custom gamma correct blend mode, which is a thing a lot of people still get wrong. Even without RGB subpixels, gamma correct AA looks great. Converting the entire desktop to Linear RGB is also not going to happen in our lifetime, but I really want it more now.

The "blurry text" that some people associate with anti-aliasing is usually just text blended with the wrong gamma curve, and without an appropriate bleed for the font in question.

* * *

If you want to make SDFs from existing input data, subpixel accuracy is crucial. Without it, fully crisp strokes actually become uneven, diagonals can look bumpy, and you cannot make clean dilated outlines or shadows. If you use an EDT, you have to start from a high resolution source and then downscale away all the errors near the edges. But if you use an ESDT, you can upscale even emoji PNGs with decent results.

It might seem pretty obvious in hindsight, but there is a massive difference between getting it sort of working, and actually getting all the details right. There were many false starts and dead ends, because subpixel accuracy also means one bad pixel ruins it.

In some circles, SDF text is an old party trick by now... but a solid and reliable implementation is still a fair amount of work, with very little to go off for the harder parts.

By the way, I did see if I could use voronoi techniques directly, but in terms of computation it is much more involved. Pretty tho:

The ESDT is fast enough to use at run-time, and the implementation is available as a stand-alone import for drop-in use.

This post started as a single live WebGPU diagram, which you can play around with. The source code for all the diagrams is available too.

I just released Autoptimize Pro 2.0 as beta and now AO(Pro) finally includes page caching. If you’re on Cloudflare (APO), if your host offers page caching at server level (nginx/ varnish or similar) or if you already have a page caching plugin then you don’t need page caching in AOPro, but based on discussions with current customers page caching is still an important missing piece on a lot of...

Source

July 14, 2023

Zephyr, an open-source real-time operating system (RTOS) launched by the Linux Foundation in 2016, has made lots of progress seven years after its announcement, and it now has an active ecosystem surrounding it. It's used in Internet of Things (IoT) sensors, Bluetooth trackers, heart rate monitors, smartwatches, and embedded controllers. A few months ago I wrote an overview article for LWN.net about the project, Zephyr: a modular OS for resource-constrained devices.

The Zephyr RTOS is used in a lot of commercial products as well as some open-source projects. Developers are also continuously adding support for new development boards. Currently, Zephyr supports more than 450 boards from various architectures.

While most of these boards are the typical developer boards used by professional developers, recent years have seen an uptick in support for more boards popular with hobbyists. Here's a list of some supported boards that will surely ring a bell if you're active in the maker community:

Note

Zephyr support for a particular board doesn't mean that all of the board's features are supported. Always consult the device page in Zephyr's documentation for detailed information about the support status.

So what can you do with Zephyr? An interesting use case for the operating system is developing Bluetooth Low Energy applications running on one of the above boards. A previous blog article explaining Zephyr's iBeacon example code gives you an idea about how such code looks like. If this piques your curiosity, consider delving deeper into the subject by reading my book Develop your own Bluetooth Low Energy Applications for Raspberry Pi, ESP32 and nRF52 with Python, Arduino and Zephyr, which uses Nordic Semiconductor's nRF52840 Dongle and nRF52840 Development Kit. But Zephyr supports a plethora of communication protocols, including Thread, a low-power IPv6-based wireless mesh-networking technology for home-automation applications, and other protocols listed in the aforementioned LWN.net article.

Even if you're not up to the task of developing applications in the C programming language, Zephyr can be interesting for you. If you prefer Python, you've surely heard about MicroPython, a programming language that implements a sizable subset of Python that can run on microcontrollers. You can read more about it in my recent LWN.net article MicroPython 1.20: Python for microcontrollers. MicroPython offers firmware downloads for more than 150 microcontroller boards, but it also has been ported to Zephyr.

In MicroPython 1.20, this port is based on Zephyr 3.1.0, which was released in June 2022; the current Zephyr release is 3.4.0. A Zephyr development environment can be used to build MicroPython for every target board supported by Zephyr, although not all have been tested. It also gives MicroPython code access to Zephyr's uniform sensor API using the zsensor module. More information can be found in MicroPython's documentation about the Zephyr port.

Thanks to Zephyr's broad hardware support, the Zephyr port allows MicroPython to run on the BBC micro:bit v2, whereas only a direct MicroPython port for the BBC micro:bit v1 exists. Other interesting boards from the above list that don't have a MicroPython firmware download but that are supported thanks to the Zephyr port are the Arduino Nano 33 IoT, the PineTime, and the RuuviTag. The process of setting up the development environment, building the Zephyr port for the target board, and then flashing the firmware to the board is explained in the port's README file.

July 06, 2023

On the Internet, you can get a lot of advice from almost anywhere when you’re looking for information on how to do your DBA job.

My blog is one of these sources of advice, and depending on the source, we generally trust the advice more or less.

But sometimes advice doesn’t take the whole picture into account, and even if it comes from a recognized authority, it can lead to more or less serious problems.

Let’s consider the following situation:

We have an InnoDB ReplicaSet with 1 replication source (primary) and 2 replicas:

 JS > rs.status()
{
    "replicaSet": {
        "name": "myReplicaSet", 
        "primary": "127.0.0.1:3310", 
        "status": "AVAILABLE", 
        "statusText": "All instances available.", 
        "topology": {
            "127.0.0.1:3310": {
                "address": "127.0.0.1:3310", 
                "instanceRole": "PRIMARY", 
                "mode": "R/W", 
                "status": "ONLINE"
            }, 
            "127.0.0.1:3320": {
                "address": "127.0.0.1:3320", 
                "instanceRole": "SECONDARY", 
                "mode": "R/O", 
                "replication": {
                    "applierStatus": "APPLIED_ALL", 
                    "applierThreadState": "Waiting for an event from Coordinator", 
                    "applierWorkerThreads": 4, 
                    "receiverStatus": "ON", 
                    "receiverThreadState": "Waiting for source to send event", 
                    "replicationLag": null, 
                    "replicationSsl": "TLS_AES_256_GCM_SHA384 TLSv1.3"
                }, 
                "status": "ONLINE"
            }, 
            "127.0.0.1:3330": {
                "address": "127.0.0.1:3330", 
                "instanceRole": "SECONDARY", 
                "mode": "R/O", 
                "replication": {
                    "applierStatus": "APPLIED_ALL", 
                    "applierThreadState": "Waiting for an event from Coordinator", 
                    "applierWorkerThreads": 4, 
                    "receiverStatus": "ON", 
                    "receiverThreadState": "Waiting for source to send event", 
                    "replicationLag": null, 
                    "replicationSsl": "TLS_AES_256_GCM_SHA384 TLSv1.3"
                }, 
                "status": "ONLINE"
            }
        }, 
        "type": "ASYNC"
    }
}

As a good DBA, we also have trending graphs and we can some aborted connections… and if we check the information about this metric in Prometheus/PMM, we can read that we can cleanup the host cache table if an host reached the max_connect_errors threshold:

So let’s do that:

 SQL > flush hosts;
Query OK, 0 rows affected, 1 warning (0.0080 sec)
Warning (code 1287): 'FLUSH HOSTS' is deprecated and will be removed in a future release.
Please use TRUNCATE TABLE performance_schema.host_cache instead

Ok, it seems that this command is deprecated, but it worked…

But after that if we check the status of our ReplicaSet, we can notice something strange:

 JS > rs.status()
{
    "replicaSet": {
        "name": "myReplicaSet", 
        "primary": "127.0.0.1:3310", 
        "status": "AVAILABLE_PARTIAL", 
        "statusText": "The PRIMARY instance is available, but one or more SECONDARY instances are not.", 
        "topology": {
            "127.0.0.1:3310": {
                "address": "127.0.0.1:3310", 
                "instanceRole": "PRIMARY", 
                "mode": "R/W", 
                "status": "ONLINE"
            }, 
            "127.0.0.1:3320": {
                "address": "127.0.0.1:3320", 
                "fenced": true, 
                "instanceErrors": [
                    "ERROR: 1 errant transaction(s) detected. Topology changes will not be possible until the instance is removed from the replicaset to have the inconsistency repaired."
                ], 
                "instanceRole": "SECONDARY", 
                "mode": "R/O", 
                "replication": {
                    "applierStatus": "APPLIED_ALL", 
                    "applierThreadState": "Waiting for an event from Coordinator", 
                    "applierWorkerThreads": 4, 
                    "receiverStatus": "ON", 
                    "receiverThreadState": "Waiting for source to send event", 
                    "replicationLag": null, 
                    "replicationSsl": "TLS_AES_256_GCM_SHA384 TLSv1.3"
                }, 
                "status": "INCONSISTENT", 
                "transactionSetConsistencyStatus": "INCONSISTENT", 
                "transactionSetConsistencyStatusText": "There are 1 transactions that were executed in this instance that did not originate from the PRIMARY.", 
                "transactionSetErrantGtidSet": "6b5c4051-1bf5-11ee-8304-c8cb9e32df8e:1"
            }, 
            "127.0.0.1:3330": {
                "address": "127.0.0.1:3330", 
                "instanceRole": "SECONDARY", 
                "mode": "R/O", 
                "replication": {
                    "applierStatus": "APPLIED_ALL", 
                    "applierThreadState": "Waiting for an event from Coordinator", 
                    "applierWorkerThreads": 4, 
                    "receiverStatus": "ON", 
                    "receiverThreadState": "Waiting for source to send event", 
                    "replicationLag": null, 
                    "replicationSsl": "TLS_AES_256_GCM_SHA384 TLSv1.3"
                }, 
                "status": "ONLINE"
            }
        }, 
        "type": "ASYNC"
    }
}

Ouch ! That FLUSH HOSTS statement created a transaction on my replica !

Let’s verify that this errant transaction belongs to the previous FLUSH command:

$ mysqlbinlog dell-bin.000002 | grep '6b5c4051-1bf5-11ee-8304-c8cb9e32df8e:1' -A 5
SET @@SESSION.GTID_NEXT= '6b5c4051-1bf5-11ee-8304-c8cb9e32df8e:1'/*!*/;
# at 3752
#230706 14:19:40 server id 338567193  end_log_pos 3829 CRC32 0x7d36a7d0 	Query	thread_id=27	exec_time=0error_code=0
SET TIMESTAMP=1688645980/*!*/;
SET @@session.sql_mode=1168113696/*!*/;
flush hosts

So indeed, the GTIDs matches…

It could have been worse: if the replica had been restarted (maintenance, crash), it would never have been able to join the ReplicaSet.

 JS > rs.status()
{
    "replicaSet": {
        "name": "myReplicaSet", 
        "primary": "127.0.0.1:3310", 
        "status": "AVAILABLE_PARTIAL", 
        "statusText": "The PRIMARY instance is available, but one or more SECONDARY instances are not.", 
        "topology": {
            "127.0.0.1:3310": {
                "address": "127.0.0.1:3310", 
                "instanceRole": "PRIMARY", 
                "mode": "R/W", 
                "status": "ONLINE"
            }, 
            "127.0.0.1:3320": {
                "address": "127.0.0.1:3320", 
                "fenced": true, 
                "instanceErrors": [
                    "ERROR: 1 errant transaction(s) detected. Topology changes will not be possible until the instance is removed from the replicaset to have the inconsistency repaired."
                ], 
                "instanceRole": "SECONDARY", 
                "mode": "R/O", 
                "replication": {
                    "applierStatus": "APPLIED_ALL", 
                    "applierThreadState": "Waiting for an event from Coordinator", 
                    "applierWorkerThreads": 4, 
                    "receiverStatus": "ON", 
                    "receiverThreadState": "Waiting for source to send event", 
                    "replicationLag": null, 
                    "replicationSsl": "TLS_AES_256_GCM_SHA384 TLSv1.3"
                }, 
                "status": "INCONSISTENT", 
                "transactionSetConsistencyStatus": "INCONSISTENT", 
                "transactionSetConsistencyStatusText": "There are 1 transactions that were executed in this instance that did not originate from the PRIMARY.", 
                "transactionSetErrantGtidSet": "6b5c4051-1bf5-11ee-8304-c8cb9e32df8e:1"
            }, 
            "127.0.0.1:3330": {
                "address": "127.0.0.1:3330", 
                "instanceRole": "SECONDARY", 
                "mode": "R/O", 
                "replication": {
                    "applierStatus": "APPLIED_ALL", 
                    "applierThreadState": "Waiting for an event from Coordinator", 
                    "applierWorkerThreads": 4, 
                    "receiverStatus": "ON", 
                    "receiverThreadState": "Waiting for source to send event", 
                    "replicationLag": null, 
                    "replicationSsl": "TLS_AES_256_GCM_SHA384 TLSv1.3"
                }, 
                "status": "ONLINE"
            }
        }, 
        "type": "ASYNC"
    }
}

The same problem arises with RAW asynchronous replication (configured manually) or with InnoDB ClusterSet.

As there is no data inconsistency as the errant transaction was only related to a `FLUSH HOSTS` command, we can fix it by inserting an empty transaction to the Primary instance:

SQL >  SET GTID_NEXT="6b5c4051-1bf5-11ee-8304-c8cb9e32df8e:1";
SQL >  START TRANSACTION;
SQL >  COMMIT;
SQL >  SET GTID_NEXT="AUTOMATIC"; 

And now the replica will be able to join back.

The supervision system gave us bad advice, which we followed without thinking about our architecture.

When replication is involved, it’s always recommended to use the LOCAL keyword with FLUSH instructions.

And for the host cache table, the TRUNCATE command in the Performance_Schema table is also recommended.

To illustrate this, take a look at the following commands and the created GTIDs:

 SQL > select @@gtid_executed\G
*************************** 1. row ***************************
@@gtid_executed: f72446ef-1bf4-11ee-8b28-c8cb9e32df8e:1-105
1 row in set (0.0007 sec)

 SQL > flush hosts;
Query OK, 0 rows affected, 1 warning (0.0078 sec)

 SQL > select @@gtid_executed\G
*************************** 1. row ***************************
@@gtid_executed: 61314e4f-1bfd-11ee-8bf1-c8cb9e32df8e:1,
f72446ef-1bf4-11ee-8b28-c8cb9e32df8e:1-105
1 row in set (0.0004 sec)

 SQL > flush local hosts;
Query OK, 0 rows affected, 1 warning (0.0003 sec)

 SQL > select @@gtid_executed\G
*************************** 1. row ***************************
@@gtid_executed: 61314e4f-1bfd-11ee-8bf1-c8cb9e32df8e:1,
f72446ef-1bf4-11ee-8b28-c8cb9e32df8e:1-105
1 row in set (0.0003 sec)

 SQL > truncate performance_schema.host_cache;
Query OK, 0 rows affected (0.0006 sec)

 SQL > select @@gtid_executed\G
*************************** 1. row ***************************
@@gtid_executed: 61314e4f-1bfd-11ee-8bf1-c8cb9e32df8e:1,
f72446ef-1bf4-11ee-8b28-c8cb9e32df8e:1-105
1 row in set (0.0007 sec)

We can notice that only the first command generated a new GTID.

Conclusion

When applying recommendations, it’s always necessary to understand the implication of the commands and whether they apply to your environment without side effects.

Enjoy MySQL !

To continue our journey to Moodle on Oracle Cloud Infrastructure using Ampere compute instances and MySQL HeatWave Database Service [1] [2], in this article we will see how to scale our architecture using multiple Moodle instances, High Availability for the Database and Read Scale-Out.

This is the architecture we will deploy:

The same principles can be applied to other projects, not just Moodle.

Multiple Compute Instances & MySQL HeatWave High Availability

The first step is to use again the Stack to deploy the initial resources. We must insure that we use a MySQL Shape that has at least 4 OCPUs to later enable Read Replicas (see [2]):

For the Compute Instances where Moodle will be installed, we use Ampere Shapes:

When deployed, we can see that now the MySQL HeatWave instance provides High Availability:

This means that there are now 3 servers in a MySQL Group Replication cluster. In the event of failure of the primary member, another node in another availability zone will automatically take over the primary role.

Shared Storage

All Moodle instances will connect to the same database instance running in MySQL HeatWave Database Service, but they also need to share several files that are usually stored locally on the web server.

To share that particular content with all the deployed Moodle web servers, we will use a NFS mount.

In OCI, we can easily create a File System to be mounted on each web servers:

We need to specify that we will use it for NFS and we provide a name for the export:

Once created we can check the Exports:

And on the export, we can get the information about the mount commands and the ports we need to open in the security list:

Please note the IP of the NFS Server, we will need it.

Mounting the NFS share

Before being able to mount our new NFS share, we need to allow the connections in the private subnet:

We need to have the following Ingress Rules:

And the following Egress Rules:

When the Security List is created we need to associate it with the private subnet:

When the security list is added, we need to connect in SSH on all the Compute Instances we deployed for Moodle. In this example we have 2 instances:

You can refer to the previous post to learn how to get the private ssh key to connect to the compute instances using their public IP.

The mount point (/var/www/moodledata) is already created, but we need to install nfs-utils:

[opc@moodleserver1 ~]$ sudo dnf install -y nfs-utils

We can now mount it:

[opc@moodleserver1 ~]$ sudo mount 10.0.1.61:/FileSystem-moodledata /var/www/moodledata
[opc@moodleserver1 ~]$ sudo chown apache. /var/www/moodledata

It’s also possible to add this in fstab to mount it automatically. Add the following line to /etc/fstab:

10.0.1.61:/FileSystem-moodledata  /var/www/moodledata   nfs  defaults	0	0

Please note that 10.0.1.61 is the IP we received the from the mount commands when we created the filesystem.

We also need to tell SELinux to let Apache (httpd) use the NFS mount:

[opc@moodleserver1 ~]$ sudo setsebool -P httpd_use_nfs=1

We need to perform this operation on all Compute Instances where Moodle is deployed.

When this is done, we can complete the Moode installation using the public IP of one of the Compute Instances in our browser:

Network Load Balancer

For the moment, we can access our Moodle Compute Instances individually using the public IP assigned to them. But now we’re going to set up a Network Load Balancer and use this new public IP to access our Moodle site. We’ll end up on any of the instances.

We use a listener on port 80 in TCP:

Now it’s time to add backends:

We choose our two Moodle Compute Instances as backends:

When they are selected, we can continue:

We also need to define a health check. You can notice that I’m using 303 as status code, as there is a redirect when we reach the root of Moodle:

And we can create the Network Load Balancer:

Once created, the Network Load Balancer will perform the health check and will be ready. A new public IP will be assigned to it:

This is the new IP used to visit our Moodle site.

Read Scale-Out

The load is already spread over multiple Compute Instances of Moodle. In case of a failure, we are covered with HA. Additionally we can also send all the reads to different MySQL HeatWave instances: Read Replicas.

To deploy read replicas, you can refer on this post. Don’t forget that you need to modify the Moodle DB’s configuration on each Moodle Servers.

Our MySQL HeatWave Instance looks like this now:

And we have 4 endpoints, 1 for each read replicas, 1 for the read replica load balancer and 1 for the primary member of the HA cluster, for the writes:

Conclusion

In conclusion, deploying multiple instances of Moodle on Oracle Cloud Infrastructure using MySQL HeatWave Database Service is a highly effective and robust solution for ensuring high availability and excellent performance of your e-learning platform. The system’s resilience and scalability make it a fantastic choice for managing heavy traffic and dynamically changing loads.

With MySQL HeatWave, you benefit not only from the power of the world’s most popular open-source database, but also from the added advantage of a robust high availability solution and the elasticity of having multiple read replicas to boost read operations and increase data redundancy. This strategy dramatically improves performance and reduces the risk of data loss, enabling you to offer a seamless, uninterrupted learning experience. And don’t forget that if you need even more performance, you can also activate HeatWave Cluster, the in-memory query accelerator available on OCI.

And of course the same architecture can be used for any other products like WordPress, Drupal, Joomla! and more !

Enjoy Moodle, MySQL HeatWave and OCI !

Stop Trying to Make Social Networks Succeed

Lot is happening in the social network landscape with the demises of Twitter and Reddit, the apparition of Bluesky and Threads, the growing popularity of Mastodon. Many pundits are trying to guess which one will be successful and trying to explain why others will fail. Which completely misses the point.

Particular social networks will never "succeed". Nobody even agree on the definition of "success".

The problem is that we all see our little bubble and generalise what we observe as universal. We have a hard time understanding Mastodon ? Mastodon will never succeed, it will be for a niche. A few of our favourite web stars goes to Bluesky ? Bluesky is the future, everybody will be there.

That’s not how it works. That’s not how it ever worked.

Like every human endeavour, every social network is there for a limited duration and will be useful to a limited niche of people. That niche may grow to the point of being huge, like Facebook and WhatsApp. But, to this day, there are more people in the world without an account on Facebook than people with one. Every single social network is only representative of a minority. And the opposite would be terrifying when you think about it (which is exactly what Meta is trying to build).

Social networks are fluid. They come, they go. For commercial social networks, the success is defined by: "do they earn enough money to make investors happy ?" There’s no metric of success for non-commercial ones. They simply exist as long as at least two users are using them to communicate. Which is why criticisms like "Mastodon could never raise enough money" or "the Fediverse will never succeed" totally miss the point.

If you live in the same occidental bubble as me, you might have never heard of WeChat, QQ or VK. Those are immensely popular social networks. In China and Russia. WeChat alone is more or less the size of Instagram in terms of active users. The war in Ukraine also demonstrated that the most popular social network in that part of the world is Telegram. Which is twice as big as Twitter but, for whatever reason, is barely mentioned in my own circles. The lesson here is simple: you are living in a small niche. We all do. Your experience is not representative of anything but your own. And it’s fine.

There will never be one social network to rule them all. There should never be one social network to rule them all. In fact, tech-savvy people should fight to ensure that no social network ever "succeed".

Human lives in communities. We join them, we sometimes leave them. Social networks should only be an underlying infrastructure to support our communities. Social networks are not our communities. Social network dies. Communities migrate and flock to different destinations. Nothing ever replaced Google+, which was really popular in my own tech circle. Nothing will replace Twitter or Reddit. Some communities will find a new home on Mastodon or on Lemmy. Some will go elsewhere. That’s not a problem as long as you can have multiple accounts in different places. Something I’m sure you do. Communities can be split. Communities can be merged. People can be part of several communities and several platforms.

Silicon Valley venture capitalists are trying to convince us that, one day, a social network will succeed, will become universal. That it should grow. That social networks are our communities. That your community should grow to succeed.

This is a lie, a delusion. Our communities are worth a lot more than the underlying tool used at some point in time. By accepting the confusion, we are destroying our communities. We are selling them, we are transforming them into a simple commercial asset for the makers of the tool we are using, the tool which exploits us.

Stop trying to make social networks succeed, stop dreaming of a universal network. Instead, invest in your own communities. Help them make long-term, custom and sustainable solutions. Try to achieve small and local successes instead of pursuing an imaginary universal one. It will make you happier.

It will make all of us happier.

As a writer and an engineer, I like to explore how technology impacts society. You can subscribe by email or by rss. I value privacy and never share your adress.

If you read French, you can support me by buying/sharing/reading my books and subscribing to my newsletter in French or RSS. I also develop Free Software.

June 27, 2023

The Bestway hottub’s water pump must regularly be descaled. It’s not easy to add huge amounts of chemicals to your pool water to have a meaningful effect when you have serious calcium buildup in your water pump unit. So I made myself a contraption.

I remember that somebody made something for this himself too. His contraption inspired me a little bit of course.

I used the tubes from a old Bestway hottub. I just cut them out of the inflatable pool before I disposed myself of the rest. Then I made two holes in a cheap water bucket and I attached the tubes with some rubber and other things to seal them to the holes.

Now I can descale my Bestway water pump unit like how the professionals do it!

Since before I got involved in the eID back in 2014, we have provided official packages of the eID for Red Hat Enterprise Linux. Since RHEL itself requires a license, we did this, first, by using buildbot and mock on a Fedora VM to set up a CentOS chroot in which to build the RPM package. Later this was migrated to using GitLab CI and to using docker rather than VMs, in an effort to save some resources. Even later still, when Red Hat made CentOS no longer be a downstream of RHEL, we migrated from building in a CentOS chroot to building in a Rocky chroot, so that we could continue providing RHEL-compatible packages. Now, as it seems that Red Hat is determined to make that impossible too, I investigated switching to actually building inside a RHEL chroot rather than a derivative one. Let's just say that might be a challenge...

[root@b09b7eb7821d ~]# mock --dnf --isolation=simple --verbose -r rhel-9-x86_64 --rebuild eid-mw-5.1.11-0.v5.1.11.fc38.src.rpm --resultdir /root --define "revision v5.1.11"
ERROR: /etc/pki/entitlement is not a directory is subscription-manager installed?

Okay, so let's fix that.

[root@b09b7eb7821d ~]# dnf install -y subscription-manager

(...)

Complete!
[root@b09b7eb7821d ~]# mock --dnf --isolation=simple --verbose -r rhel-9-x86_64 --rebuild eid-mw-5.1.11-0.v5.1.11.fc38.src.rpm --resultdir /root --define "revision v5.1.11"
ERROR: No key found in /etc/pki/entitlement directory.  It means this machine is not subscribed.  Please use 
  1. subscription-manager register
  2. subscription-manager list --all --available (available pool IDs)
  3. subscription-manager attach --pool <POOL_ID>
If you don't have Red Hat subscription yet, consider getting subscription:
  https://access.redhat.com/solutions/253273
You can have a free developer subscription:
  https://developers.redhat.com/faq/

Okay... let's fix that too, then.

[root@b09b7eb7821d ~]# subscription-manager register
subscription-manager is disabled when running inside a container. Please refer to your host system for subscription management.

Wut.

[root@b09b7eb7821d ~]# exit
wouter@pc220518:~$ apt-cache search subscription-manager
wouter@pc220518:~$

As I thought, yes.

Having to reinstall the docker host machine with Fedora just so I can build Red Hat chroots seems like a somewhat excessive requirement, which I don't think we'll be doing that any time soon.

We'll see what the future brings, I guess.

Pourquoi n’y a-t-il pas de Google européen ?

Et pourquoi c’est une bonne chose.

Google, pardon Alphabet, Facebook, pardon Meta, Twitter, Netflix, Amazon, Microsoft. Tous ces géants font partie intégrante de notre quotidien. Tous ont la particularité d’être 100% américains.

La Chine n’est pas complètement en reste avec Alibaba, Tiktok et d’autres moins populaire chez nous, mais brassant des milliards d’utilisateurs.

Et en Europe ? Beaucoup moins, au grand dam des politiciens qui ont l’impression que le bonheur d’une population, et donc ses votes, se mesure au nombre de milliardaires qu’elle produit.

Pourtant, dans le domaine Internet, l’Europe est loin d’être ridicule. Elle est même primordiale.

Car si Internet, interconnexion entre les ordinateurs du monde entier, existait depuis la fin des années 60, aucun protocole ne permettait de trouver de l’information. Il fallait savoir exactement ce que l’on cherchait. Pour combler cette lacune, Gopher fut développé aux États-Unis tandis que le Web, combinaison du protocole HTTP et du langage HTML, était inventé par un citoyen britannique et un citoyen belge qui travaillaient dans un centre de recherche européen situé en Suisse. Mais, anecdote croustillante, leur bureau débordait la frontière et on peut dire aujourd’hui que le Web a été inventé en France. Difficile de faire plus européen comme invention ! On dirait la blague européenne officielle ! (Note: tout comme Pluton restera toujours une planète, les Britanniques resteront toujours européens. Le Brexit n’est qu’une anecdote historique que la jeune génération s’empressera, j’espère, de corriger).

Bien que populaire et toujours existant aujourd’hui, Gopher ne se développera jamais réellement comme le Web pour une sombre histoire de droits et de licence, tué dans l’œuf par la quête de succès économique immédiat.

Alors même que Robert Cailliau et Tim Berners-Lee inventaient le Web dans leur bureau du CERN, un étudiant finlandais issu de la minorité suédoise du pays concevait Linux et le rendait public. Pour le simple fait de s’amuser. Linux est aujourd’hui le système d’exploitation le plus populaire du monde. Il fait tourner les téléphones Android, les plus gros serveurs Web, les satellites dans l’espace, les ordinateurs des programmeurs, les montres connectées, les mini-ordinateurs. Il est partout. Linus Torvalds, son inventeur, n’est pas milliardaire et trouve ça très bien. Cela n’a jamais été son objectif.

Mastodon, l’alternative décentralisée à Twitter créée par un étudiant allemand ayant grandi en Russie, a le simple objectif de permettre aux utilisateurs des réseaux sociaux de se passer des monopoles industriels et de pouvoir échanger de manière saine, intime, sans se faire agresser ni se faire bombarder de pub. La pub et l’invasion de la vie privée, deux fléaux du Web moderne ! C’est d’ailleurs en réaction qu’a été créé le réseau Gemini, une alternative au Web conçue explicitement pour empêcher toute dérive commerciale et remettre l’humain au centre. Le réseau Gemini a été conçu et initié par un programmeur vivant en Finlande et souhaitant garder l’anonymat. Contrairement à beaucoup de projets logiciels, Gemini n’évolue plus à dessein. Le protocole est considéré comme terminé et n’importe qui peut désormais publier sur Gemini ou développer des logiciels l’utilisant en ayant la certitude qu’ils resteront compatibles tant qu’il y’aura des utilisateurs.

On entend souvent que les Européens n’ont pas la culture du succès. Ces quelques exemples, et il y’en a bien d’autres, prouvent le contraire. Les Européens aiment le succès, mais pas au détriment du reste de la société. Un succès est perçu comme une œuvre pérenne, s’inscrivant dans la durée, bénéficiant à tous les citoyens, à toute la société voire à tout le genre humain.

Google, Microsoft, Facebook peuvent disparaître demain. Il est même presque certain que ces entreprises n’existent plus d’ici quarante ou cinquante ans. Ce serait même potentiellement une excellente chose. Mais pouvez-vous imaginer un monde sans le Web ? Un monde sans HTML ? Un monde sans Linux ? Ces inventions, initialement européennes, sont devenues des piliers de l’humanité, sont des technologies désormais indissociables de notre histoire.

La vision américaine du succès est souvent restreinte à la taille d’une entreprise ou la fortune de son fondateur. Mais pouvons-nous arrêter de croire que le succès est équivalent à la croissance ? Et si le succès se mesurait à l’utilité, à la pérennité ? Si nous commencions à valoriser les découvertes, les fondations technologiques léguées à l’humanité ? Si l’on prend le monde à la lueur de ces nouvelles métriques, si le succès n’est plus la mesure du nombre de portefeuilles vidés pour mettre le contenu dans le plus petit nombre de poches possible, alors l’Europe est incroyablement riche en succès.

Et peut-être est-ce une bonne chose de promouvoir ces succès, d’en être fier ?

Certains sont fiers de s’être enrichis en coupant le plus d’arbres possible. D’autres sont fiers d’avoir planté des arbres qui bénéficieront aux générations futures. Et si le véritable succès était de bonifier, d’entretenir et d’augmenter les communs au lieu d’en privatiser une partie ?

À nous de choisir les succès que nous voulons admirer. C’est en choisissant de qui nous chantons les louanges que nous décidons de la direction dès progrès futurs.

Ingénieur et écrivain, j’explore l’impact des technologies sur l’humain. Abonnez-vous à mes écrits en français par mail ou par rss. Pour mes écrits en anglais, abonnez-vous à la newsletter anglophone ou au flux RSS complet. Votre adresse n’est jamais partagée et effacée au désabonnement.

Pour me soutenir, achetez mes livres (si possible chez votre libraire) ! Je viens justement de publier un recueil de nouvelles qui devrait vous faire rire et réfléchir.

June 25, 2023

Time again to make some releases of 2 of the ansible roles I maintain.

This time none of the commits are created by me :-)

Thanks to https://github.com/fazlerabbi37 for your contributions!

Have fun!

qemu_img 2.2.0

stafwag.qemu_img 2.2.0 is available at: https://github.com/stafwag/ansible-role-qemu_img

playbook

Changelog

cloud_localds 2.1.1

playbook

Changelog

  • network config templating
    • bugfix. This release implements network config templating. The previous release copied the template instead of using templating. Thanks to https://github.com/fazlerabbi37

June 24, 2023

A lot of people have or are buying those cheap inflatable Bestway hottubs.

A lot of people have in the past realized after about half a year of continuous running that the water pump unit is of very low quality.

I have been running my Bestway hottub for about five or six years now. Because of this low quality I had to enroll myself into the world of Bestway parts and repairs. I regularly had and have to repair various things about my Bestway hottub. Usually something about the water pump unit.

Last time, in 2019, it was the water flow sensor. Being a good engineer, I of course hacked it. I have also bought one time a second hand infamous #58113 motor (the number is already infamous and known in the Bestway parts community).

Today I had the ER02 error back. No water flow. But after some testing I knew that it was not the water flow sensor this time. Then it’s probably the motor itself. These #58113 motors often have it that their impeller comes loose inside of the motor.

Instead of ordering either a new impeller or a new motor, I decided to investigate it this time. And try to figure out what the engineering mistake is that the person who designed this motor made.

Getting the motor out is probably already plenty challenging for most owners of a Lay-Z-Spa. It’s not too complicated though: turn the motor pump upside down. Take off the bottom panel. Loosen two screws of the motor. Disconnect the electric cable. Pull the motor out. For the motor you don’t need to open the top of the water pump unit. If your problem is the water flow sensor, then you do need to open the top instead.

I found out what is wrong about the motor (the rubber bearings are just cheap) and I will now present a hack that you can do to salvage your Bestway #58113 motor with four cheap washers that will keep the impeller better in place.

Here you have the impeller (or/plus commutator) and the rubber “bearing” for it (in background the disassembled stator):

And this is the shaft and again that rubber “bearing”. The shaft will go through the middle of the impeller (plus commutator) and at the ends two of those rubber bearings go to keep the shaft nicely centered. All this is plastic. Super low quality. Almost guaranteed to fail after a few months of operation.

This is the stator. One of those rubber bearings must go in the middle of it. And then the shaft in the bearing. Keep the rubber seal good. Else when reassembling the motor water will splash all over the place. This is not good.

This picture illustrates how the shaft goes in the rubber bearing and then in the front cap

This picture shows the normal assembly of shaft, impeller (plus commutator) and bearing. Normally there is a by the factory added washer too. But mine was completely gone. Debris of it was found in the commutator. This shows how low quality this piece of shit is. This of course should never happen.

This will be the hack. We will place three simple washers on the shaft between the impeller and the bearing.

Update: After a first inspection after two days of running with the washers I noticed that although the washers claim to be stainless steel, I saw and I realized that all steel eventually rusts. I made a little plastic washer instead and I greased it with silicone grease that I once bought to grease the seal of my Scuba dry suite.

This is the plastic washer I made to replace the metallic ones, with that grease applied:

This is that silicone grease. I greased the entire shaft with it too:

This will once reassembled keep the entire assembly (bearings, shaft, commutator) tighter together with the stator. Without these three washers the plastic washer, the whole thing starts wiggling and eventually comes loose. Impeller will become erratic and destroys the washer (in my case) and likely also one of the two rubber bearings. After that it’s pretty much game over and you’ll see the ER02 error.

We will also place a washer on the back of the impeller (or commutator) on/over the shaft (update: use the factory provided one). That will look like this (so I’m holding the impeller upside down now). It will fall off if you assemble it upfront. So this is just to show how it will be like once inside of the stator (that or I just made too much pictures, and now I have to write more blog content around it). Front cap, some more washers and stator in the background.

Let’s start assembling it all together. First the shaft with bearing that goes into the hole of the stator. With the factory provided washer. Don’t try to fiddle the impeller together with the shaft into the stator. You’ll just miserably fail due to it all being a bit magnetic of course (it’s an electric motor, remember). Do the shaft with bearing and washer first.

That looks like this (OMG I made too many pictures)

Now we will put the impeller (commutator) over the shaft. Do this gently so that the shaft does not go out of the rubber bearing.

Now place the front cap back. Ensure that the other rubber bearing is in the front cap’s middle center hole. Make sure that the little piece of shaft you have left after the three washers goes into that front cap’s bearing.

When closed that will look (when looking through the water entrance hole) like this. Look carefully and you’ll see the three washers the plastic washer. They will keep everything in place from now on. Note that since the update mentioned earlier I’m using a white plastic washer instead of three metallic ones.

Now you just close the motor by screwing the front cap tight

If this doesn’t work (I’ve had motors with the commutator magnet broken into pieces) then you can fairly easily find these motors on the second hand market. The replacement is not very hard so you don’t need to buy an entire new water pump unit.

June 23, 2023

How to Kill a Decentralised Network (such as the Fediverse)

The year is 2023. The whole Internet is under the control of the GAFAM empire. All of it? Well, not entirely. Because a few small villages are resisting the oppression. And some of those villages started to agregate, forming the "Fediverse".

With debates around Twitter and Reddit, the Fediverse started to gain fame and attention. People started to use it for real. The empire started to notice.

Capitalists Against Competition

As Peter Thiel, one of Facebook’s prominent investor, put it: "Competition is for losers." Yep, those pseudo "market is always right" people don’t want a market when they are in it. They want a monopoly. Since its inception, Facebook have been very careful to kill every competition. The easiest way of doing it being by buying companies that could, one day, become competitors. Instagram, WhatsApp to name a few, were bought only because their product attracted users and could cast a shadow on Facebook.

But the Fediverse cannot be bought. The Fediverse is an informal group of servers discussing through a protocol (ActivityPub). Those servers may even run different software (Mastodon is the most famous but you could also have Pleroma, Pixelfed, Peertube, WriteFreely, Lemmy and many others).

You cannot buy a decentralised network!

But there’s another way: make it irrelevant. That’s exactly what Google did with XMPP.

How Google joined the XMPP federation

At the end of the 20th century, instant messengers (IM) were all the rage. One of the first very successful ones was ICQ, quickly followed by MSN messenger. MSN Messenger was the Tiktok of the time: a world where teenagers could spend hours and days without adults.

As MSN was part of Microsoft, Google wanted to compete and offered Google Talk in 2005, including it in the Gmail interface. Remember that, at the time, there was no smartphone and very little web app. Applications had to be installed on the computer and Gmail web interface was groundbreaking. MSN was even at some point bundled with Microsoft Windows and it was really hard to remove it. Building Google chat with the Gmail web interface was a way to be even closer to the customers than a built-in software in the operating system.

While Google and Microsoft were fighting to achieve hegemony, free software geeks were trying to build decentralised instant messaging. Like email, XMPP was a federated protocol: multiple servers could talk together through a protocol and each user would connect to one particular server through a client. That user could then communicate with any user on any server using any client. Which is still how ActivityPub and thus the Fediverse work.

In 2006, Google talk became XMPP compatible. Google was seriously considering XMPP. In 2008, while I was at work, my phone rang. On the line, someone told me: "Hi, it’s Google and we want to hire you." I made several calls and it turned out that they found me through the XMPP-dev list and were looking for XMPP servers sysadmins.

So Google was really embracing the federation. How cool was that? It meant that, suddenly, every single Gmail user became an XMPP user. This could only be good for XMPP, right? I was ecstatic.

How Google killed XMPP

Of course, reality was a bit less shiny. First of all, despites collaborating to develop the XMPP standard, Google was doing its own closed implementation that nobody could review. It turns out they were not always respecting the protocol they were developing. They were not implementing everything. This forced XMPP development to be slowed down, to adapt. Nice new features were not implemented or not used in XMPP clients because they were not compatible with Google Talk (avatars took an awful long time to come to XMPP). Federation was sometimes broken: for hours or days, there would not be communications possible between Google and regular XMPP servers. The XMPP community became watchers and debuggers of Google’s servers, posting irregularities and downtime (I did it several times, which is probably what prompted the job offer).

And because there were far more Google talk users than "true XMPP" users, there was little room for "not caring about Google talk users". Newcomers discovering XMPP and not being Google talk users themselves had very frustrating experience because most of their contact were Google Talk users. They thought they could communicate easily with them but it was basically a degraded version of what they had while using Google talk itself. A typical XMPP roster was mainly composed of Google Talk users with a few geeks.

In 2013, Google realised that most XMPP interactions were between Google Talk users anyway. They didn’t care about respecting a protocol they were not 100% in control. So they pulled the plug and announced they would not be federated anymore. And started a long quest to create a messenger, starting with Hangout (which was followed by Allo, Duo. I lost count after that).

As expected, no Google user bated an eye. In fact, none of them realised. At worst, some of their contacts became offline. That was all. But for the XMPP federation, it was like the majority of users suddenly disappeared. Even XMPP die hard fanatics, like your servitor, had to create Google accounts to keep contact with friends. Remember: for them, we were simply offline. It was our fault.

While XMPP still exist and is a very active community, it never recovered from this blow. Too high expectation with Google adoption led to a huge disappointment and a silent fall into oblivion. XMPP became niche. So niche that when group chats became all the rage (Slack, Discord), the free software community reinvented it (Matrix) to compete while group chats were already possible with XMPP. (Disclaimer: I’ve never studied the Matrix protocol so I have no idea how it technically compares with XMPP. I simply believe that it solves the same problem and compete in the same space as XMPP).

Would XMPP be different today if Google never joined it or was never considered as part of it? Nobody could say. But I’m convinced that it would have grown slower and, maybe, healthier. That it would be bigger and more important than it is today. That it would be the default decentralised communication platform. One thing is sure: if Google had not joined, XMPP would not be worse than it is today.

It was not the first: the Microsoft Playbook

What Google did to XMPP was not new. In fact, in 1998, Microsoft engineer Vinod Vallopllil explicitly wrote a text titled "Blunting OSS attacks" where he suggested to "de-commoditize protocols & applications […]. By extending these protocols and developing new protocols, we can deny OSS project’s entry into the market."

Microsoft put that theory in practice with the release of Windows 2000 which offered support for the Kerberos security protocol. But that protocol was extended. The specifications of those extensions could be freely downloaded but required to accept a license which forbid you to implement those extensions. As soon as you clicked "OK", you could not work on any open source version of Kerberos. The goal was explicitly to kill any competing networking project such as Samba.

This anecdote was told Glyn Moody in his book "Rebel Code" and demonstrates that killing open source and decentralised projects are really conscious objectives. It never happens randomly and is never caused by bad luck.

Microsoft used a similar tactic to ensure dominance in the office market with Microsoft Office using proprietary formats (a file format could be seen as a protocol to exchange data). When alternatives (OpenOffice then LibreOffice) became good enough at opening doc/xls/ppt formats, Microsoft released a new format that they called "open and standardised". The format was, on purpose, very complicated (20.000 pages of specifications!) and, most importantly, wrong. Yes, some bugs were introduced in the specification meaning that a software implementing the full OOXML format would behave differently than Microsoft Office.

Those bugs, together with political lobbying, were one of the reasons that pushed the city of Munich to revert its Linux migration. So yes, the strategy works well. Today, docx, xlsx and pptx are still the norms because of that. Source: I was there, indirectly paid by the city of Munich to make LibreOffice OOXML’s rendering closer to Microsoft’s instead of following the specifications.

UPDATE:

Meta and the Fediverse

People who don’t know history are doomed to repeat it. Which is exactly what is happening with Meta and the Fediverse.

There are rumours that Meta would become "Fediverse compatible". You could follow people on Instagram from your Mastodon account.

I don’t know if those rumours have a grain of truth, if it is even possible for Meta to consider it. But there’s one thing my own experience with XMPP and OOXML taught me: if Meta joins the Fediverse, Meta will be the only one winning. In fact, reactions show that they are already winning: the Fediverse is split between blocking Meta or not. If that happens, this would mean a fragmented, frustrating two-tier fediverse with little appeal for newcomers.

UPDATE: Those rumours have been confirmed as at least one Mastodon admin, kev, from fosstodon.org, has been contacted to take part in an off-the-record meeting with Meta. He had the best possible reaction: he refused politely and, most importantly, published the email to be transparent with its users. Thanks kev!

I know we all dream of having all our friends and family on the Fediverse so we can avoid proprietary networks completely. But the Fediverse is not looking for market dominance or profit. The Fediverse is not looking for growth. It is offering a place for freedom. People joining the Fediverse are those looking for freedom. If people are not ready or are not looking for freedom, that’s fine. They have the right to stay on proprietary platforms. We should not force them into the Fediverse. We should not try to include as many people as we can at all cost. We should be honest and ensure people join the Fediverse because they share some of the values behind it.

By competing against Meta in the brainless growth-at-all-cost ideology, we are certain to lose. They are the master of that game. They are trying to bring everyone in their field, to make people compete against them using the weapons they are selling.

Fediverse can only win by keeping its ground, by speaking about freedom, morals, ethics, values. By starting open, non-commercial and non-spied discussions. By acknowledging that the goal is not to win. Not to embrace. The goal is to stay a tool. A tool dedicated to offer a place of freedom for connected human beings. Something that no commercial entity will ever offer.

As a writer and an engineer, I like to explore how technology impacts society. You can subscribe by email or by rss. I value privacy and never share your adress.

If you read French, you can support me by buying/sharing/reading my books and subscribing to my newsletter in French or RSS. I also develop Free Software.

June 20, 2023

Le génocide du sac à dos

L’actualité nous semble parfois effroyable, innommable, inhumaine. L’horreur est-elle absolue ou n’est-elle qu’une question de point de vue ?

Dans le bunker étanche, les deux scientifiques contemplaient les écrans de contrôle, les yeux hagards. De longues trainées de sueurs dégoulinaient sur leur visage.

— C’est raté, dit la première.

— Ça ne peut pas ! Ce n’est pas possible ! Ce voyage dans le temps est la dernière chance de sauver l’humanité !

— Je te dis que c’est raté. Regarde les caméras de surveillance. Les robots tueurs se rapprochent. La planète continue à brûler. Rien n’a changé. Nous sommes les dernières survivantes.

La seconde secouait machinalement la tête, tapotait sur des voyants.

— Ce n’est pas possible. Ça ne pouvait pas manquer. La mission était pourtant simple. Le professeur tout bébé dans un landau dans une plaine de jeux. Nous avions même la localisation exacte et la date.

— Il y’avait plusieurs landaus.

— Les ordres étaient clairs. Les tuer tous. Le sort de la planète dépendait du fait que le Professeur ne puisse pas grandir et créer son armée de destruction. C’était immanquable.

Derrière les humaines, la porte s’ouvrit et les robots firent leur apparition, leur silhouette se détachant sur le paysage apocalyptique de la planète en train de se consumer.

— « Se rendre le 8 juin 2023 au Pâquier d’Annecy et détruire les organismes dans les landaus de la pleine de jeu. » C’était pourtant pas compliqué. Comment cela a-t-il pu foirer ?

— Malgré le conditionnement mental, il n’a pas pu, répliqua la première. Il a hésité une fraction de seconde.

— Tout ça à cause d’un type avec un sac à dos.

— À quoi tient le destin d’une planè…

Elles n’achevèrent pas et s’écroulèrent, mortes, au pied des terrifiants automates floqués du célèbre logo de l’entreprise d’intelligence artificielle fondée en 2052 par celui qui s’était fait appeler « le Professeur ».

Ingénieur et écrivain, j’explore l’impact des technologies sur l’humain. Abonnez-vous à mes écrits en français par mail ou par rss. Pour mes écrits en anglais, abonnez-vous à la newsletter anglophone ou au flux RSS complet. Votre adresse n’est jamais partagée et effacée au désabonnement.

Pour me soutenir, achetez mes livres (si possible chez votre libraire) ! Je viens justement de publier un recueil de nouvelles qui devrait vous faire rire et réfléchir.

June 19, 2023

We need more of Richard Stallman, not less

Disclaimer: I’m aware that Richard Stallman had some questionable or inadequate behaviours. I’m not defending those nor the man himself. I’m not defending blindly following that particular human (nor any particular human). I’m defending a philosophy, not the philosopher. I claim that his historical vision and his original ideas are still adequate today. Maybe more than ever.

The Free Software movement has been mostly killed by the corporate Open Source. The Free Software Foundation (FSF) and its founder, Richard Stallman (RMS), have been decried for the last twenty years, including by my 25-year-old self, as being outdated and inadequate.

I’ve spent the last 6 years teaching Free Software and Open Source at École Polytechnique de Louvain, being forced to investigate the subject and the history more than I anticipated in order to answer students’ questions. I’ve read many historical books on the subject, including RMS’s biography and many older writings.

And something struck me.

RMS was right since the very beginning. Every warning, every prophecy realised. And, worst of all, he had the solution since the start. The problem is not RMS or FSF. The problem is us. The problem is that we didn’t listen.

The solution has always been there: copyleft

In the early eighties, RMS realised that software was transformed from "a way to use a machine" to a product or a commodity. He foresaw that this would put an end to collective intelligence and to knowledge sharing. He also foresaw that if we were not the master of our software, we would quickly become the slave of the machines controlled by soulless corporations. He told us that story again and again.

Forty years later, we must admit he was prescient. Every word he said still rings true. Very few celebrated forward thinkers were as right as RMS. Yet, we don’t like his message. We don’t like how he tells it. We don’t like him. As politicians understood quickly, we care more about appearance and feel-good communication than about the truth or addressing the root cause.

RMS theorised the need for the "four freedoms of software".

- The right to use the software at your discretion

- The right to study the software

- The right to modify the software

- The right to share the software, including the modified version

How to guarantee those freedoms ? RMS invented copyleft. A solution he implemented in the GPL license. The idea of copyleft is that you cannot restrain the rights of the users. Copyleft is the equivalent of the famous « Il est interdit d’interdire » (it is forbidden to forbid).

In hindsight, the solution was and still is right.

Copyleft is a very deep concept. It is about creating and maintaining commons. Commons resources that everybody could access freely, resources that would be maintained by the community at large. Commons are frightening to capitalist businesses as, by essence, capitalist businesses try to privatise everything, to transform everything into a commodity. Commons are a non-commodity, a non-product.

Capitalist businesses were, obviously, against copyleft. And still are. Steve Ballmer famously called the GPL a "cancer". RMS was and still is pictured as a dangerous maniac, a fanatic propagating the cancer.

Bruce Perens and Eric Raymond tried to find a middle ground and launched the "Open Source" movement. Retrospectively, Open Source was a hack. It was originally seen as a simple rebranding of "Free Software", arguing that "free" could be understood as "without price or value" in English.

RMS quickly pointed, rightly, that the lack of "freedom" means that people will forget about the concept. Again, he was right. But everybody considered that "Free Software" and "Open Source" were the same because they both focused on the four freedoms. That RMS was nitpicking.

RMS biggest mistake

There was one weakness in RMS theory: copyleft was not part of the four freedoms he theorised. Business-compatible licenses like BSD/MIT or even public domain are "Free Software" because they respect the four freedoms.

But they can be privatised.

And that’s the whole point. For the last 30 years, businesses and proponents of Open Source, including Linus Torvalds, have been decrying the GPL because of the essential right of "doing business" aka "privatising the common".

They succeeded so much that the essential mission of the FSF to guarantee the common was seen as "useless" or, worse, "reactionary". What was the work of the FSF? The most important thing is that they proof-bombed the GPL against weaknesses found later. They literally patched vulnerabilities. First the GPLv3, to fight "Tivoisation" and then AGPL, to counteract proprietary online services running on free software but taking away freedom of users.

But all this work was ridiculed. Microsoft, through Github, Google and Apple pushed for MIT/BSD licensed software as the open source standard. This allowed them to use open source components within their proprietary closed products. They managed to make thousands of free software developers work freely for them. And they even received praise because, sometimes, they would hire one of those developers (like it was a "favour" to the community while it is simply business-wise to hire smart people working on critical components of your infrastructure instead of letting them work for free). The whole Google Summer of Code, for which I was a mentor multiple years, is just a cheap way to get unpaid volunteers mentor their future free or cheap workforce.

Our freedoms were taken away by proprietary software which is mostly coded by ourselves. For free. We spent our free time developing, debugging, testing software before handing them to corporations that we rever, hoping to maybe get a job offer or a small sponsorship from them. Without Non-copyleft Open Source, there would be no proprietary MacOS, OSX nor Android. There would be no Facebook, no Amazon. We created all the components of Frankenstein’s creature and handed them to the evil professor.

More commons

The sad state of computing today makes computer people angry. We see that young student are taught "computer" with Word and PowerPoint, that young hackers are mostly happy with rooting Android phones or blindly using the API of a trendy JS framework. That Linux distributions are only used by computer science students in virtualised containers. We live in the dystopia future RMS warned us about.

Which, paradoxically, means that RMS failed. He was a Cassandra. Intuitively, we think we should change him, we should replace the FSF, we should have new paradigms which are taking into account ecology and other ethical stances.

We don’t realise that the solution is there, in front of us for 40 years: copyleft.

Copyleft as in "Forbidding privatising the commons".

We need to rebuild the commons. When industries are polluting the atmosphere or the oceans, they are, in fact, privatising the commons ("considering a common good as their private trash"). When an industry receives millions in public subsidies then make a patent, that industry is privatising the common. When Google is putting the Linux kernel in a phone that cannot be modified easily, Google is privatising the common. Why do we need expensive electric cars? Because the automotive industry has been on a century-long mission to kill public transport or the sole idea of going on foot, to destroy the commons.

We need to defend our commons. Like RMS did 40 years ago. We don’t want to get rid of RMS, we need more of his core philosophy. We were brainwashed into thinking that he was an extremist just like we are brainwashed to think that taking care of the poor is socialist extremism. In lots of occidental countries, political positions seen as "centre" twenty years ago are now seen as "extreme left" because the left of twenty years ago was called extremist. RMS suffered the same fate and we should not fall for it.

Fighting back

What could I do? Well, the first little step I can do myself is to release every future software I develop under the AGPL license. To put my blog under a CC By-SA license. I encourage you to copyleft all the things!

We need a fifth rule. An obligation to maintain the common to prevent the software of being privatised. This is the fifth line that RMS grasped intuitively but, unfortunately for us, he forgot to put in his four freedoms theory. The world would probably be a very different place if he had written the five rules of software forty years ago.

But if the best time to do it was forty years ago, the second-best moment is right now. So here are

The four freedoms and one obligation of free software

- The right to use the software at your own discretion

- The right to study the software

- The right to modify the software

- The right to redistribute the software, including with modifications

- The obligation to keep those four rights, effectively keeping the software in the commons.

We need to realise that any software without that last obligation will, sooner or later, become an oppression tool against ourselves. And that maintaining the commons is not only about software. It’s about everything we are as a society and everything we are losing against individual greed. Ultimately, our planet is our only common resource. We should defend it from becoming a commodity.

Copyleft was considered a cancer. But a cancer to what? To the capitalist consumerism killing the planet? Then I will proudly side with the cancer.

As a writer and an engineer, I like to explore how technology impacts society. You can subscribe by email or by rss. I value privacy and never share your adress.

If you read French, you can support me by buying/sharing/reading my books and subscribing to my newsletter in French or RSS. I also develop Free Software.

June 18, 2023

Voor een deel komt dit doordat Westerse hoogtechnologische wapens nog niet volledig toegezegd worden aan de oorlog in Oekraïne. Voor een ander deel niet.

Ongeacht de fantagtische mega whoo ideeën van Westerlingen en hun oorlogsmateriaalproducerende nageslacht, blijkt een echte oorlog met een echte tegenstander (dus niet één zoals in bijna alle Westerse oorlogen tegen zandmannen met refurbished Kalashnikovs) een oorlog te zijn zoals de Tweede Wereldoorlog was:

Het belangrijkste is en blijft de aanvoerlijn.

Daar is nagenoeg niet in geïnvesteerd. Want dat kon tijdens Rambo III en Terminator niet verkocht worden aan de filmindustrie van Hollywood.

De aanvoerlijn is nog steeds voor zowel Oekraïne als Rusland zoals het tijdens de Tweede Wereldoorlog was. En omdat Rusland momenteel een veel beter georganiseerde aanvoerlijn heeft als Oekraïne, zullen zij deze oorlog voorlopig gezien dan ook winnen.

De filmindustrie van Hollywood is daar niet de schuldige van. Maar wel de propagandaindustrie van het Westen. Zij hebben gekozen voor deze richting van kartonnen huizen zonder de diepte van een echte investering. Zij holden ons uit. Want enkel de buitenkant doet er toe.

JDAMs zijn zo’n voorbeeld daarvan: goed tegen zandmannen die geen GPS signaal kunnen jammen. De Russen kunnen dat wel. Maar daar gingen we niet tegen vechten. Dus investeren in bv. inertial navigation systems was geen doel. Kartonnen huizen. Geen diepte. De definitie van het Westen.

De dag van vandaag moet modern militair materiaal er vooral mat zwart en koel uitzien. Zoals de lak van een macho-auto of iPhone. De buitenkant doet er toe. De man die met het spuitpistool komt is belangrijker. Vooral als die met idiote maar wel modieuze camouflagepatronen afkomt (die vooral onbewezen waren, en soldatenlevens hebben gekost). Het is de karton dat er toe doet. Dat is modern. Het moet er goed uitzien. Goed voelen. Mooi zijn. Het moet niet goed zijn. Het moet goed voelen.

Wij Westerlingen zijn zwakke prutsers geworden. Prutsers die een miljoen keer teveel geld uitgeven aan zaken die er gewoon niet toe doen en die zelfs gevaarlijk slecht zijn.

ps. Ik plaatste geen links. Maar ga er maar van uit dat ik “modieuze camouflage”, “JDAMs wiens GPS signaal gejammed kan worden”, “Rambo III” en “Terminator” en “zandmannen met refurbished Kalashnikovs” had kunnen linken. Je zou ook versteld staan van waar ik naar zou linken. Hoe debiel het allemaal is. Zelfs fundamenteel en vooral ook ontegensprekelijk debiel.

Ons materiaal is slecht. Omdat het niet werkt in een echte oorlogssituatie.

June 17, 2023

The Python REPL (read-eval-print loop), which is Python's interactive interpreter, is a great way for quickly testing simple Python commands. I use it quite often as a powerful command-line calculator, but also for exploring new Python libraries.

Many interesting Python libraries use asynchronous I/O with asyncio. This means that, instead of just calling a function directly, you have to await a coroutine. However, if you try this in the REPL, you encounter the following error message:

$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(5)
  File "<stdin>", line 1
SyntaxError: 'await' outside function
>>>

This is expected, because this line of code would never run in a Python script either. You would need to define a top-level coroutine with async def that calls one or more coroutines with await, and run the top-level coroutine with asyncio.run(). The canonical "Hello world" example from the Python documentation of coroutines looks something like this:

$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> async def main():
...     print("Hello")
...     await asyncio.sleep(5)
...     print("world")
...
>>> asyncio.run(main())
Hello
world
>>>

There's a five-second delay between the output of "Hello" and "world".

This seems a bit cumbersome. If you just want to call the asyncio.sleep() coroutine, you can simplify this to:

$ python
Python 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> asyncio.run(asyncio.sleep(5))
>>>

This is already better! However, it still means that every time you want to call a coroutine, you need to remember to wrap it inside an asyncio.run() call.

Fortunately, Python 3.8 introduced a top-level await if you run the asyncio module as a script:

$ python -m asyncio
asyncio REPL 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> await asyncio.sleep(5)
>>>

The REPL helpfully mentions Use "await" directly instead of "asyncio.run()". before importing the asyncio module. [1] Then you can simply type await asyncio.sleep(5) without having to call asyncio.run().

Although this might not seem like much of an improvement in this particular case, when using asynchronous libraries in a Python REPL, having to add asyncio.run() for every coroutine call quickly becomes tedious.

For example, I can now easily request the Bluetooth adapters on my system (using Home Assistant's bluetooth-adapters package):

$ python -m asyncio
asyncio REPL 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> from bluetooth_adapters import get_adapters
>>> adapters = get_adapters()
>>> await adapters.refresh()
>>> adapters.adapters
{'hci0': {'address': '9C:FC:E8:XX:XX:XX', 'sw_version': 'tux', 'hw_version': 'usb:v1D6Bp0246d0540', 'passive_scan': True, 'manufacturer': 'Intel Corporate', 'product': '0029', 'vendor_id': '8087', 'product_id': '0029'}}

Or, I can conduct a quick scan for Bluetooth Low Energy (BLE) devices in the vicinity (using the Bleak package):

$ python -m asyncio
asyncio REPL 3.10.6 (main, May 29 2023, 11:10:38) [GCC 11.3.0] on linux
Use "await" directly instead of "asyncio.run()".
Type "help", "copyright", "credits" or "license" for more information.
>>> import asyncio
>>> from bleak import BleakScanner
>>> devices = await BleakScanner.discover()
>>> [device.name for device in devices]
['Qingping Alarm Clock', 'abeacon_AC7D', 'TP358 (52C6)', '1B-0F-09-4F-89-F3', 'ThermoBeacon', 'F9-DA-D2-0D-62-24', '6A-C8-79-F4-E1-E5', 'Qingping BT Clock Lite', 'LYWSD02', 'Ruuvi 7E0E', 'TY', 'TP393 (2A3D)', 'Flower care', '52-9E-F1-64-DB-DF']

Give this top-level await approach a try with some of your favorite asynchronous Python libraries. You'll definitely become more productive in the REPL.

[1]

Note that the REPL is now calling itself "asyncio REPL 3.10.6" instead of "Python 3.10.6".

June 16, 2023

So I heard Edward November on the (online) radio a couple of times already and the first and only single “Cold Street Light” is magic. He (aka Edmund Lauret) should either stop making music entirely or (a lot more difficult) he could try to do even better. The latter will be very hard though, because “Cold Street Light” is a perfectly unhappy pop-song, beautifully moody...

Source

June 15, 2023

De la merdification des choses

Les vieux ressassent souvent que « c’était mieux avant » et que « tout se désagrège ». Le trope semble éculé. Mais s’il contenait une part de vérité ? Et si, réellement, nous étions dans une période où la plupart des services devenaient merdiques ? Et si le capitalo-consumérime était entré dans sa phase de « merdification » ?

Le terme original « enshitification » a été proposé par l’auteur/blogueur Cory Doctorow qui parle quotidiennement du phénomène sur son blog. Je propose la traduction « merdification ».

Mais qu’est-ce que la merdification ?

Une histoire de business model

Dans notre société capitalo-consumériste, il est nécessaire de gagner de l’argent en proposant un produit pour lequel d’autres sont prêts à payer. Pour le travailleur, c’est son temps et ses compétences. Pour une entreprise, c’est souvent plus complexe et trouver un bon business model est compliqué.

Avec Netscape et la première bulle Internet, fin du millénaire précédent, est apparue une idée nouvelle : plutôt que de faire un vrai business model, l’entreprise va simplement tenter de se faire connaître pour se faire racheter. Soit par une entreprise plus grosse, soit par le public lors d’une introduction en bourse.

L’avantage est que, contrairement à une véritable entreprise qui vend des produits, le délai de rentabilité est beaucoup plus court. Investissez 100 millions dans une entreprise et revendez-la 1 milliard trois ans plus tard !

L’entreprise s’est alors transformée en « startup ». Le but d’une startup n’est pas de proposer un service à des clients ni de faire des bénéfices, le but d’une startup est de grossir et de se faire connaître. L’argent est fourni par des investisseurs qui veulent un retour important et rapide. Ce qu’on appelle les VC (Venture Capitalists).

L’argent de ces VC va permettre à l’entreprise de grossir et d’attirer le prochain round de VC jusqu’au jour où l’entreprise est assez grosse pour attirer l’attention d’un acheteur potentiel. Cette croissance doit se faire tant en nombre d’utilisateurs que d’employés, les deux étant les critères qui intéressent les acheteurs. On utilise le terme « acqui-hire » lorsque le but est de simplement faire main basse sur les employés, leur compétence et le fait qu’ils sont déjà une équipe soudée. Auquel cas, le produit vendu par l’entreprise sera purement et simplement supprimé après quelques mois durant lesquels l’entreprise acheteuse ne cesse de prétendre le contraire. Exemples historiques : rachat de Mailbox par Dropbox, du calendrier Sunrise par Microsoft ou de Keybase par Zoom. Ce qui entraine des situations cocasses comme cet ex-collègue qui, ayant signé un contrat pour rejoindre Sunrise à New York, s’est retrouvé, pour son premier jour de travail, dans un bureau Microsoft à Bruxelles.

Une autre raison pour valoriser une entreprise est son nombre d’utilisateurs (même gratuits, surtout gratuits). L’idée est de récupérer une base d’utilisateurs, des données les concernant et, surtout, de tuer toute éventuelle concurrence. Facebook a racheté Instagram et Whatsapp pour cette simple raison : les produits devenaient très populaires et pouvaient, à terme, faire de la concurrence.

Contrairement à une entreprise « traditionnelle », le but d’une startup est donc de se faire racheter. Le plus vite possible. De lever de l’argent puis de faire ce qu’on appelle un « exit ».

Dans les programmes de coaching de startup, c’est réellement ce qu’on apprend : comment « pitcher » à des investisseurs, comment faire des métriques attractives pour ces investisseurs (les fameux KPI, qui comprennent le nombre de followers sur Twitter et Facebook, je n’invente rien), comment attirer des utilisateurs à tout prix (le « growth hacking ») et comment planifier son exit en étant attractif pour les gros acheteurs. Faire des slides pour investisseurs est désormais plus important que de satisfaire des clients.

Les monopoles sont tellement prégnants dans tous les secteurs que, même dans les écoles de commerce, le but avoué est désormais de faire des entreprises qui soient « vendables » pour les monopoles. J’ai personnellement entendu des « faut pas aller dans telle direction, plus personne ne voudra te racheter après ça ».

Une odeur de Ponzi

Nous avons donc créé une génération de services, en ligne ou non, qui cherchent la croissance à tout prix sans aucun objectif de rentabilité. Ne devant pas être rentables, ces services ont forcément écrasé la concurrence. Uber tente de remplacer les taxis en perdant chaque année des milliards (oui, des milliards) de dollars fournis par les investisseurs (l’Arabie Saoudite dans ce cas-ci) et, de l’aveu même de son rapport annuel aux actionnaires, sans aucun espoir d’être un jour profitable. Amazon a historiquement fait la plupart de ses livraisons à perte afin d’empêcher l’apparition d’un concurrent sérieux. Twitter n’a jamais été profitable.

Ce système ne peut se perpétuer que tant que les investisseurs peuvent revendre, plus cher, à d’autres investisseurs. C’est le principe de la pyramide de Ponzi. Forcément, à la fin, il faut bien des pigeons qui achètent très cher et ne peuvent jamais revendre. Le pigeon idéal reste le particulier d’où l’objectif ultime d’être un jour coté en bourse.

L’arnaque est savamment entretenue grâce à la présence de milliardaires qui font rêver tous les apprentis sorciers du business. S’ils sont milliardaires, c’est que leur business fait des bénéfices plantureux, non ? Non ! Le premier, Marc Andreessen, est devenu milliardaire en revendant Nestcape, une société qui n’a jamais gagné un kopeck. Jeff Bezos n’est pas devenu milliardaire en vendant des livres par correspondances, mais en vendant des actions Amazon. Elon Musk ne gagne pas d’argent en vendant des Teslas, mais bien des actions Tesla. On pourrait même dire que vendre des Tesla n’est qu’une des manières de faire de l’esbroufe afin de faire augmenter le cours de l’action, ce qui est le véritable business de Musk (qui a très bien compris que Twitter était un outil merveilleux pour manipuler les cours de la bourse).

Notons cependant l’originalité de Google et de Facebook. Les deux géants ont en effet développé un business particulier : le fait de vendre des « vues de publicité » pour lesquelles ils ont le contrôle total des métriques. En gros, vous payez ces deux monstres pour afficher X milliers de publicités et, après quelques jours, vous recevez un message qui vous dit « Voilà, c’est fait, votre publicité a reçu X milliers de vue, vous trouverez la facture en pièce jointe » sans aucune manière de vérifier. Mais cette arnaque-là est une autre histoire.

Revenons à notre pyramide de Ponzi : le problème d’une pyramide de Ponzi, c’est qu’elle finit tôt ou tard par craquer. Il n’y a plus assez de pigeons pour entrer dans le jeu. La bourse s’écroule. Les investisseurs rechignent et les individus ont déjà tous des centaines de comptes pour une pléthore de services plus ou moins gratuits, soi-disant financés par la publicité, publicité qui concerne souvent d’autres services ou produits eux-mêmes financés par la publicité.

La société capitalo-monopolistique rentre alors dans une nouvelle phase. Après la croissance infinie, voici le temps de passer à la caisse. Après les promesses, la merdification.

Les techniques de merdification

Le principe de la merdification est simple : maintenant que les utilisateurs sont captifs, que les concurrents ont quasiment disparu, que les business indépendants ont été acculés à la faillite ou rachetés, on peut exploiter l’utilisateur jusqu’au trognon.

Certains groupes d’investisseurs se sont spécialisés dans ces techniques. Cory Doctorow les regroupe sous le terme « Private Equity » (PE). Leur job ? À partir d’un business existant, extraire un maximum d’argent en un minimum de temps, disons entre deux et cinq ans.

Comment ?

Premièrement, en augmentant les tarifs et en supprimant les programmes gratuits. Les utilisateurs sont habitués, migrer vers un autre service est difficile, la plupart vont payer. Surtout si cette hausse est progressive. L’objectif n’est pas d’avoir de nouveaux utilisateurs, mais bien de faire cracher ceux qui sont déjà là. On va donc leur pourrir la vie au maximum : tarifs volontairement complexes et changeant, rebranding absurdes pour justifier de nouveaux tarifs, blocage de certaines fonctionnalités, problèmes techniques empêchant la migration vers un autre service, etc.

En second lieu, on va bien entendu stopper tout investissement dans l’infrastructure ou le produit. Un maximum d’employés vont être licenciés pour ne garder que l’équipage minimal, si possible sous-payé. Le support devient injoignable ou complètement incompétent, la qualité du produit se dégrade tout à fait.

Bref, c’est la merdification.

C’est destructif ? C’est bien l’objectif. Car la véritable astuce est encore plus retorse : fort de son historique et de sa réputation, la société peut certainement obtenir des prêts bancaires. Ces prêts amèneront une manne d’argent qui permettra de payer… les personnes travaillant pour le Private Equity (qui se sont arrogés des postes dans l’entreprise). Certains montages permettent même à l’entreprise de prendre un emprunt pour se racheter elle-même… aux investisseurs. Qui récupèrent donc directement leur mise, tout le reste n’étant plus que du bénéfice.

Une fois que tout est à terre, il ne reste plus qu’à mettre l’entreprise en faillite afin qu’elle soit insolvable. Les utilisateurs sont, de toute façon, déjà partis depuis longtemps.

Les conséquences de la merdification

Si les conséquences pour le client sont évidentes, elles le sont encore plus pour le travailleur. S’il n’a pas été viré, le travailleur doit donc désormais travailler beaucoup plus, dans une infrastructure qui part à vau-l’eau et sans aucune perspective autre que de se faire insulter par les clients.

Les « faux indépendants » (livreurs Deliveroo, chauffeurs Uber, etc.) voient fondre leurs marges alors que les règles, elles, deviennent de plus en plus drastiques et intenables. Le terrifiant spectre du chômage nous fait prendre en pitié les employés forcés de nous fournir des services merdiques. Nous les remercions. Nous leur mettons des étoiles par pitié, parce que sinon ils risquent de se faire virer. Et nous payons pour un service de merde. En l’acceptant avec le sourire. Ou alors nous les engueulons alors qu’ils ne peuvent rien faire.

Le phénomène de merdification n’est pas cantonné aux startups Internet, même s’il y est particulièrement visible. Il explique beaucoup de choses notamment dans la grande distribution, dans le marché de l’emploi, dans la disparition progressive des commerçants indépendants au profit de grandes enseignes. On peut même également le voir à l’œuvre dans le cinéma !

Il y’a des chances que la plupart des films à l’affiche dans votre cinéma soient des reprises ou des continuations de franchises existantes, franchises qui sont exploitées jusqu’au trognon jusqu’à devenir des sous-merdes. Écrire un scénario est désormais un art oublié et chaque film n’a plus qu’un objectif : produire une bande-annonce alléchante. En effet, une fois le ticket acheté et le pigeon assis dans son siège avec son popcorn, rien ne sert de lui fournir quoi que ce soit. Il a déjà payé ! Un peu comme si les films n’étaient plus qu’une version allongée de la bande-annonce. Les séries ne cherchent plus à construire quoi que ce soit vu que chaque série d’épisodes (même plus des saisons entières) n’est tournée que si les précédents ont fait un score minimal de vision. Les histoires sont décapitées avant même de commencer.

La blogueuse Haley Nahman a d’ailleurs analysé une normalisation des couleurs des séries et des films qui pourrait être une conséquence de cette merdification.

Réagir

Prendre conscience de cette merdification, la nommer est une étape importante. Et réaliser que ce n’est pas une fatalité. Ce n’est pas l’incompétence ou la paresse des travailleurs qui est en cause. Il s’agit d’un phénomène volontaire et conscient destiné à soutirer un maximum de revenus de notre infrastructure. Il s’agit d’une étape inéluctable du capitalisme monopolistique dans lequel nous vivons.

Les infrastructures publiques vendues à des entreprises privées ont été une aubaine incroyable pour les merdificateurs. Oui, prendre le train est devenu cher et merdique. Parce que c’est l’objectif : empocher un maximum de bénéfices privés en provenance d’investissements publics. La merdification est une véritable spoliation des biens publics. Cela même pour les entreprises privées qui, très souvent, ont obtenu de l’argent public pour aider à se lancer et à « faire rayonner l’économie de notre belle région » (dixit le ministre qui a voté le budget). Notons que ce type de merdification de l’espace public a toujours existé. Zola l’a parfaitement décrit dans « La curée ».

À titre individuel, il n’y a pas grand-chose à faire si ce n’est tenter de soutenir les petites entreprises, les commerces indépendants, ceux qui vivent de la satisfaction de leur clientèle. Et faire attention à ne pas se laisser enfermer dans des services commerciaux qui, si alléchants soient-ils, n’ont d’autres choix que de disparaitre ou se merdifier.

Mais ne nous voilons pas la face, ce n’est pas près de s’arrêter. Certains psychopathes semblent avoir comme objectif de merdifier la planète entière pour accroitre leur profit. Et, jusqu’à présent, rien ne semble pouvoir les arrêter.

Ingénieur et écrivain, j’explore l’impact des technologies sur l’humain. Abonnez-vous à mes écrits en français par mail ou par rss. Pour mes écrits en anglais, abonnez-vous à la newsletter anglophone ou au flux RSS complet. Votre adresse n’est jamais partagée et effacée au désabonnement.

Pour me soutenir, achetez mes livres (si possible chez votre libraire) ! Je viens justement de publier un recueil de nouvelles qui devrait vous faire rire et réfléchir.

June 10, 2023

Il n’est plus possible de faire de la philosophie sans faire de la science-fiction

Au détour d’une conversation à Épinal, l’auteur et philosophe Xavier Mauméjean me glissa cette phrase curieuse : « Aujourd’hui, il n’est plus possible de faire de la philosophie sans faire de la science-fiction ».

Interpelé, je retournai des jours durant cette phrase dans mon esprit avant que l’évidence ne m’apparût.

Le présent a l’épaisseur mathématique d’une droite, la consistance d’un point. Il est insaisissable, mouvant. À ce titre, il n’existe pas de littérature du présent. L’humain ne peut écrire que sur deux sujets : le passé et le futur. Les deux étant complémentaires.

Lire sur le passé nous édifie sur la nature humaine, sur notre place dans le monde, dans la civilisation. Cela démystifie, et c’est essentiel, notre univers. Le passé nous enseigne les lois scientifiques.

Se projeter dans le futur nous fait réfléchir aux conséquences de nos actes, nous fait peser nos choix. Or il n’y a pas de littérature du futur sans imaginaire. Le futur n’est, par définition, qu’imagination. Un imaginaire qui obéit à des lois, les lois scientifiques susnommées. Réfléchir au futur, c’est donc faire de la science-fiction.

La science-fiction, sous toutes ses formes, est la clé de notre capacité d’influencer le monde, l’essence même de notre survie.

Mais attention aux étiquettes. Il serait tentant de penser qu’un livre se passant dans le passé parle du passé et un livre se passant dans le futur parle du futur. C’est bien entendu simpliste et trompeur. Tant de livres historiques nous emmènent à réfléchir à notre futur, à notre être et à notre devenir. Un livre peut se passer en l’an 3000 et ne brasser que du vent.

Malgré son importance vitale, la science-fiction a toujours mauvaise presse, est reléguée aux étagères les moins accessibles des librairies, est rejetée par les lecteurs.

Refuser l’étiquette « science-fiction » n’est-il pas le symptôme d’une peur de se projeter dans le futur ? D’affronter ce qui nous semble inéluctable ? Mais tant que nous aurons de l’imagination, rien ne sera inéluctable. Le futur n’a qu’une constante : il est la conséquence de nos actions.

Pour reprendre les mots de Vinay Gupta, le futur est un pays étranger. Un pays vers lequel nous nous contentons aujourd’hui d’envoyer nos déchets, un pays dont nous tentons de détruire les ressources, comme si nous étions en guerre. Un pays où vivent nos enfants.

Peut-être est-il temps de faire la paix avec le futur. D’entretenir de bonnes relations diplomatiques. Des relations épistolaires qui portent un nom : la science-fiction.

Mais peut-être ce nom est-il trompeur. Peut-être que la science-fiction n’existe pas. Peut-être que toute littérature est en soi, un ouvrage de science-fiction.

On ne peut écrire sans philosopher. On ne peut philosopher sans faire de la science-fiction. On ne peut être humain sans faire la paix avec le futur.

Ingénieur et écrivain, j’explore l’impact des technologies sur l’humain. Abonnez-vous à mes écrits en français par mail ou par rss. Pour mes écrits en anglais, abonnez-vous à la newsletter anglophone ou au flux RSS complet. Votre adresse n’est jamais partagée et effacée au désabonnement.

Pour me soutenir, achetez mes livres (si possible chez votre libraire) ! Je viens justement de publier un recueil de nouvelles qui devrait vous faire rire et réfléchir.

June 09, 2023

As I blogged before, I've been working on a Planet Venus replacement. This is necessary, because Planet Venus, unfortunately, has not been maintained for a long time, and is a Python 2 (only) application which has never been updated to Python 3.

Python not being my language of choice, and my having plans to do far more than just the "render RSS streams" functionality that Planet Venus does, meant that I preferred to write "something else" (in Perl) rather than updating Planet Venus to modern Python.

Planet Grep has been running PtLink for over a year now, and my plan had been to update the code so that Planet Debian could run it too, but that has been taking a bit longer.

This month, I have finally been able to work on this, however. This screenshot shows two versions of Planet Debian:

The rendering on the left is by Planet Venus, the one on the right is by PtLink.

It's not quite ready yet, but getting there.

Stay tuned.