Planet Grep

Planet'ing Belgian FLOSS people

Planet Grep is maintained by Wouter Verhelst. All times are in UTC.

October 24, 2021

This post is meant to encourage me to read a bit more (paper books). By the way, I thought I was reading four books simultaneously, but when I put them next to each other it turned out to be seven.

Form left to right (writer-title(year) pages read-total pages):

Daniele Benedettelli - Creating Cool Mindstorms NXT Robots(2008) 24-575
Leo Tolstoj - Oorlog en Vrede(1869, NL translation 1973) 115-462
Dirk De Wachter - De kunst van het ongelukkig zijn(2019) 35-101
Allen/Fonagy/Bateman - Mentaliseren(2008/2019 edition) 30-368
LEGO and philosophy(2017) 56-226
Charlie Mackesy - The Boy, the mole, the fox and the horse(2019)
Michael Collins - Carrying the Fire(1974) - finished

Personal goal: finish at least three more of these before 2022.

I just finished Michael Collins - Carrying the Fire and it took me five weeks, which I consider a bit too long for a 470-ish pages book. It was a very good book though. If you're into space travel, then I would definitely recommend it. It also proves how the Sixties was vastly different to today, for example of the fourteen astronauts selected in 1963, four died during training. Two of the nine also died. Such numbers are unacceptable in 2021, even for 'dangerous' jobs.

The Daniele Benedettelli book is about programming Finite State Machines using Lego robots. I don't know much about programming, but this looks like fun. Thing is I need to build some Lego robots (like this one) to continue this book.

And I probably need to start from page 1 again in War and Peace because of the many characters that I forgot.

Some other books that I read the past three years are:

Celestin-Westreich/Celestin - Observeren en Rapporteren
Dick Swaab - Ons creatieve brein
Dirk De Wachter - Borderline Times
Dirk De Wachter - De wereld van De Wachter
Etienne Vermeersch - Over God
Etienne Vermeersch - Provencaalse gesprekken
Jan Van de Craats - Basisboek wiskunde
Jude Woodward - The US vs China
Paul Verhaeghe - Autoriteit
Paul Verhaeghe - Identiteit
Randall Munroe - Thing Explainer
Randall Munroe - What If
Rebecca Smethurst - Space, 10 things you should now
Robert Bly - De Wildeman
Terry Goodkind - Law of Nines
Terry Goodkind - Severed Souls
Terry Goodkind - The first Confessor
Terry Goodkind - The Omen Machine
Terry Goodkind - The Third Kingdom
Terry Goodkind - Warheart
Thomas D'ansembourg - Stop met aardig zijn

Most of it non-fiction apparently. I really enjoyed 'Borderline Times' and both books of Paul Verhaeghe and Dick Swaab. I couldn't really get into Robert Bly or Thomas D'Ansembourg (but collaborative communication gives me a lot of insight in people).

October 21, 2021

FOSDEM 2022 will take place on Saturday 5 and Sunday 6 February 2022. It will be an online event. After several long and passionate debates, we have decided to make FOSDEM 2022 an online event. There are a lot of good arguments in favor of physical, hybrid, and online events each. We do not wish to rehash all of them in public, so please understand if we do not engage in public debates on their relative merits. This was not an easy decision, but at least it's a decision. Similar to CCC, we would prefer something else, but it舰

October 20, 2021

For the past few years, I've examined Drupal.org's contribution data to understand how the Drupal project works. Who develops Drupal? How diverse is the Drupal community? How much of Drupal's maintenance and innovation is sponsored? Where do sponsorships come from?

The report might be of interest even if you don't use Drupal. It provides insights into the inner workings of one of the largest Open Source projects in the world.

This year's report shows that:

  • Compared to last year, we have fewer contributions and fewer contributors. The slowdown is consistent across organizations, countries, project types, and more. I believe this is the result of COVID-19, where we are in the Drupal Super Cycle, and many Drupal shops being too busy growing.
  • Despite a slowdown, it's amazing to see that just in the last year, Drupal welcomed more than 7,000 individual contributors and over 1,100 corporate contributors.
  • Two-thirds of all contributions are sponsored, but volunteer contributions remain important to Drupal's success.
  • Drupal's maintenance and innovation depends mostly on smaller Drupal agencies and Acquia. We don't see many contributions from hosting companies, multi-platform digital agencies, system integrators, or end users.
  • Drupal's contributors have become more diverse, but are still not diverse enough.

For comparison, you can also look at the 2016 report, 2017 report, 2018 report, 2019 report, and the 2020 report.

Methodology

What data did I analyze?

I looked at all Drupal.org issues marked "closed" or "fixed" in the 12-month period from July 1, 2020 to June 30, 2021. This is across issues in Drupal Core and all contributed projects, including all major versions of Drupal.

What are Drupal.org issues?

Each "Drupal.org issue" tracks an idea, feature request, bug report, task, or more. It's similar to "issues" in GitHub or "tickets" in Jira. See https://www.drupal.org/project/issues for the list of all issues.

What are Drupal.org credits?

In the spring of 2015, I proposed some ideas for how to give credit to Drupal contributors. A year later, Drupal.org added the ability for contributors to attribute their work to an organization or customer sponsor, or mark it the result of volunteer efforts.

Example issue credit on drupal org
A screenshot of an issue comment on Drupal.org. You can see that jamadar worked on this patch as a volunteer, but also as part of his day job working for TATA Consultancy Services on behalf of their customer, Pfizer.

Drupal.org's credit system is unique and groundbreaking within the Open Source community. It provides unprecedented insights into the inner workings of a large Open Source project. There are a few limitations with this approach, which I'll address at the end of this report.

How is the Drupal community doing?

In the 12-month period between July 1, 2020 and June 30, 2021, Drupal.org's credit system received contributions from 7,420 different individuals and 1,186 different organizations. We saw a 10% decline in individual contributors, and a 2% decrease in organizational contributors.

Contributions by individuals vs organizations

For this report's time period, 23,882 issues were marked "closed" or "fixed", a 23% decline from the 2019-2020 period. This averages out to 65 issues marked "closed" or "fixed" each day.

In total, the Drupal community worked on 3,779 different Drupal.org projects this year compared to 4,195 projects in the 2019-2020 period — a 10% year-over-year decline.

Metric 2019 - 2020 2020 - 2021 Delta
Number of individual contributors 8,303 7,420 -12%
Number of organizational contributors 1,216 1,186 -2%
Number of issues "fixed" or "closed" 31,153 23,882 -23%
Number of projects worked on 4,195 3,779 -10%

Understanding the slowdown in contribution

Individual contributors slowed down

To understand the slowdown, I looked at the behavior of the top 1,000 contributors:

  • The top 1,000 individual contributors are responsible for 65% of all contributions. The remaining 6,420 individuals account for the remaining 35%. Overall, Drupal follows a long tail model.
  • In the last year, 77 of the top 1,000 individual contributors stopped contributing to Drupal, 671 contributed less, and 252 contributed more.

A 7.7% annual attrition rate in the top 1,000 contributors is very low. It means that the average contributor in the top 1,000 is active for 13 years. In other words, Drupal's top 1,000 contributors are extremely loyal — we should be grateful for their contributions and continued involvement in the Drupal project.

While we can't compare Open Source projects like Drupal to commercial companies, it might be useful to know that most commercial organizations are very happy with an attrition rate of 15% or less. This means that an employee stays with their employer for almost 6.5 years. Nowadays, a lot of people don't stay with their employer for that long. When it’s put that way, you can see that an attrition rate of 7.7% is very good!

The big takeaway is that the top individual and organizational contributors aren't leaving Drupal. They just became less active in 2020-2021.

Organizational contributors also slowed down

Next, I looked at the behavior of the top 250 organizations:

  • The top 250 organizational contributors are responsible for 82% of all contributions. The other 936 organizations account for the remaining 18%.
  • In the last year, 8 organizations (3%) stopped contributing, 168 (67%) contributed less, and 74 (30%) contributed more.
  • Five of the 8 organizations that stopped contributing were end users; they most likely switched their website away from Drupal. The remaining 3 were digital agencies. The end user attrition rate in the top 250 was 2%, while the digital agency attrition rate was 0.4%.

The top Drupal agencies remain very committed to Drupal. While many agencies contributed less, very few agencies stopped contributing to Drupal altogether.

Why are individuals and organizations contributing less?

As part of my research, I reached out to some of the top contributing Drupal agencies. The main reason why they are contributing less is that they are too busy growing:

  • We grew 33% so far in 2021. We have grown our contribution as well, but there has been a shift from code contributions to non-code contributions. We've contributed less code because Drupal has all the features we need to deliver amazing digital experiences, and has become really stable and robust. There has been less code to contribute. — Baddý Sonja Breidert, CEO of 1xINTERNET, Germany
  • We have grown 35% in the last year — from around 65 employees to 90. — Nick Veenhof, CTO of DropSolid, Belgium
  • Customer investment in digital has accelerated by several years the past 12 months. We grew our Drupal practice by 35% in the past year. — Paul Johnson, Drupal Director at CTI Digital, UK
  • We grew 27% in revenue last year. We expect to continue on that growth trajectory. Our only concern is shortage of Drupal talent. — Janne Kalliola, CEO of Exove, Finland
  • We grew 40% over the last year. This has been driven by an increased demand for large Drupal projects on tight deadlines. With more time pressures from clients and changing personal commitments, it’s been more difficult for people to find the time to contribute. But also, more of our contribution shifted from Drupal.org to GitHub, and doesn't use the credit system. — Stella Power, Managing Director of Annertech, Ireland
  • We experienced unexpected sales growth during COVID. Thanks to Drupal Commerce, we grew 95% in 2020 and 25% year to date. In addition, two of our leading contributors pursued other opportunities. As new team members get onboarded and the workload stabilizes, I'm hopeful we see our overall contributions increase again in 2022. — Ryan Szrama, CEO of Centarro, United States

It's great to see so many Drupal agencies doing well.

Other than being too busy with client work, the following secondary reasons were provided:

  • Drupal is a stable and mature software project. Drupal has all the features we need to deliver ambitious digital experiences. Furthermore, Drupal has never been this stable and robust; we don't have many bug fixes to contribute, either.
  • There is a shortage of Drupal talent; the people we hire don't know how to contribute yet.
  • COVID eliminated in-person events and code sprints. In-person events inspired our employees to contribute and collaborate. Without in-person events, it's hard to instill employees with a passion to contribute.
  • It's more difficult to teach new employees how to contribute when everyone is remote.
  • People want a vision for Drupal that they can rally behind. We have already achieved the vision: Drupal is for ambitious digital experiences. People want to know: what is next?
  • The tools and processes to contribute are becoming more complex; contribution has become more difficult and less desirable.
  • We are getting more efficient at managing major Drupal releases. Rector automates more and more of the upgrade work. When we work smarter, contribution drops.

There is no doubt that COVID has accelerated a lot of digital transformation projects, but it has also slowed down contribution. Parents are busy home-schooling their children, people have Zoom-fatigue, some families may have lost income, etc. COVID added both stress and extra work to people's lives. For many, this made contribution more difficult or less possible.

Drupal Super Cycle

Drupal agencies provided many valid reasons for why contribution is down. In addition to those, I believe a Drupal Super Cycle might exist. The Drupal Super Cycle is a new concept that I have not talked about before. In fact, this is just a theory — and only time will tell if it is valid.

The Drupal Super Cycle is a recognition that Drupal's development cycle ebbs and flows between a "busy period" and "quiet period" depending on when the next major release takes place. There is a "busy period" before a major release, followed by a "quiet period" after each major release.

Major Drupal releases only happen every 2 or 3 years. When a major release is close, contributors work on making their projects compatible. This requires extra development work, such as adopting new APIs, subsystems, libraries, and more. Once projects are compatible, the work often shifts from active development to maintenance work.

A visual representation of the Drupal Super Cycle; contribution accelerates just before a major release and slows down after.
A slide from the my DrupalCon Europe 2021 keynote where I explain the Drupal Super Cycle theory.

The last major Drupal release was Drupal 9, released in June of 2020. Last year's report analyzed contribution activity between July 1, 2019 and June 30, 2020. This period includes the 11-month period leading up to the Drupal 9 release, the Drupal 9 release itself, and 1 month after the Drupal 9 release. It's the "busy period" of the Super Cycle because the Drupal community is getting thousands of contributed modules ready for Drupal 9.

This year's report analyzes contribution data starting 1 month after the Drupal 9 release. There was no major Drupal release this year, and we are still 9 to 14 months away from Drupal 10, currently targeted for the summer of 2022. We are in the "quiet period" of the Super Cycle.

If the Drupal Super Cycle concept is valid, we should see increased activity in next year's report, assuming we remain on track for a Drupal 10 release in June of 2022. Time will tell!

What is the community working on?

Contribution credits decreased across all project types, but increased for Drupal Core.

A graph showing the year over year growth of contributions per project type: only contributions to core grew

Core contributions saw a 7% year-over-year increase in credits, while work on contributed projects — modules, themes and distributions — are all down compared to last year.

Who are Drupal's top individual contributors?

The top 30 individual contributors between July 1, 2020 and June 30, 2021 are:

A graph showing the top 30 individual contributors ranked by the quantity of their contributions.
A graph showing the top 30 individual contributors ranked by the impact of their contributions.

For the weighted ranking, I weighed each credit based on the adoption of the project the credit is attributed to. For example, each contribution credit to Drupal Core is given a weight of 10, because Drupal Core has about 1 million active installations. Credits to the Webform module, which has over 450,000 installations, get a weight of 4.5. And credits to Drupal's Commerce project get 0.5 points, as it is installed on around 50,000 sites.

The weighting algorithm also makes adjustments for Drupal's strategic initiatives. Strategic initiatives get a weight of 10, the highest possible score, regardless of whether these are being developed in Drupal Core's Git repository or in a sandbox on Drupal.org.

The idea is that these weights capture the end user impact of each contribution, but also act as a proxy for the effort required to get a change committed. Getting a change accepted in Drupal Core is both more difficult and more impactful than getting a change accepted to a much smaller, contributed project.

This weighting is far from perfect, but so is the unweighted view. For code contributions, the weighted chart may be more accurate than a purely unweighted approach. I included both charts:

No matter how you look at the data, all of these individuals put an incredible amount of time and effort into Drupal.

It's important to recognize that most of the top contributors are sponsored by an organization. We value the organizations that sponsor these remarkable individuals. Without their support, it could be more challenging for these individuals to contribute.

How much of the work is sponsored?

When people contribute to Drupal, they can tag their contribution as a "volunteer contribution" or a "sponsored contribution". Contributions can be marked both volunteer and sponsored at the same time (shown in jamadar's screenshot near the top of this post). This could be the case when a contributor does paid work for a customer, in addition to using unpaid time to add extra functionality or polish.

For those credits with attribution details, 16% were "purely volunteer" (7,034 credits). This is in stark contrast to the 68% that were "purely sponsored" (29,240 credits). Put simply, roughly two-thirds of all contributions are "purely sponsored". Even so, volunteer contribution remains very important to Drupal.

A graph showing how many of the contributions are volunteered vs sponsored.

Volunteers contribute across all areas of the project. A lot of volunteer time and energy goes towards non-product related contributions such as event organization, mentoring, and more. Non-code contributions like these are very valuable, yet they are under-recognized in many Open Source communities.

Contributions by project type

Who are Drupal's top organizational contributors?

Similar to the individual contributors, I've ranked organizations by both "unweighted contributions" and "weighted contributions". Unweighted scores are based solely on volume of contributions, while weighted scores also try to take into account both the effort and impact of each contribution.

A graph showing the top 30 organizational contributors ranked by the quantity of their contributions.
A graph showing the top 30 organizational contributors ranked by the impact of their contributions.

If you are an end user looking for a company to work with, these are some of the companies I'd work with first. Not only do they know Drupal best, but they also help improve your investment in Drupal. If you are a Drupal developer looking for work, these are some of the companies I'd apply to first.

A variety of different types of companies are active in Drupal's ecosystem:

Category Description
Traditional Drupal businesses Small-to-medium-sized professional services companies that primarily make money using Drupal. They typically employ fewer than 100 employees. Because they specialize in Drupal, many of these companies contribute frequently and are a huge part of our community. Examples are Third and Grove, OpenSense Labs, Srijan, etc.
Digital marketing agencies Larger full-service agencies that have marketing-led practices using a variety of tools, typically including Drupal, Adobe Experience Manager, Sitecore, WordPress, etc. Many of these larger agencies employ thousands of people. Examples are Wunderman Thompson, Possible, and Mirum.
System integrators Larger companies that specialize in bringing together different technologies into one solution. Example system integrators are Accenture, TATA Consultancy Services, EPAM Systems, and CI&T.
Hosting companies Examples are Acquia, Pantheon, and Platform.sh, but also Rackspace or Bluehost.
End users Examples are the European Commission or Pfizer.

A few observations:

  • Most of the sponsors in the top 30 are traditional Drupal businesses with fewer than 100 employees. With the exception of Acquia, Drupal's maintenance and innovation largely depends on these small Drupal businesses.
  • The larger, multi-platform digital marketing agencies are barely contributing to Drupal. Only 1 digital marketing agency shows up in the top 30: Intracto with 410 credits. Hardly any appear in the entire list of contributing organizations. I'm frustrated that we have not yet found the right way to communicate the value of contribution to these companies. We need to incentivize these firms to contribute with the same level of commitment that we see from traditional Drupal businesses.
  • The only system integrator in the top 30 is CI&T with 1,177 credits. CI&T is a smaller system integrator with approximately 5,200 employees. We see various system integrators outside of the top 30, including EPAM Systems (138 credits), TATA Consultancy Services (109 credits), Publicis Sapient (60 credits), Capgemini (40 credits), Globant (8 credits), Accenture (2 credits), etc.
  • Various hosting companies make a lot of money with Drupal, yet only Acquia appears in the top 30 with 1,263 credits. The contribution gap between Acquia and other hosting companies remains very large. Pantheon earned 71 credits compared to 122 last year. Platform.sh earned 8 credits compared to 23 in the last period. In general, there is a persistent problem with hosting companies not contributing back.
  • We only saw 1 end user in the top 30 this year: Thunder (815 credits). Many end users contribute though: European Commission (152 credits), Pfizer (147 credits), bio.logis (111 credits), Johnson & Johnson (93 credits), University of British Columbia (105 credits), Georgia Institute of Technology (75 credits), United States Department of Veterans Affairs (51 credits), NBCUniversal (45 credits), Princeton University (43 credits), Estée Lauder (38 credits), University of Texas at Austin (22 credits), and many more.
A graph showing that Acquia is by far the number one contributing hosting company.
A graph showing that CI&T is by far the number one contributing system integrator.

I often recommend end users to mandate contributions from their partners. Pfizer, for example, only works with agencies that contribute back to Drupal. The State of Georgia started doing the same; they made Open Source contribution a vendor selection criteria. If more end users took this stance, it could have a big impact on Drupal. We'd see many more digital agencies, hosting companies, and system integrators contributing to Drupal.

While we should encourage more organizations to sponsor Drupal contributions, we should also understand and respect that some organizations can give more than others — and that some might not be able to give back at all. Our goal is not to foster an environment that demands what and how others should give back. Instead, we need to help foster an environment worthy of contribution. This is clearly laid out in Drupal's Values and Principles.

How diverse is Drupal?

Supporting diversity and inclusion is essential to the health and success of Drupal. The people who work on Drupal should reflect the diversity of people who use the web.

I looked at both the gender and geographic diversity of Drupal.org contributors.

Gender diversity

While Drupal is slowly becoming more diverse, less than 9% of the recorded contributions were made by contributors who do not identify as men. The gender imbalance in Drupal remains profound. We need to continue fostering diversity and inclusion in our community.

A graph showing contributions by gender: 67% of the contributions come from people who identify as male.

A few years ago I wrote a post about the privilege of free time in Open Source. I made the case that Open Source is not a meritocracy. Not everyone has equal amounts of free time to contribute. For example, research shows that women still spend more than double the time as men doing unpaid domestic work, such as housework or childcare. This makes it more difficult for women to contribute to Open Source on an unpaid, volunteer basis. Organizations capable of giving back should consider financially sponsoring individuals from underrepresented groups to contribute to Open Source.

A graph that shows that compared to males, female contributors do more sponsored work, and less volunteer work.
Compared to men, women do more sponsored work, and less volunteer work. We believe this is because men have the privilege of more free time.

Free time being a privilege is just one of the reasons why Open Source projects suffer from a lack of diversity.

The gender diversity chart above shows that there is a growing number of individuals that no longer share their gender identity on Drupal.org. This is because a couple of years ago, the gender field on Drupal.org profile was deprecated in favor of a Big 8/Big 10 demographics field.

Today, over 100,000 individuals have filled out the new "Big 8/Big 10" demographics field. The new demographics field allows for more axes of representation, but is also somewhat non-specific within each axis. Here are the results:

A graph showing different axes of diversity in Drupal

Diversity in leadership

Drupal.org recently introduced the ability for contributors to identify what contributor roles they fulfill. The people who hold these key contribution roles can be thought of as the leaders of different aspects of our community, whether they are local community leaders, event organizers, project maintainers, etc. As more users begin to fill out this data, we can use it to build a picture of the key contributor roles in our community. Perhaps most importantly, we can look at the diversity of individuals who hold these key contributor roles. In next year's report we will provide a focused picture of diversity in these leadership positions.

Geographic diversity

We saw individual contributors from 6 continents and 121 countries. Consistent with the trends described above, most countries contributed less compared to a year earlier. Here are the top countries for 2020-2021:

 A graph showing the top 20 contributing countries in 2021.
The top 20 countries from which contributions originate. The data is compiled by aggregating the countries of all individual contributors behind each issue. Note that the geographical location of contributors doesn't always correspond with the origin of their sponsorship. Wim Leers, for example, works from Belgium, but his funding comes from Acquia, which has the majority of its customers in North America. Wim's contributions count towards Belgium as that is his country of residence.

Europe contributes more than North America. However, contribution from Europe continues to decline, while all other continents have become more active contributors.

A graph that shows most contributions in 2021 come from Europe and North America.

Asia, South America, and Africa remain big opportunities for Drupal; their combined population accounts for 6.3 billion out of 7.5 billion people in the world.

Limitations of the credit system

It is important to note a few of the current limitations of Drupal.org's credit system:

  • The credit system doesn't capture all code contributions. Parts of Drupal are developed on GitHub rather than Drupal.org. Contributions on GitHub usually aren't credited on Drupal.org. For example, a lot of the work on the Automatic Updates initiative is happening on GitHub instead of Drupal.org, and companies like Acquia and Pantheon don't get credit for that work.
  • The credit system is not used by everyone. Because using the credit system is optional, many contributors don't. For example, while they could, not all event organizers and speakers capture their work in the credit system. As a result, contributions often have incomplete or no contribution credits. Where possible, we should automatically capture credits. For example, translation efforts on https://localize.drupal.org are not currently captured in the credit system, but could be automatically.
  • The credit system doesn't accurately value complexity and quality. One person might have worked several weeks for just 1 credit, while another person might receive a credit for 10 minutes of work. Each year we see a few individuals and organizations trying to game the credit system. In this post, I used a basic weighting system based on project adoption. In future, we should consider refining that by looking at issue priority, patch size, number of reviews, etc. This could help incentivize people to work on larger and more important problems and save smaller issues, such as coding standards improvements, for new contributor sprints.

Because of these limitations, the actual number of contributions and contributors could be much higher than what we report.

Conclusions

While we have fewer contributions and fewer contributors compared to last year, it is not something to be worried about. We can attribute this to various things, such as COVID-19, agency growth, and the Drupal Super Cycle.

Our data confirms that Drupal is a vibrant community full of contributors who are constantly evolving and improving the software. It's amazing to see that just in the last year, Drupal welcomed more than 7,000 individual contributors and over 1,100 corporate contributors.

To grow and sustain Drupal, we should support those that contribute to Drupal and find ways to get those that are not contributing involved in our community. We are working on several new ways to make it easier for new contributors to get started with Drupal, which I covered in my latest DrupalCon keynote. Improving diversity within Drupal is critical, and we should welcome any suggestions that encourage participation from a broader range of individuals and organizations.

Special thanks to Tim Lehnen, CTO at the Drupal Association, for supporting me during my research.

October 19, 2021

Transparent encryption is relatively easy to implement, but without understanding what it actually means or why you are implementing it, you will probably make the assumption that this will prevent the data from being accessed by unauthorized users. Nothing can be further from the truth.

Listing the threats to protect against

Let's first list the threats you want to protect against. It is beneficial that these threats are also scored in the organization for their likelihood of occurrence and effect, so that you can optimize and prioritize the measures appropriately.

  • Data leakage through theft or loss of storage media
  • Data leakage through unauthorized data access (OS level)
  • Data leakage through unauthorized data access (middleware/database level)
  • Data leakage through application vulnerability (including injection attacks)
  • Loss of confidentiality through data-in-transit interception
  • Loss of confidentiality through local privilege escalation

While all the "data leakage" threats are also about loss of confidentiality, and any loss of confidentiality can also result in data leakage, I made the distinction in name as the data intercepted through that threat is generally not as 'bulky' as the others.

To visualize the threats, consider the situation of an application that has a database as its backend. The application is hosted on a different system than the database. In the diagram, the blue color indicates an application-specific focus. This does not mean it isn't infrastructure oriented anymore, but more that it can't be transparently implemented without the application supporting it.

Application and database interaction

There are eight roles listed (well, technically seven roles but let's keep it simple and make "physical access" also a role), ranging from the application user to the physical access:

  • The application user interacts with the application itself, for instance from a browser to the web application.
  • The application administrator also interacts with the application, but has more privileges. The user might also have access to the system on which the application itself resides (but that isn't further modelled here).
  • The network poweruser is a user that has access to the network traffic between the client and application, as well as to the network traffic between the application and the database. Depending on the privileges of the users, these powerusers can be administrators on systems that reside in the same network.
  • The database / middleware user is a role that has access to the application data in the database directly (so not (only) through the application). This can commonly be a supporting function in the organization.
  • The database / middleware administrator is the administrator of the database engine (or other middleware component that is used).
  • The system administrator is the administrator for the server on which the database is hosted.
  • The system user is an unprivileged user that has access to the server on which the database is hosted.
  • The physical access is a role that has physical access to the server and storage.

Further, while the example is easiest to understand with a database system, be aware that there exist many other middleware services that manage data (like queueing systems) and the same threats and measurements apply to them as well.

Transparent encryption is a physical medium data protection measure

Transparent encryption, such as through LUKS (with DM-Crypt) on Linux, will encrypt the data on the disks, while still presenting the data unencrypted to the users. All users. Its purpose hence is not to prevent unauthorized users from accessing the data directly, but to prevent the storage media to expose the data if the media is leaked or lost.

Transparent Disk Encryption

In the diagram, you notice that the transparent disk encryption only takes effect between the server and its storage. Hence, the only 'inappropriate' access that it is mitigating is the physical access to the server storage. Note that physical access to the server itself is still an important attack vector that isn't completely mitigated here - attackers with physical access to servers will not have a too hard time to find an entrypoint to the system. Advanced attackers might even be able to capture the data from memory without being detected.

Transparent disk encryption is very sensible when dealing with removable media (like USB sticks), especially if they contain any (possible) confidential data and the method for transparent encryption is supported on all systems where you are going to use the removable media. In larger enterprises, it also makes sense to apply as well when multiple teams or even companies have physical access and could attempt to maliciously access the systems.

For server disks or SAN storage for instance, this has to be balanced against the downsides of the encryption. You can do disk encryption from the storage array for instance, but this might impact the array's capability for deduplication and compression. If your data centers are highly secured, and you do not allow the storage media to leave the premises without being properly wiped or destroyed, then such transparent encryption imo has little value.

Of course, when you have systems hosted in third party locations, then you do have a higher risk that the media are being removed or stolen, especially if those locations are accessed by many others, and your own space isn't physically further protected. So while a company-controlled data center with tight access requirements, policies and controls that no media leaves the premises and what not could easily evaluate to not apply transparent disk encryption, using a public cloud service or a non-private colocation facility should assess encryption capabilities on disk (and higher).

Furthermore, a properly configured database system will not expose its data to unauthorized users to start with, so the system user role should not have access to the data. But once you have local access to a system, there is always the threat that a privilege escalation bug is triggered that allows the (previously lower privileged) user to access protected files.

Transparent database encryption isn't that much better

Some database technologies (or more general middleware) offer transparent encryption themselves. In this case, the actual database files on the system are encrypted by the database engine, but the database users still see the data as it is unencrypted.

Transparent Database Encryption

Here again, it is important to know what you are protecting yourself from. Transparent database/middleware encryption does prevent the non-middleware administrators from directly viewing the data through the files. However, system administrators generally have the means to become the database (or middleware) administrator, so while the threat is not direct, it is still indirectly there.

The threat of privilege escalation on the system level is partially mitigated. While a full system compromise will lead to the system user getting system administrator privileges, partial compromise (such as receiving access to the data files, but not to the encryption key itself, or not being able to impersonate users but just access data) will be mitigated by the transparent database encryption.

Important to see here is that the threats related to the physical access are also mostly mitigated by the transparent database encryption, with the exception that database-only encryption might result in the encryption key being leaked if it is situated on the system storage.

Most of the threats however are still not mitigated: network interception (if it doesn't use a properly configured TLS channel), admin access, database user access, application admin and application users (through application vulnerability) can still get access to all that data. The only focus these measures have is data loss through physical access.

Database or middleware supported, application-driven encryption is somewhat better

Some database technologies support out-of-the-box data encryption through the appropriate stored procedures or similar. In this case, the application itself is designed to use these encryption methods from the database (or middleware), and often holds the main encryption key itself (rather than in the database).

Database or middleware supported data encryption

While this prevents some of the attack vectors (for instance, some attacks against the application will not result in getting a context that is able to decrypt the data) and mitigates the attack vectors related to direct database user access, there are still plenty of issues here.

System administrators and database administrators are still able to control the encryption/decryption process. Sure, it becomes harder and requires more thought and expertise (like modifying the stored procedures to also store the key or the data in a different table for them to access), but it remains possible.

Because of the attack complexity, this measure is one that starts to meet certain expectations. And because the database or middleware is still responsible for the encryption/decryption part, it can still use its knowledge of the data for things like performance tuning.

Application-managed data encryption is a highly appreciated measure

With application-managed data encryption, the application itself will encrypt and decrypt the data even before it is sent over to the database or middleware.

Application-managed data encryption

With this measure, many of the threats are mitigated. Even network interception is partially prevented, as the network interception now is only still possible to obtain data between the client and the application, and not between the application and database. Also, all roles that are not application-related will no longer be able to get to the data.

Personally, I think that application-managed data encryption is always preferred over the database- or middleware supported encryption methods. Not only does it remove many threats, it is also much more portable, as you do not need a database or middleware that supports it (and thus have to include logic for that in the application).

Of course, applications will need to ensure that they can still use the functionalities of the database and middleware appropriately. If you store names in the database in an encrypted fashion, it is no longer possible to do a select on its content appropriately.

Client-managed data encryption

The highest level of protection against the threats listed, but of course also the most impactful and challenging to implement, is to use client-managed data encryption.

Client-managed data encryption

A web application might for instance have a (properly designed) encryption method brought to the browser (e.g. using javascript), allowing the end user to have sensitive data be encrypted even before it is transmitted over the network.

In that case, none of the attack vectors will be able to obtain the data. Of course, there are plenty of other attack vectors (protecting web applications is an art by itself), but for those we covered, client-managed encryption does tick many of the boxes.

However, client-managed data encryption is also very complex to do securely while still being able to fully support the users. Most applications that employ this focus already on sensitive material (like password managers) and use end user provided information to generate the encryption keys. You need to be able to deal with stale versions (old javascript libraries), multitude of browsers (if it is browser-based), vulnerabilities within browsers themselves and the web application, etc.

Network encryption

Network encryption (as in the use of TLS encrypted communications) only focuses on the confidentiality and integrity of the communication, in our example towards the network poweruser that might be using network interception.

Network encryption

While the majority of other threats are still applicable, I do want to point out that network encryption is an important measure against other threats. For instance, with network encryption, attackers cannot easily inject code or data in existing flows. In case of the client-managed data encryption approach for instance, the use of network encryption is paramount, as otherwise an 'in the middle' attacker can just remove the client-side encryption part of the code that is transmitted.

Conclusions

I hope that this article provides better insights in when transparent encryption is sensible, and when not. With the above assessment, it should be obvious that transparent (and thus without any application support) encryption methods do not cover all the threats out there, and it is likely that your company already has other means to cover the threats that it does handle.

Full overview

The above image shows all the different encryption levels and where in the application, database and system interactions they are situated.

Feedback? Comments? Don't hesitate to drop me an email, or join the discussion on Twitter.

October 17, 2021

Unbound

In previous blog posts, I described howto setup stubby as a DNS-over-TLS resolver. I used stubby on my laptop(s) and unbound on my internal network.

I migrated to unbound last year and created a docker container for it. Unbound is a popular DNS resolver, it’s less known that you can also use it as an authoritative DNS server.

This work was based on Debian Buster, I migrated the container to Debian Bullseye reorganize it a bit to make it easier to store the zones configuration outside the container like a configmap or persistent volume on Kubernetes.

Version 2.0.0 is available at https://github.com/stafwag/docker-stafwag-unbound.

Version 2.0.0:

Changelog:

  • Updated the base image to debian:bullseye.
  • Updated create_zone_config.sh to be able to run outside the container.
  • Removed the zones.conf generation from the entrypoint
  • Start the container as the unbound user
  • Updated to logging.conf
  • Set the pidfile /tmp/unbound.pid
  • Added remote-control.conf
  • Updated the documentation

docker-stafwag-unbound

Dockerfile to run unbound inside a docker container. The unbound daemon will run as the unbound user. The uid/gid is mapped to 5000153.

Installation

clone the git repo

$ git clone https://github.com/stafwag/docker-stafwag-unbound.git
$ cd docker-stafwag-unbound

Configuration

Port

The default DNS port is set to 5353 this port is mapped with the docker command to the default port 53 (see below). If you want to use another port, you can edit etc/unbound/unbound.conf.d/interface.conf.

scripts/create_zone_config.sh helper script

The create_zone_config.sh helper script, can we help you to the zones.conf configuration file. It’s executed during the container build and creates the zones.conf from the datafiles in etc/unbound/zones.

If you want to use a docker volume or configmaps/persistent volumes on Kubernetes. You can use this script to generate the zones.conf a zones data directory.

create_zone_config.sh has following arguments:

  • -f Default: /etc/unbound/unbound.conf.d/zones.conf The zones.conf file to create
  • -d Default: /etc/unbound/zones/ The zones data source files
  • -p Default: the realpath of zone files
  • -s Skip chown/chmod

Use unbound as an authoritative DNS server

To use unbound as an authoritative authoritive DNS server - a DNS server that hosts DNS zones - add your zones file etc/unbound/zones/.

During the creation of the image scripts/create_zone_config.sh is executed to create the zones configuration file.

Alternatively, you can also use a docker volume to mount /etc/unbound/zones/ to your zone files. And a volume mount for the zones.conf configuration file.

You can use subdirectories. The zone file needs to have $ORIGIN set to our zone origin.

Use DNS-over-TLS

The default configuration uses quad9 to forward the DNS queries over TLS. If you want to use another vendor or you want to use the root DNS servers director you can remove this file.

Build the image

$ docker build -t stafwag/unbound . 

To use a different BASE_IMAGE, you can use the –build-arg BASE_IMAGE=your_base_image.

$ docker build --build-arg BASE_IMAGE=stafwag/debian:bullseye -t stafwag/unbound .

Run

Recursive DNS server with DNS-over-TLS

Run

$ docker run -d --rm --name myunbound -p 127.0.0.1:53:5353 -p 127.0.0.1:53:5353/udp stafwag/unbound

Test

$ dig @127.0.0.1 www.wagemakers.be

Authoritative dns server.

If you want to use unbound as an authoritative dns server you can use the steps below.

Create a directory with your zone files:

[staf@vicky ~]$ mkdir -p ~/docker/volumes/unbound/zones/stafnet
[staf@vicky ~]$ 
[staf@vicky stafnet]$ cd ~/docker/volumes/unbound/zones/stafnet
[staf@vicky ~]$ 

Create the zone files

Zone files

stafnet.zone:

$TTL  86400 ; 24 hours
$ORIGIN stafnet.local.
@  1D  IN  SOA @  root (
            20200322001 ; serial
            3H ; refresh
            15 ; retry
            1w ; expire
            3h ; minimum
           )
@  1D  IN  NS @ 

stafmail IN A 10.10.10.10

stafnet-rev.zone:

$TTL    86400 ;
$ORIGIN 10.10.10.IN-ADDR.ARPA.
@       IN      SOA     stafnet.local. root.localhost.  (
                        20200322001; Serial
                        3h      ; Refresh
                        15      ; Retry
                        1w      ; Expire
                        3h )    ; Minimum
        IN      NS      localhost.
10      IN      PTR     stafmail.

Make sure that the volume directoy and zone files have the correct permissions.

$ sudo chmod 750 ~/docker/volumes/unbound/zones/stafnet/
$ sudo chmod 640 ~/docker/volumes/unbound/zones/stafnet/*
$ sudo chown -R root:5000153 ~/docker/volumes/unbound/

Create the zones.conf configuration file.

[staf@vicky stafnet]$ cd ~/github/stafwag/docker-stafwag-unbound/
[staf@vicky docker-stafwag-unbound]$ 

The script will execute a chown and chmod on the generated zones.conf file and is excute with sudo for this reason.

[staf@vicky docker-stafwag-unbound]$ sudo scripts/create_zone_config.sh -f ~/docker/volumes/unbound/zones.conf -d ~/docker/volumes/unbound/zones/stafnet -p /etc/unbound/zones
Processing: /home/staf/docker/volumes/unbound/zones/stafnet/stafnet.zone
origin=stafnet.local
Processing: /home/staf/docker/volumes/unbound/zones/stafnet/stafnet-rev.zone
origin=1.168.192.IN-ADDR.ARPA
[staf@vicky docker-stafwag-unbound]$ 

Verify the generated zones.conf

[staf@vicky docker-stafwag-unbound]$ sudo cat ~/docker/volumes/unbound/zones.conf
auth-zone:
  name: stafnet.local
  zonefile: /etc/unbound/zones/stafnet.zone

auth-zone:
  name: 1.168.192.IN-ADDR.ARPA
  zonefile: /etc/unbound/zones/stafnet-rev.zone

[staf@vicky docker-stafwag-unbound]$ 

run the container

$ docker run --rm --name myunbound -v ~/docker/volumes/unbound/zones/stafnet:/etc//unbound/zones/ -v ~/docker/volumes/unbound/zones.conf:/etc/unbound/unbound.conf.d/zones.conf -p 127.0.0.1:53:5353 -p 127.0.0.1:53:5353/udp stafwag/unbound

test

[staf@vicky ~]$ dig @127.0.0.1 soa stafnet.local

; <<>> DiG 9.16.1 <<>> @127.0.0.1 soa stafnet.local
; (1 server found)
;; global options: +cmd
;; Got answer:
;; WARNING: .local is reserved for Multicast DNS
;; You are currently testing what happens when an mDNS query is leaked to DNS
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37184
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;stafnet.local.     IN  SOA

;; ANSWER SECTION:
stafnet.local.    86400 IN  SOA stafnet.local. root.stafnet.local. 3020452817 10800 15 604800 10800

;; Query time: 0 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sun Mar 22 19:41:09 CET 2020
;; MSG SIZE  rcvd: 83

[staf@vicky ~]$ 

Links

October 13, 2021

Update: Found it, fixed it. We can computer, after all :) We seem to be losing some email sent to our mailing lists. If you send anything important, please check the list archive to make certain it has arrived.

Last week, Drupalists around the world gathered virtually for DrupalCon Europe 2021.

In good tradition, I delivered my State of Drupal keynote. You can watch the video of my keynote, download my slides (156 MB), or read the brief summary below.

I talked about end-of-life schedules for various Drupal versions, delivered some exciting updates on Drupal 10 progress, and covered the health of the Drupal community in terms of contributor dynamics. Last but not least, I talked about how we are attracting new users and contributors by making it much easier to contribute to Drupal.

Drupal 7 and Drupal 8 end-of-life

If you are using Drupal 7 or Drupal 8, time is of the essence to upgrade to Drupal 9. Drupal 7 end-of-life is scheduled for November 2022.

Drupal 8's end-of-life is more pressing, as it is scheduled for November 2nd, 2021 (i.e. in less than a month). If you are wondering why Drupal 8 is end-of-life before Drupal 7, that is because we changed how we develop Drupal in 2016. These changes have been really great for Drupal. They've made it much easier to upgrade to the latest version without friction.

As a community, we've spent thousands of hours building tools and automations to make migrating to Drupal 9 as simple as possible.

Drupal 10 timeline

Next, I gave an update on Drupal 10 timelines. Timing-wise, our preferred option would be to ship Drupal 10 in June 2022. That date hinges on how much work we can get done in the next few months.

Drupal and timelines

Drupal core strategic initiatives

After these timelines, I walked through the six strategic initiatives for Drupal core. We've made really great progress on almost all of them. To see our progress in action, I invited key contributors to present video updates.

A slide with progress bars for each of the 6 initiatives; 3 of them are over 80% complete.

Project Browser

You may recall that I introduced the Project Browser initiative in my April 2021 State of Drupal presentation. The idea is to make it easy for site builders to find and install modules right from their Drupal site, much like an app store on a smartphone. The goal of this initiative is to help more evaluators and site builders fall in love with Drupal.

Today, just six months later, we have a working prototype! Take a look at the demo video:

Decoupled Menus

Drupal is an excellent headless CMS with support for REST, JSON:API and GraphQL.

As a next step in our evolution, we want to expand the number of web service endpoints Drupal offers, and build a large repository of web components and JavaScript framework integrations.

With that big goal in mind, we launched the Decoupled Menus initiative about one year ago. The goal was to create a small web component that could ship quickly and solve a common use case. We focused on one component so we could take all the learnings from that one component to improve our development infrastructure and policies to help us create many more web service end points and JavaScript components.

I talked about the various improvements we made to Drupal.org to support the development and management of more JavaScript components. I also showed that we've now shipped Drupal menu components for React, Svelte and more. Take a look at the video below to see where we're at today:

Our focus on inviting more JavaScript developers to the Drupal community is a transformative step. Why? Headless momentum is growing fast, largely driven by the growth of JavaScript frameworks. Growing right along with it is the trend of composability, or the use of independent, API-first micro-services. Building more web service endpoints and JavaScript components extends Drupal's leadership in both headless development and composability. This will continue to make Drupal one of the most powerful and flexible tools for developers.

Easy Out of the Box

The goal of this initiative is to have Layout Builder, Media, and Claro added to the Standard Profile. That means these features would be enabled by default for any new Drupal user.

Unfortunately, we have not made a lot of progress on this initiative. In my presentation, I talked about how I'd like to find a way for us to get it done by Drupal 10. My recommendation is that we reduce the scope of work that is required to get them into Standard Profile.

Automatic Updates

The Automatic Updates initiative's goal is to make it easier to update Drupal sites. Vulnerabilities in software, if left unchecked, can lead to security problems. Automatic updates are an important step toward helping Drupal users keep their sites secure.

The initiative made excellent progress. For the very first time, I was able to show a working development version:

Drupal 10 Readiness

The Drupal 10 Readiness initiative is focused on upgrading the third-party components that Drupal depends on. This initiative has been a lot of work, but we are largely on track.

A slide from the DriesNote saying that the Drupal 10 upgrade work is 300% more automated than Drupal 9.

The most exciting part? The upgrade to Drupal 10 will be easy thanks to careful management of deprecated code and continued investment in Rector. As it stands, upgrading modules from Drupal 9 to Drupal 10 can almost be entirely automated, which is a big 300% improvement compared to the Drupal 8 to Drupal 9 upgrade.

New front end theme

We are nearly at the finish line for our new front end theme, Olivero. In the past few months, a lot of effort has gone into ensuring that Olivero is fully accessible, consistent with our commitment to accessibility.

Olivero already received a glowing review from the National Federation of the Blind (USA):

Olivero is very well done and low-vision accessible. We are not finding any issues with contrast, focus, or scaling, the forms are very well done, and the content is easy to find and navigate.

Something to be really proud of!

The health of Drupal's contribution dynamics

Next, I took a look at Drupal's contribution data. These metrics show that contributions are down. At first I panicked when I saw this data, but then I realized that there are some good explanations for this trend. I also believe this trend could be temporary.

Contribution metrics

To learn more about why this was happening, I looked at the attrition rate of Drupal's contributors — the percentage of individuals and organizations who stopped contributing within the last year. I compared this data to industry averages for software and services companies.

Slide with data that shows Drupal's top contributors are very loyal
While typical attrition for software and services companies is considered "good" at 15%, Drupal's attrition rate for its Top 1,000 contributors is only 7.7%. The attrition rate for Drupal agencies in the Top 250 organizations is only 1.2%.

I was very encouraged by this data. It shows that we have a very strong, loyal and resilient community of contributors. While many of our top contributors are contributing less (see the full recording for more data), almost none of them are leaving Drupal.

There are a number of reasons for the slowdown in contribution:

  • The COVID-19 pandemic has made contribution more difficult and/or less desirable.
  • We are in the slow period of the "Drupal Super Cycle" — after every major release, work shifts from active development to maintenance.
  • Anecdotally, many Drupal agencies have told me they have less time to contribute because they are growing so fast (see quotes in image below). That is great news for Drupal adoption.
  • Drupal is a stable and mature software project. Drupal has nearly all the features organizations need to deliver state-of-the-art digital experiences. Because of Drupal's maturity, there are simply fewer bug fixes and feature improvements to contribute.
  • Rector-automations have led to less contribution. It's good to work smarter, not harder.

I'll expand on this more in my upcoming Who sponsors Drupal development post.

Slide with quotes from Drupal agencies CEOs stating that they are growing fast

The magic of contribution

I wrapped up my presentation by talking about some of the things that we are doing to make it easier to adopt Drupal. I highlighted DrupalPod and Simplytest as two examples of amazing community-driven innovations.

A slide promoting DrupalPod and Simplytest

After people adopt Drupal, we need to make it easier for them to become contributors. To make contribution easier, Drupal has started adopting GitLab in favor of our home-grown development tools. Many developers outside the Drupal ecosystem are accustomed to using tools like GitLab. Allowing them to use tools with which they are already familiar is an important step to attracting new contributors. Check out this video to get the latest update on our GitLab effort:

Thank you

To wrap up I'd like to thank all of the people and organizations who have contributed to Drupal since the last DriesNote. It's pretty amazing to see the momentum on our core initiatives! As always, your contributions are inspiring to me!

Thank you for the many contribution

October 11, 2021

FOSDEM 2022 will take place on Saturday 5 and Sunday 6 February 2022. The exact format is yet to be decided. As every year, we have started planning for real in August. As evident from our lack of updates since then, it's a bit harder this year. There are a lot of strong opinions about what the best, or least bad, FOSDEM 2022 could look like. Finding consensus is harder than we would like it to be. Going forward, we will do a better job of keeping you informed going forward. Apologies; we're also burned out and just want舰

October 08, 2021

Chers parents,

Nous sommes le 8 octobre 2051. J’ai aujourd’hui 30 ans et un peu de recul sur mon enfance, mon éducation et le monde dans lequel j’ai grandi.

Ma génération est confrontée à une problématique sans précédent dans l’histoire de l’humanité : devoir gérer les déchets de la génération précédente.

Jusqu’aux années 1970, la planète se régénérait naturellement. Les déchets humains étaient absorbés et recyclés spontanément. À partir de votre génération, ce ne fut plus le cas. Vous fûtes la toute première génération de l’histoire à produire et consommer plus que ce que la terre ne le permettait.

Vous nous laissez sur les bras l’excédent de déchets.

Le pire, c’est que vous le saviez.

Quand je me réfère aux archives et à mes souvenirs de prime jeunesse, votre époque n’était guère accueillante. Vous aviez des voitures consommant de l’énergie fossile et des fumeurs au cœur des villes ! Aujourd’hui, la voiture électrique ne sert que pour se déplacer entre les centres citadins. Elles sont strictement interdites dans les zones urbaines où tout se fait à pied, à vélo, à trottinette ou en taxi-tram autonome. Malgré tout, notre air est moins respirable que le vôtre !

Merci d’avoir œuvré à cette transformation. Peut-être était-ce le minimum à faire pour que nous survivions. Car si vous avez agi, souvent avec beaucoup de bonne volonté, c’était rarement dans le bon sens.

Comme cette manie que vous aviez de vouloir économiser l’électricité. J’ai du mal à croire que, même à votre époque, l’électricité n’était pas abondante et peu polluante pour les individus. Si j’en crois les archives, les années 2020 voyaient de réguliers pics de surproduction d’électricité dus aux panneaux solaires et vous démanteliez des centrales nucléaires parfaitement fonctionnelles. Vous perdiez vraiment votre temps à vous convaincre de mettre des ampoules économiques si polluantes à produire ? Un peu comme le coup de faire pipi dans la douche ou de ne pas imprimer les emails. Vous pensiez sérieusement que nous allions vous remercier pour cela ?

Vous semblez avoir dépensé tellement d’énergie et de temps pour tenter, parfois vainement, d’économiser 10% de votre consommation privée de ce qui n’était de toute façon qu’une goutte d’eau face à l’industrie. Vous culpabilisiez les individus alors que votre consommation personnelle représentait le quart de l’électricité consommée globalement (dont le tiers uniquement pour le chauffage). Même si vous aviez arrêté de consommer complètement de l’électricité à titre individuel, cela n’aurait eu qu’un impact imperceptible pour nous.

Par contre, vous nous laissez sur le dos des gigatonnes de déchets de ces appareils dont plus personne ne voulait, car ils consommaient un peu trop. Chaque année, culpabilisés par le marketing, vous vous équipiez d’une nouvelle génération d’appareils qui consommaient « moins », de vêtements « fair trade », de gourdes prétendument recyclables et de vaisselle en bambou. Le tout ayant fait le tour du monde pour rester brièvement dans vos armoires avant de combler les décharges sur lesquelles nous vivons désormais.

Vous semblez vous être évertué à acheter le plus de gadgets inutiles possibles, mais en vous rassurant, car, cette année, la fabrication du gadget en question avait émis 10% de CO2 en moins que celui de l’année précédente et que l’emballage était « presque entièrement recyclable  ». Ses composants avaient fait trois fois le tour du globe, mais, rassurez-vous, deux arbres avaient été plantés. Aujourd’hui encore, nous avons du mal à comprendre comment vous aviez matériellement le temps de faire autant d’achats. Il semblerait que vous deviez passer plus de temps à faire « du shopping » et à remplir vos armoires qu’à réellement utiliser vos achats. Armoires pleines à craquer que nous devons vider les jours qui suivent votre décès, moitié pleurant votre perte, moitié râlant sur votre propension à tout garder.

Consommer des gadgets était peut-être la seule façon que vous pouviez imaginer pour poursuivre la lubie de votre génération : créer des emplois. Toujours plus d’emplois. Une partie de ces emplois consistaient d’ailleurs explicitement à vous convaincre d’acheter plus. Comment avez-vous moralement pu accomplir ces tâches explicitement morbides ? Parce que c’était votre travail, certainement. L’histoire démontre que les pires exactions furent commises par des gens dont « c’était le travail ». Pousser les autres à consommer fait désormais partie de ces crimes historiques contre l’humanité. Utiliser le prétexte écologique pour consommer encore plus ne fait qu’aggraver la culpabilité de ceux qui furent impliqués.

Pendant 40 ans, vous avez eu comme politique de créer autant d’emplois que possible, emplois dont le rôle premier était de transformer les ressources en déchets. Pendant 40 ans, vous vous êtes démenés pour remplir le plus vite possible votre poubelle planétaire : nous, l’an 2050.

Nous, vos enfants, sommes votre poubelle. Ce pays lointain qui vous semblait abstrait, nous vivons dedans.

Il a fallu attendre notre génération pour décider que tout vendeur d’un bien ou d’un emballage ni immédiatement consommable ni naturellement dégradable était tenu de racheter ses produits à la moitié du prix, quel que soit l’état. De faire ainsi remonter la chaîne à chaque pièce, chaque composant. Au final, le producteur est en charge de l’évacuation et forcé de gérer son impact.

Bien sûr, il y’eut une énorme perturbation dans les services logistiques qui ont, soudainement, dû fonctionner dans les deux sens. Les industries se sont adaptées en tentant de développer des produits qui dureraient le plus longtemps possible et en favorisant la réparabilité ou la démontrabilité. Soudainement, c’était un argument de vente. Le marketing n’a pas mis longtemps à retourner sa veste et à tenter de vous convaincre que la location, même à très longs termes, était une liberté par rapport à la possession. La réparation a créé une activité économique que vous assimileriez peut-être à des emplois. Paradoxalement, une activité économique naturelle s’est développée le jour où nous avons arrêté de tenter de la créer artificiellement. Où nous avons considéré qu’il devait être possible de vivre sans travail. Nous espérons, de cette manière, redevenir une génération qui ne produit pas plus de déchet que ce que la planète peut absorber. Que ce soit en CO2, en microparticules, en métaux lourds.

Le réchauffement climatique et les feux de forêt ne nous aident pas, mais nous avons bon espoir d’y arriver.

Il n’empêche que, même si on y arrive, on doit toujours se coltiner vos 50 ans de déchets. Ils ne sont pas prêts de disparaitre vos jouets en plastique bon marché pas cher achetés pour calmer le petit dernier dans le magasin ou le téléphone super révolutionnaire devenu un presse-papier has-been 2 ans plus tard. Sans compter que le prix de leur fabrication et de leur transport nous accompagne à chacune de nos inspirations dans l’air chargé de CO2.

Chacune de nos respirations nous rappelle votre existence. Nous fait nous demander pourquoi vous n’avez pas agi ? Pourquoi avons-nous dû attendre de vous enterrer ou vous mettre à la retraite pour pouvoir faire quelque chose ?

Et puis certains d’entre nous me racontent qu’ils ont eu des parents qui fumaient. Qu’il était normal de fumer dans les rues à proximité des enfants voir dans les maisons ou les voitures.

Votre génération dépensait donc de l’argent dans le seul et unique but de se détruire la santé, de détruire la santé de ses propres enfants tout en polluant l’atmosphère, tout en polluant l’eau ? Vous financiez une florissante industrie dont le seul et unique objectif était la destruction de la santé de ses clients, des enfants de ses clients, de l’entourage de ses clients et de la nature ? On estime aujourd’hui que près de 1% du CO2 excédentaire dans l’atmosphère est dû à l’industrie du tabac. On s’en serait bien passé.

Par contre, il faut le reconnaitre, nous avons plein de photos et de documents historiques qui prouvent que vous étiez militants, que vous signiez des pétitions et que vous « marchiez pour le climat ». En fumant des clopes.

C’est devenu une moquerie récurrente quand on parle de vous. La génération des écolos-fumeurs. L’image est devenue célèbre pour illustrer ce mélange de bonne volonté collective inutile et paresseuse, cette propension à culpabiliser les individus pour des broutilles, à accomplir des actions collectives symboliques sans enjeu et à se voiler la face devant les comportements réellement morbides.

Vous hurliez « Priorité à la sauvegarde de la planète ! ». Ce à quoi les politiciens répondaient « Tout à fait ! Priorité à l’économie et la sauvegarde de la planète ! ». Puis, la gorge un peu enrouée, chacun rentrait chez soi, satisfait. Avant d’organiser un grand atelier participatif « Méditation transcendentale et toilettes sèches » où vous vous faisiez passer un joint de tabac industriel mélangé d’herbe bio issue du potager partagé.

Notre génération est permissive. Dans beaucoup de parties du monde, l’usage de drogue récréative est autorisé ou toléré. Par contre, toute émission de particules toxiques est strictement interdite dans les lieux publics. Ce n’était vraiment pas difficile à mettre en place et la seule raison que nous voyons pour laquelle vous ne l’avez pas fait c’est que vous ne le vouliez pas.

Malgré vos discours, vous ne vouliez absolument pas construire un monde meilleur pour nous. Il suffisait de vous poser la question : « est-ce que j’ai envie que mes enfants fument ? » Même parmi les fumeurs invétérés, je pense que très peu auraient répondu par l’affirmative. « Ai-je envie que mes enfants subissent le poids écologique de vingt téléphones portables pour lesquels j’ai, au total, dépensé un an de salaire ? De milliers de kilomètres de diesel et de cinq voitures de société ? ». Il aurait suffi de vous poser la question. Interdire la cigarette dans l’espace public aurait été une manière toute simple d’affirmer que vous pensiez un peu à nous.

Mais vous ne pensiez pas à nous. Vous n’avez jamais pensé à nous. Vous avez juste voulu vous donner bonne conscience en ne changeant strictement rien à vos habitudes, même les plus stupides. Pour votre décharge, vous n’avez pas hérité non plus d’une situation facile de vos propres parents, cette génération qui après une gueule de bois post-mai 68, s’est accaparé toutes les richesses et les a gardées en votant Reagan/Tatcher et allongeant son espérance de vie. Sans jamais vous laisser votre place.

Quand nous en discutons entre nous, nous pensons que, finalement, nous avons de la chance d’être là. On doit gérer vos poubelles, mais vous auriez pu, pour le même prix, nous annihiler. Vous nous avez traités comme un pays vierge, un pays lointain à conquérir pour en exploiter les ressources à n’importe quel prix. Un pays qui vous appartenait de droit, car les autochtones n’offraient aucune résistance active.

Ce qui est fait est fait. Il nous reste la tâche ardue de ne pas faire pareil et tenter d’offrir un monde meilleur à nos enfants. Non pas en prétendant penser à eux pour nous donner bonne conscience, mais en tentant de penser comme ils le feront. En les traitant comme un pays ami à respecter, un partenaire. Non plus comme une poubelle sans fond.

Signé : votre futur

Note de l’auteur : L’idée de considérer le futur comme un pays avec qui entretenir des relations internationales m’a été inspirée par Vinay Gupta lors d’une rencontre au parlement européen en 2017. Vinay a ensuite publié une analyse très intéressante où il suggère de voir toutes nos actions à travers le filtre du futur que nous réservons aux enfants de cette planète.

https://medium.com/@vinay_12336/a-simple-plan-for-repairing-our-society-we-need-new-human-rights-and-this-is-how-we-get-them-cee5d6ededa9

Bien que ces deux inspirations n’aient pas été conscientes au moment de la rédaction de ce texte, elles m’apparaissent comme indubitables à la relecture.

Photo by Simon Hurry on Unsplash

Oubliez un instant les réseaux sociaux et abonnez-vous par mail ou par RSS (max 2 billets par semaine et rien d’autre). Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, offrez et partagez ses livres.

Ce texte est publié sous la licence CC-By BE.

October 04, 2021

Security vendors are touting the benefits of "zero trust" as the new way to approach security and security-conscious architecturing. But while there are principles within the zero trust mindset that came up in the last dozen years, most of the content in zero trust discussions is tied to age-old security propositions.

October 01, 2021

Cover Image - Tackling
"The problem isn't that Johnny can't read. The problem isn't even that Johnny can't think. The problem is that Johnny doesn't know what thinking is; he confuses it with feeling."
– Thomas Sowell

I'm not one to miss an important milestone, so let me draw your attention to a shift in norms that's taking place in the Ruby open source community: it's now no longer expected to be tolerant of views that differ.

This ought to be a remarkable change: previously, a common refrain was that "in order to be tolerant, we cannot tolerate intolerance." This was the rationale for excluding certain people, under the guise of inclusivity. Well, that line of reasoning is now on its way out, and intolerance is now openly advocated for, with lots of heart emoji to boot.

heart

The Anatomy of Man - Da Vinci (1513)

Code of Misconduct

Source for this is a series of changes to the Ruby Code of Conduct, which subtly tweak the language. The stated rationale is to "remove abuse enabling language."

There are a few specific shifts to notice here:

  • Objections no longer have to be based on reasonable concerns.
  • All that matters is that someone could consider something to be harassing behavior.
  • Behavior is now mainly unacceptable if it targets protected classes.
  • Tolerance of opposing views is removed entirely as expected conduct.

Also noticeable is that this is done through multiple small changes, each stacking on top of the next over a few days, as a perfect illustration of "boiling the frog."

This ought to set off alarm bells. If concerns no longer have to be reasonable, then completely unreasonable complaints will have to be taken seriously. If opposing views are no longer welcome, then casting doubt on accusations of abuse is also misconduct. If only protected classes are singled out as worthy of protection, then it creates a grey area of traits which are acceptable to use as weapons to bully people.

It shouldn't take much imagination to see how these changes can actually enable abuse, if you know how emotional blackmail works: it's when an abuser makes other people responsible for managing the abuser's feelings, which are unstable and not grounded in mutual respect and obligation. If Alice's behavior causes Bob to be upset, Bob castigates Alice as an offender. If Bob's behavior causes Alice to be upset, then Alice is making Bob feel unsafe, and it's still Alice's fault, who needs to make amends.

A good example is how the social interaction style of people with autism can be trivially recast as deliberate insensitivity. Cancelled Googler James Damore made exactly this point in The Neurodiversity Case for Free Speech. This is also excellently illustrated in Splain it to Me which highlights how one person's gift of information can almost always be recast as an attempt to embarrass another as ignorant.

For all this to seem sensible, the people involved have to have enormous blinders on, suffering from the phenomenon that Sowell so aptly described: the focus isn't on thinking out a set of effective and consistent rules, but rather on letting the feelings do the driving, letting the most volatile members dominate over everyone else. Quite possibly they themselves have one or more emotional abusers in their lives, who have trained them to see such asymmetry as normal. "Heads I win, tails you lose" is a recipe for gaslighting, after all.

The Ruby community is of course free to decide what constitutes acceptable behavior. But there is little evidence there is widespread support for such a change. On HackerNews, the change in policy was widely criticized. Discussion on the proposals themselves was locked within a day, for being "too heated," despite involving only a handful of people. This moderator action seems itself an example of the new policy, letting feelings dominate over reality: after proposing a controversial change, maintainers plug their ears because they do not wish to hear opposing views, even before they are actually uttered in full.

A man kneeling and placing a laurel branch upon a pile of burning books

Marco Dente (ca. 1515-1527)

Harassment Policy

Way back in 2013, something similar happened at the PyCon conference in the notorious DongleGate incident. After overhearing a joke between two men seated in the audience, activist Adria Richards decided to take the offenders' picture and post it on Twitter. She was widely praised in media for doing so, and it resulted in the loss of the jokester's job.

What was crucial to notice, and which many people didn't, was that "harassing photography" was explicitly against the conference's anti-harassment policy. By any reasonable interpretation of the rules, Richards was the harasser, who wielded social media as a weapon for intimidation. She should've been sanctioned and told in no uncertain terms that such behavior was not welcome.

Of course, that did not happen. Citing concerns about women in tech, she appealed exactly to those "protected classes" to justify her behavior. She cast herself in the role of defender of women, while engaging in an unquestionable attack.

It's easy to show that this was not motivated by fairness or equality: had the joke been made by a woman instead, Richards wouldn't have been able to make the same argument. The accusation of sexism seemed to derive from the sexual innuendo in the joke, an assumed male-only trait. Indeed, the only reason it worked was because of her own sexism: she assumed that when one man makes a joke, he is an avatar of oppression by men in the entire industry. She treated him differently because of his sex, so her accusation of sexism was a cover for her own.

Even more ridiculous was that her actual job was "Developer Relations." She was supposedly tasked with improving relations with and between developers, but did the exact opposite, creating a scandal that would resonate for years. What it really showed was that she was volatile and a liability for any company that would hire her in this role.

Somehow, this all went unnoticed. Nobody involved seemed to actually think it through. The entire story ran purely on hurt feelings, narrating the entire experience from one person's subjective point of view. This is now a common thread in many environments that are supposed to be professional: the people in charge have no idea how to keep their own members in check, and allow them to hijack everyone's resources and time for grievances and external drama.

As a rare counter-example, consider crypto-exchange CoinBase. They explicitly went against the grain a year ago, by announcing they were a mission-focused company, who would concentrate their efforts on their actual core competence. Today, things are looking much brighter for them, as the negative response and doom-saying in media turned out to be entirely irrelevant. On the inside, the reaction was mostly positive. The employees that left in anger were eventually replaced, with a group of equally diverse people.

The School of Athens

The School of Athens - Raphael (1508)

Professing

Professionalism seems to be a concept that is very poorly understood. In the direct sense, it's a set of policies and strategies that allow people with wildly different interests to come together and get productive work done regardless.

In a world where many people wish to bring "their entire selves to work," this can't happen. If it's more important to keep everyone's feelings in check, and less important to actually deliver results, then there's no room for fixing mistakes. It creates an environment where pointing out problems is considered an unwelcome insensitivity, to which the response is to gang up on the messenger and shoot them for being abusive.

The most common strategy is simply to shame people into silence. If that doesn't work, their objections are censored out of sight, and then reframed as bigotry if anyone asks. The narrative machine will spin up again, using emotionally charged terms such as "harassment" and "sexism."

The idea of "victim blaming" is particularly pernicious here: any time someone invokes it, without knowing all the details, they must have pre-assumed they know who is the victim and who is the offender. This is where the concept of "protected classes" comes into play again.

While it's supposed to mean that we cannot discriminate e.g. on the basis of sex, what it means in practice is that one assumes automatically that men are the offenders and that women are being victimized. Even if it's the other way around. Indeed, such a model is the cornerstone of intersectionality, a social theory which teaches that on every demographic axis, one can identify exclusive categories of oppressors and the oppressed. White oppresses black, straight oppresses gay, cis oppresses trans, and so on.

If you engage such bigoteers in debate, the experience is pretty much like talking to a brick wall. You are not speaking to someone who is interested in being correct, merely in remaining on the right side. This seems to be the axiom from which they start, and a core part of their self-image. If you insist on peeling off the fallacies and mistakes in reasoning, you only invoke more ire. Your line of reasoning is upsetting to them, and therefor, you are a bigot who needs to leave, or be forcefully expelled. In the name of tolerance, for the sake of diversity and inclusion, they flatten the actual complexities of life and become utterly intolerant and exclusionary.

It's no coincidence that these cultural flare ups first came to a head in environments like open source, where results speak the loudest. Or in STEM and video games, where merit reigns supreme. When faced with widespread competence, the incompetent resort to lesser weapons and begin to undermine social norms, to try and mend the gap between their self-image and what they are actually able to do.

* * *

Personally, I'm quite optimistic, because the game is now clearly visible. In their zeal for ideological purity, activists have blown straight past their own end zone. When they tell you they are no longer interested in tolerance, you should believe them. It represents a complete abandonment of the principles that allowed liberal society to grow and flourish.

That means tolerance now again belongs to the adults in the room, who are able to separate fact from fiction, and feelings from actual principled conviction. We can only hope these children finally learn.

Bij problemen met draadloze apparaten is het vaak moeilijk om precies de oorzaak aan te duiden. Wireshark is een populaire opensource-packetsniffer die zowel op Windows, Linux als macOS draait. Het is een standaard hulpmiddel geworden in de gereedschapskist van netwerkbeheerders. Je onderzoekt er zowel wifi- als ethernetverkeer mee, bijvoorbeeld om netwerkproblemen te analyseren.

Met een nRF52840 Dongle van Nordic Semiconductor en een plug-in voor Wireshark kun je er ook Bluetooth Low Energy-, Zigbee- en Thread-verkeer mee uit de lucht plukken. Voor PCM beschreef ik de hele procedure om BLE en Zigbee te sniffen.

Opmerking

De firmware is ook compatibel met de April USB Dongle 52840 van April Brother. De externe antenne maakt een groot verschil in bereik met de PCB-antenne van de nRF52840 Dongle.

Nordic Semiconductor biedt de firmware voor de nRF52840 Dongle en de bijbehorende plug-ins voor Wireshark hier aan:

Ik gebruik de nRF Sniffer for Bluetooth LE continu om BLE-apparaten te debuggen. Zo kun je eenvoudig met de displayfilters van Wireshark filteren op specifieke types BLE-pakketten. Zo filter je bijvoorbeeld op iBeacon-pakketten:

(btcommon.eir_ad.entry.company_id == 0x004c) && (btcommon.eir_ad.entry.data[:2] == 02:15)

Dat beperkt de getoonde pakketten tot degene met manufacturer-specific data van company ID 0x004c (van Apple) en met de eerste twee bytes gelijk aan 0x0215. 1

Maar hoe kom je aan die displayfilter? Als je op manufacturer-specific data van company ID 0x004c wilt filteren in Wireshark, klik je eenvoudigweg op het veld Company ID in het paneel met pakketdetails van een iBeacon-pakket, rechtskik je en kies je dan Apply as Filter en dan Selected. Dat voegt een displayfilter toe voor alle pakketten met de geselecteerde waarde voor het company ID.

De extra filter voor de eerste twee bytes is wat meer werk als je de syntax niet kent. Selecteer gewoon het volledige veld Data in het paneel met pakketdetails van een iBeacon-pakket, rechtsklik en kies dan Apply af Filter en dan ... and Selected. Dat voegt deze filter toe als extra vereiste aan de al gebruikte filter. Maar nu filter je alle pakketten met exact dezelfde data als dit geselecteerde pakket:

btcommon.eir_ad.entry.data == 02:15:18:ee:15:16:01:6b:4b:ec:ad:96:bc:b9:6d:16:6e:97:00:00:00:00:d8

Als je wilt filteren op de eerste twee bytes, voeg je [:2] aan het dataveld toe. Die vergelijk je dan met de bytes 02:15.

1

Waarom 0x0215? Dit vind je in de specificatie van iBeacon. Apple gebruikt in zijn manufacturer-specific data een TLV-formaat (type-length-value). De 0x02 staat voor het type iBeacon en de 0x15 stelt de lengte van de data erna voor (21 decimaal: 16 bytes voor de UUID, 2 voor de major, 2 voor de minor en 1 voor de measured power).

September 30, 2021

For the past two years I’ve been working on something less visible but no less important.

Since DrupalCon Amsterdam 2019 (an actual in-person conference — sounds surreal in 2021, doesn’t it?!) I’ve been working on Acquia Migrate Accelerate, or “AMA” for short. In a few days, another DrupalCon Europe is starting … so perfect timing for a recap! :D

Why?

Drupal 8 comes with an awesome migration system built in, originating in the Migrate Drupal 7 module. It standardized many migration best practices. But it still required a huge time investment to learn it.

Of course, there’s the “Migrate Drupal UI” (migrate_drupal_ui) module in Drupal core. But that does not allow for granular migrations. It allows for a one-shot migration: you see which things will be migrated and which won’t. You can click a button and hope for the best. It only works for the very simplest of sites. It is impressively minimal in terms of the code it needs, but unfortunately it also means one pretty much needs to be a expert in migrations to use it successfully.

It will be of little help as soon as you run into important data you want to migrate for which no migration path exists.

See Mauricio Dinarte’s excellent “31 days of Drupal migrations”. In those 31 blog posts, you’ll learn to know and appreciate the migration system (I sure did!). Unfortunately, that still won’t fully prepare you: you’ll need to decipher/reverse engineer the intricacies of how the data gets stored in Drupal 7 with its entities, revisions and fields — and with each field type having its own intricacies — and map that to Drupal 9 equivalents.

And how does one migrate straight from Drupal 7 with its more fragmented ecosystem? 1

For example: media handling. There are easily a dozen approaches possible in Drupal 7. Each in use on tens of thousands of sites. In Drupal 8 & 9, everything has standardized on Media and Media Library. But how do you get your Drupal 7 site’s content in there?

Another example: location data. location was very popular, but is now dead. geofield was equally popular but still alive. geolocation was less popular but now more. addressfield was popular, address is the successor. None of the Drupal 9 modules offer Location’s feature set. How do you migrate this data?

Goal

The goal for AMA (the vision of https://www.drupal.org/u/grasmash and especially https://www.drupal.org/u/webchick!) is to empower the non-technical user to be able to perform migrations. A UI that is aimed at the site builder POV: one should be able to select which content types (also vocabularies, menus, et cetera) make sense to migrate, and then not have to bother with technical details such as “migration plugins” or YAML files.

Acquia Migrate Accelerate:

For example, AMA shows just “Page” in the UI. Under the hood (and you can see this in the UI too, but it’s just not prominent), that corresponds to the following migration plugin definitions:

  • d7_node_type:page
  • d7_field_instance:node:page
  • d7_field_formatter_settings:node:page
  • d7_field_instance_widget_settings:node:page
  • d7_node_complete:page
  • d7_url_alias:node:page
  • node_translation_menu_links:node:page

In other words: the support configuration for nodes of the page bundle (the first 4), then all actual entity/field data (d7_node_complete:page), followed by URL aliases and menu links referencing pages.

However, to be able to do this, we need many more migrations in Drupal core to be derived: view modes, fields, formatters and widget should all have an entity type+bundle-specific derivative. That’d allow each bundle to be migrated individually. Which enables the site builder to check that their pages and everything related to it has been correctly migrated before moving on to the next data concept to migrate. So far we’ve not yet been able to convince the migration system maintainers of the value of this. 2

(AMA does many more things, but that’s not in scope of this blog post.)

Closed & Open

Acquia understandably wants its customers to be able to use AMA, and not its competitors’ users. Like all Drupal modules, the AMA module is GPL 2+ licensed. Only the React UI is closed source. The automated recommendations engine is closed source. Obviously the code to spin up AMA environments in Acquia Cloud is closed source 3.

But … all of the work that goes into making migrations reliable is open source. At the time of writing, we have ~100 unique patches that are being applied, 39 of which to Drupal core! While I was writing this, https://www.drupal.org/project/drupal/issues/3190818 got committed, plus a few were committed recently but did not yet ship in a 9.2.x point release, so soon that number will be lower :)

An overview

In the past 20 months we’ve hardened many migrations, and created new ones from scratch. Thousands of Drupal sites have already benefited — many more than there are Acquia customers.

The highlights:

Overall, 29 Drupal core patches 4 and 18 Drupal contrib patches have been committed! Plus another 36 core patches 5 and 32 34 contrib patches are successfully being used, and will hopefully land in the near future. (Not counting commits to the migration modules we now (co-)maintain.) Many dozens of migration paths from Drupal 7 have been stabilized, especially by https://www.drupal.org/project/media_migration.

A comprehensive overview (all patches are uncommitted unless stated otherwise):

D7D9 for all

We aim to continue to do to the work to get patches committed: address feedback, add test coverage, and so on. We want to help everyone migrate from Drupal 7 to 9!

Teamwork

These many hardened migrations are thanks to the titanic work of:

If you found this interesting, check out Gabe’s write-up of the application architecture that powers the awesome React-based UI that Peter built.


  1. Some would say richer↩︎

  2. It also implicitly reveals one of the most fundamental assumptions in the migration system: that only the final state after running all migrations matters. For developers who know both Drupal 7 and 9’s data models really well, this may be fine. But for a non-expert, it’d be simpler if they were able to migrate each the entities of each entity type+bundle and then inspect the results, not to mention that it’d take less time to get some confidence in the migration! For example, first the “tags” taxonomy terms, then the “image” media items, then the “blog post” content items. Verifying the correct validation of each of those clusters of data is simpler conceptually. Site builders, if you want this, please leave a comment in https://www.drupal.org/node/3097336↩︎

  3. Acquia Cloud handles the creation of an ephemeral Drupal 9 migration environment, with a Drupal 9 site automatically generated, with all equivalent D9 modules pre-composer required, and all modules with a vetted migration path pre-installed. For Acquia the value is obvious: its customers are more likely to succesfully mgirate to Drupal 9 sooner, the customer is more likely to stay a customer. We’ve currently got over 100 customers using AMA. ↩︎

  4. Committed core patches: #3096676, #2814953 (this one has the majority of the work done by the community!), #3122056, #3126063, #3126063, #2834958 (from those first 14), #3152789, #3151980, #3151993, #3153791, #2925899, #3165944, #3176394, #3178966, #3187320, #3187415, #3187418, #3187463, #3189463, #3187263 (from 2020), #3190815, #3190818, #3191490, #3097312, #3212539, #3213616, #3224620, #3227549, #3085192 (from 2021). ↩︎

  5. We started out with 14 core patches. Of those, #3115073, #3122649, #3096972, #3108302, #3097336, #3115938, #3123775 still remain. Other core patches we’ve contributed in 2020 that are not yet committed: #2845340, #3151979, #3051251, #3154156, #3156083, #3156730, #3156733, #3165813, #3166930, #3167267, #3186449, #3187334, #3187419, #3187474, #3187616. And those in 2021: #2859314, #3200949, #3204343, #3198732, #3204212, #3202462, #3118262, #3213636, #3218294, #3219078, #3219140, #3226744, #3227361, #3227660 ↩︎

  6. Added October 1, 2021. ↩︎ ↩︎

September 28, 2021

Not that long ago, a vulnerability was found in Microsoft Azure Cosmos DB, a NoSQL SaaS database within the Microsoft Azure cloud. The vulnerability, which is dubbed ChaosDB by the Wiz Research Team, uses a vulnerability or misconfiguration in the Jupyter Notebook feature within Cosmos DB. This vulnerability allowed an attacker to gain access to other's Cosmos DB credentials. Not long thereafter, a second vulnerability dubbed OMIGOD showed that cloud security is not as simple as some vendors like you to believe.

These vulnerabilities are a good example of how scale is a cloud threat. Companies that do not have enough experience with public cloud might not assume this in their threat models.

September 27, 2021

SReview, the video review and transcode tool that I originally wrote for FOSDEM 2017 but which has since been used for debconfs and minidebconfs as well, has long had a sizeable component for inspecting media files with ffprobe, and generating ffmpeg command lines to convert media files from one format to another.

This component, SReview::Video (plus a number of supporting modules), is really not tied very much to the SReview webinterface or the transcoding backend. That is, the webinterface and the transcoding backend obviously use the ffmpeg handling library, but they don't provide any services that SReview::Video could not live without. It did use the configuration API that I wrote for SReview, but disentangling that turned out to be very easy.

As I think SReview::Video is actually an easy to use, flexible API, I decided to refactor it into Media::Convert, and have just uploaded the latter to CPAN itself.

The intent is to refactor the SReview webinterface and transcoding backend so that they will also use Media::Convert instead of SReview::Video in the near future -- otherwise I would end up maintaining everything twice, and then what's the point. This hasn't happened yet, but it will soon (this shouldn't be too difficult after all).

Unfortunately Media::Convert doesn't currently install cleanly from CPAN, since I made it depend on Alien::ffmpeg which currently doesn't work (I'm in communication with the Alien::ffmpeg maintainer in order to get that resolved), so if you want to try it out you'll have to do a few steps manually.

I'll upload it to Debian soon, too.

September 24, 2021

I published the following diary on isc.sans.edu: “Keep an Eye on Your Users Mobile Devices (Simple Inventory)“:

Today, smartphones are everywhere and became our best friends for many tasks. Probably your users already access their corporate mailbox via a mobile device. If it’s not yet the case, you probably have many requests to implement this. They are two ways to achieve this: you provide corporate devices to all users. From a risk perspective, it’s the best solution: you select the models and control them. But it’s very expensive and people don’t like to carry two devices (a personal and a corporate one). Hopefully, if you use a Microsoft Exchange platform, there are ways to authorize personal devices to access corporate emails with a software component called ActiveSync. ActiveSync allows deploying basic security policies like forcing the device to be locked with a password, force a minimum password length, etc. However, it’s not a real MDM (“Mobile Device Management”)… [Read more]

The post [SANS ISC] Keep an Eye on Your Users Mobile Devices (Simple Inventory) appeared first on /dev/random.

September 23, 2021

I published the following diary on isc.sans.edu: “Excel Recipe: Some VBA Code with a Touch of Excel4 Macro“:

Microsoft Excel supports two types of macros. The legacy format is known as “Excel4 macro” and the new (but already used for a while) is based on VBA. We already cover both formats in many diaries. Yesterday, I spotted an interesting sample that implements… both!

The malicious file was delivered through a classic phishing email and is called “Document_195004540-Copy.xls” (SHA256:4f4e67dccb3dfc213fac91d34d53d83be9b9f97c0b75fbbce8a6d24f26549e14). The file is unknown on VT at this time. It looks like a classic trap… [Read more]

The post [SANS ISC] Excel Recipe: Some VBA Code with a Touch of Excel4 Macro appeared first on /dev/random.

September 20, 2021

Vous pouvez dès à présent précommander la version audiolivre de Printeurs et donner votre avis sur la voix à choisir. Ce qui me fait réfléchir à la voix, au bruit, au marketing, au crowdfunding et aux inondations…

Mon roman Printeurs va prendre de la voix et sera bientôt produit sous forme d’un audiolivre. Un format avec lequel je ne suis pas du tout familier (je suis un lecteur visuel), mais dont je me réjouis d’écouter le résultat. Je suis d’ailleurs curieux d’avoir les avis des gros consommateurs de livres audio sur ce qui fait un « bon » audiolivre. Qu’aimez-vous ? À quoi doit-on faire attention ? Et qu’est-ce qui vous fait arrêter votre écoute à tous les coups ?

Afin de financer cette entreprise, mon éditeur a mis en place une campagne de crowdfunding au cours de laquelle vous pouvez précommander la version audio de Printeurs. Vous aurez même la possibilité de donner votre avis sur des voix présélectionnées. Je suis vraiment curieux de lire l’avis des amateurs du genre.

Précommander la version audio de Printeurs :https://fr.ulule.com/ludomire/?reward=752654
Voter pour votre voix préférée (lien réservé aux souscripteurs) : https://fr.ulule.com/ludomire/news/decouvrez-le-casting-de-voix-pour-le-livre-audio-p-312900
Explications techniques sur l’adaptation audio : https://fr.ulule.com/ludomire/news/les-adaptations-en-livres-audio-312264/

La voix

La voix est un médium particulier. Lorsqu’on parle, le charisme et les intonations ont souvent plus d’importance que le contenu lui-même. Les incohérences sont gommées par le rythme. Un exemple parmi tant d’autres : j’ai été récemment interviewé par Valentin Demé pour le podcast Cryptoast afin de parler des monopoles et de la blockchain.

Pendant une heure, je parle en laissant mes idées vagabonder. Des idées bien moins formées que ce que j’écris d’habitude, des intuitions, des explorations. D’après les réactions, ce que je dis semble intéressant. Mais il faudrait garder à l’esprit que, à l’exception d’un discours entièrement préparé (un cours par exemple), les informations sont beaucoup plus aléatoires et toujours à prendre avec un grain de sel. Paradoxalement, la voix est plus convaincante alors qu’elle est moins rigoureuse. On apprend et réfléchit dans les livres, on se fait convaincre par les discours. La politique est une affaire de voix. La science est une affaire d’écrit.

Ploum sur Cryptoast : https://www.youtube.com/watch?v=vq6o_30LxJM

Le crowdfunding en question

Cette campagne de crowdfunding ne concerne pas que Printeurs. C’est avant tout une campagne englobant toutes les nouveautés de la collection SFFF Ludomire notamment la version papier en quatre volumes du One Minute de Thierry Crouzet. One Minute est un roman de SF se déroulant durant… une seule et unique minute, comme le dit le titre. Chacun des 365 chapitres dure… une minute. J’ai beaucoup apprécié la version Wattpad et je me réjouis de lire cette version entièrement retravaillée.

Encore de la pub pour une campagne de crowdfunding ? Autant je suis enthousiaste sur le contenu, autant je vous comprends.

La campagne crowdfunding de Printeurs m’a laissé un souvenir assez amer. Certes, elle a été un incroyable succès (grâce à vous qui me lisez) mais j’ai eu l’impression de spammer sans arrêt mes réseaux. De produire le bruit contre lequel je me bats tellement. J’en suis sorti lessivé et ceux qui me suivent également. Le problème, comme me l’a fait remarquer mon éditeur, c’est que le spam… ça fonctionne !

Ces campagnes sont désormais beaucoup plus nombreuses. Il faut se différencier, se professionnaliser. Bref, le marketing redevient essentiel alors que, dans mon esprit, l’un des buts initiaux du crowdfunding était de se passer de cette étape. Ironiquement, le marketing se concentre, non plus sur le produit lui-même, mais sur la promotion… de la campagne de financement ! Alors que cette méthode est censée rapprocher le créateur du consommateur, elle l’éloigne paradoxalement.

C’est un questionnement que se pose également Lionel, mon éditeur. Comment se faire connaitre et se financer sans pour autant tomber dans le spam ? Thierry lui-même m’a confié ne pas avoir la moindre envie de promouvoir la campagne liée à la parution de son roman.

La campagne Ludomire 2021 :https://fr.ulule.com/ludomire/
Crouzet raconte One Minute : https://tcrouzet.com/2021/09/14/de-lecriture-de-la-vie-du-roman/
Réflexions sur le crowdfunding : http://ludom.cc/index.php/2021/09/08/levolution-du-crowdfunding-selon-mon-experience/

Le prix libre ?

La problématique n’est pas uniquement limitée au crowdfunding. Le prix libre est également impacté. Il y a quelques années, je fus l’un des pionniers francophones du Prix Libre sur le Web à travers un billet au titre provocant : « Ce blog est payant ! ». Force est de constater que le concept s’est largement popularisé, au point d’avoir sa page Wikipédia.

Un peu trop popularisé peut-être. Désormais, le prix libre est partout et, comme par magie, se fédère sur quelques plateformes centralisées. Alias parle justement de son questionnement à propos de Tipeee, plateforme que j’ai également quittée.

Il y’a une fatigue indéniable du public : nous sommes sollicités tout le temps pour financer tous les projets imaginables, depuis les aiguilles à tricoter connectées révolutionnaires à l’installation de pots de fleurs sur la voirie de notre quartier. Outre les sous, il s’agit de jongler entre les différentes plateformes, les sommes, récurrentes ou non. J’ai également le sentiment que ce sont toujours les mêmes qui contribuent à tout, pas spécialement les plus aisés.

J’en ai déduit une sorte de loi générale sur Internet qui fait que toutes les bonnes idées sont soit pas assez populaires pour être largement utiles, soit tellement populaires que ça en fait des mauvaises idées. Les réseaux sociaux, la mobilité en sont les illustrations les plus marquantes. Le prix libre est-il en train de suivre cette voie ?

Les alternatives que nous construisons ne sont-elles séduisantes que parce qu’elles sont des alternatives justement ? Le succès n’entraîne-t-il pas obligatoirement un excès inexorable ? Je pense par exemple au réseau minimaliste Gemini dont je vous ai parlé.

Le prix libre sur Wikipedia :https://fr.wikipedia.org/wiki/Prix_libre
Alias quitte Tipeee : https://erdorin.org/il-est-temps-de-changer-de-tipeee/
Le drama tipeee (lien gemini) : gemini://lord.re/fast-posts/62-le-drama-tipee-2021/index.gmi

Le livre suspendu et les inondations

Face à ce constat, j’ai décidé de retirer tous les appels aux dons sur mon blog et encourager l’achat de livres. Je trouve que les livres sont parmi les objets les plus symboliques de l’humanité. Un livre n’est jamais inutile. Il peut dormir des années voire des siècles sur des étagères avant de renaître et d’illuminer une journée ou une vie. Le livre, y compris au format électronique, c’est le cadeau par excellence : un monde à découvrir, un objet à transmettre, des explorations intellectuelles à partager, dans le présent et le futur.

Acheter mes livres :  https://ploum.net/livres/

Le livre papier ne connait que deux dangers : le feu et l’eau. C’est malheureusement ce qui est arrivé cet été dans mon pays. Si je n’ai pas été personnellement touché, ce fut bien le cas de ma ville (Ottignies) et surtout de la région d’où sont originaires mon épouse, mes parents et mes ancêtres (vallée de la Vesdre).

Si vous avez perdu votre bibliothèque suite aux inondations ou si vous connaissez quelqu’un dans le cas, envoyez-moi un petit mot, je vous ferais parvenir un exemplaire de Printeurs. Je dispose également de plusieurs ouvrages de la collection Ludomire que j’enverrai volontiers aux bibliothèques qui cherchent à se reconstruire. N’hésitez pas à prendre contact et à faire l’intermédiaire pour des personnes à qui cela pourrait apporter un petit sourire. C’est toujours bon à prendre dans cette période difficile de reconstruction où la vie, comme la Vesdre, semble avoir repris son cours normal. Sauf pour ceux qui ont tout perdu, qui vivent dans l’humidité, qui sont nourris par la Croix-Rouge et dont le cœur s’étreint d’angoisse à chaque nouvelle pluie un peu drue.

Il me reste quelques exemplaires du livre « Les aventures d’Aristide, le lapin cosmonaute ». Ils sont normalement en vente, mais je les offre avec plaisir aux familles avec enfant (idéalement 5-9 ans) qui sont en manque de livre, que ce soit à cause des inondations ou pour des raisons qui ne me regardent pas.

Envoyez-moi un mail en précisant quel livre vous ferait plaisir (ou bien les deux) à l’adresse suspendu at ploum.net.

Bonne lecture et bonne écoute !

Oubliez un instant les réseaux sociaux et abonnez-vous par mail ou par RSS (max 2 billets par semaine et rien d’autre). Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, offrez et partagez ses livres.

Ce texte est publié sous la licence CC-By BE.

September 19, 2021

playbook

I wrote a few articles:

on my blog on how to use cloud images with cloud-init on a “non-cloud” environment.

I finally took the time to create an Ansible role for it. You’ll find the READE.md below.

Virt_install_vm 1.0.0 is available at: https://github.com/stafwag/ansible-role-virt_install_vm

Have fun!


Ansible Role: virt_install_vm

An Ansible role to install a libvirt virtual machine with virt-install and cloud-init. It is “designed” to be flexible.

An example template is provided to set up a Debian system.

Requirements

The role is wrapper around the following roles:

Install the required roles with

$ ansible-galaxy install -r requirements.yml

this will install the latest default branch releases.

Or follow the installation instruction for each role on Ansible Galaxy.

https://galaxy.ansible.com/stafwag

Supported GNU/Linux Distributions

It should work on most GNU/Linux distributions. cloud-cloudds is required. cloud-clouds was available on Centos/RedHat 7 but not on Redhat 8. You’ll need to install it manually to use it role on Centos/RedHat 8.

  • Archlinux
  • Debian
  • Centos 7
  • RedHat 7
  • Ubuntu

Role Variables and templates

Variables

See the documentation of the roles in the Requirements section.

  • virt_install_vm: “namespace”

    • skip_if_deployed: boolean default: false.

                            When true:
                              Skip role if the VM is already deployed. The role will exit successfully.
                            When false:
                              The role will exit with an error if the VM is already deployed.
      

Templates.

  • templates/simple_debian: Example template to create a Debian virtual machine.

This template use cloud_localds.cloudinfo to configure the cloud-init user-data.

See the Usage section for an example.

Usage

Create a virtual machine template

This is a file with the role variables to set set up a virtual machine with all the common settings for the virtual machines. In this example vm.hostname and vm.ip_address can be configured for each virtual machine.

  • debian_vm_template.yml:
qemu_img:
  dest: "/var/lib/libvirt/images/.qcow2"
  format: qcow2
  src: /Downloads/isos/debian/cloud/debian-10-generic-amd64.qcow2
  size: "50G"
  owner: root
  group: kvm
  mode: 660
cloud_localds:
  dest: "/var/lib/libvirt/images/_cloudinit.iso"
  config_template: "templates/simple_debian/debian.j2"
  network_config_template: "templates/simple_debian/debian_netconfig.j2"
  cloud_config:
    system_info:
      default_user:
        name: ansible
        passwd: ""
        ssh_authorized_keys:
          - ""
    network:
      dns_nameservers:
        9.9.9.9
      dns_search:
        intern.local
      interface:
        name:
          enp1s0
        address:
          ""
        gateway:
          192.168.123.1
    disable_cloud_init: true
    reboot:
      true
virt_install_import:
  wait: 0
  name: ""
  os_type: Linux
  os_variant: debian10
  network: network:default
  graphics: spice
  disks:
    - "/var/lib/libvirt/images/.qcow2,device=disk"
    - "/var/lib/libvirt/images/_cloudinit.iso,device=cdrom"

Playbook

Playbook to setup a virtual machine:

- name: Install tstdebian2
  hosts: kvmhost
  become: true
  vars:
    vm:
      hostname:
        tstdebian2
      ip_address:
        192.168.123.2/24
  pre_tasks:
    - name: Load the vm template
      include_vars: debian_vm_template.yml
    - name: display qemu_img
      debug:
        msg: 
          - "qemu_img: "
  roles:
    - stafwag.virt_install_vm

September 17, 2021

I published the following diary on isc.sans.edu: “Malicious Calendar Subscriptions Are Back?“:

Did this threat really disappear? This isn’t a brand new technique to deliver malicious content to mobile devices but it seems that attackers started new waves of spam campaigns based on malicious calendar subscriptions. Being a dad, you can imagine that I always performed security awareness with my daughters. Since they use computers and the Internet, my message was always the same: “Don’t be afraid to ask me, there are no stupid questions or shame if you think you did something wrong”… [Read more]

The post [SANS ISC] Malicious Calendar Subscriptions Are Back? appeared first on /dev/random.

September 15, 2021

De VRT radio streams, zoals deze (http://icecast.vrtcdn.be/radio1-high.mp3) hebben sinds kort de neiging om reklame te spelen als je ze start. Niet altijd, maar wel regelmatig.

Zonet (21u en enkele seconden) wou ik naar het nieuws luisteren , maar er begon reklame te spelen. Ik laadde de stream opnieuw en de reklame begon opnieuw (terwijl het nieuws bezig was, maar dat kreeg ik dus niet).

Een paar dagen geleden had ik dat ook al met de Radio 1 stream, telkens reklame als je de stream start. Dit is irritant.

Tijd dus om het script aan te passen en de eerste 60 seconden van de stream te muten. De cronjob om het nieuws te spelen kan dan een minuut eerder starten.

 

UPDATE 2021-09-15: dit werkt

#!/bin/bash

export HOME=/var/www
pkill mplayer

mplayer -volume 0 -slave -input file=/var/www/master http://icecast.vrtcdn.be/radio1-high.mp3 &

sleep 60

echo volume 100 1 > /var/www/master
exit

Naming conventions. Picking the right naming convention is easy if you are all by yourself, but hard when you need to agree upon the conventions in a larger group. Everybody has an opinion on naming conventions, and once you decide on it, you do expect everybody to follow through on it.

Let's consider why naming conventions are (not) important and consider a few examples to help in creating a good naming convention yourself.

My laptop is a 2011 MacBook Air. I’m not a huge Apple fan, it’s just that at the time it had the most interesting hardware features compared to similar laptops. And it’s quite sturdy, so that’s nice.

Over the years I have experimented with installing Linux in parallel to the OS X operating system, but in the end I settled on installing my favorite Linux tools inside OS X using Homebrew, because having two different operating systems on one laptop was Too Much Effortâ„¢. In recent times Apple has decided, in it’s infinite wisdom (no sarcasm at all *cough*), that it will no longer provide operating system upgrades for older hardware. Okay, then. Lately the laptop had become slow as molasses anyway, so I decided to replace OS X entirely with Ubuntu. No more half measures! I chose 20.04 LTS for the laptop because reasons. 🙂

The laptop was really slow…

According to the Ubuntu Community Help Wiki, all hardware should be supported, except Thunderbolt. I don’t use anything Thunderbolt, so that’s OK for me. The installation was pretty straightforward: I just created a bootable USB stick and powered on the Mac with the Option/Alt (⌥) key pressed. Choose EFI Boot in the Startup Manager, and from there on it’s all a typical Ubuntu installation.

screenshot
Startup Manager

I did not bother with any of the customizations described on the Ubuntu Wiki, because everything worked straight out of the box, and besides, the wiki is terribly outdated anyway.

The end result? I now have a laptop that feels snappy again, and that still gets updates for the operating system and the installed applications. And it’s my familiar Linux. What’s next? I’m thinking about using Ansible to configure the laptop.

To finish, I want to show you my sticker collection on the laptop. There’s still room for a lot more!

sticker collection on my laptop. Photo copyright: me.

The post Installing Ubuntu 20.04 LTS on 2011 MacBook Air appeared first on amedee.be.

September 10, 2021

Cover Image

Cultural Assimilation, Theory vs Practice

The other day, I read the following, shared 22,000+ times on social media:

"Broken English is a sign of courage and intelligence, and it would be nice if more people remembered that when interacting with immigrants and refugees."

This resonates with me, as I spent 10 years living on the other side of the world. Eventually I lost my accent in English, which took conscious effort and practice. These days I live in a majority French city and neighborhood, as a native Dutch speaker. When I need to call a plumber, I first have to go look up the words for "drainage pipe." When my barber asks me what kind of cut I want, it mostly involves gesturing and "short".

This is why I am baffled by the follow-up, by the same person:

"Thanks to everyone commenting on the use of 'broken' to describe language. You're right. It is problematic. I'll use 'beginner' from now on."

It's not difficult to imagine the pile-on that must've happened for the author to add this note. What is difficult to imagine is that anyone who raised the objection has actually ever thought about it.

mines

Minesweeper

Consider what this situation looks like to an actual foreigner who is learning English and trying to speak it. While being ostensibly lauded for their courage, they are simultaneously shown that the English language is a minefield where an expression as plain as "broken English" is considered a faux pas, enough to warrant a public correction and apology.

To stay in people's good graces, you must speak English not as the dictionary teaches you, but according to the whims and fashions of a highly volatile and easily triggered mass. They effectively demand you speak a particular dialect, one which mostly matches the sensibilities of the wealthier, urban parts of coastal America. This is an incredibly provincial perspective.

The objection relies purely on the perception that "broken" is a word with a negative connotation. It ignores the obvious fact that people who speak a language poorly do so in a broken way: they speak with interruptions, struggling to find words, and will likely say things they don't quite mean. The dialect demands that you pretend this isn't so, by never mentioning it directly.

But in order to recognize the courage and intelligence of someone speaking a foreign language, you must be able to see past such connotations. You must ignore the apparent subtleties of the words, and try to deduce the intended meaning of the message. Therefor, the entire sentiment is self-defeating. It fell on such deaf ears that even the author seemingly missed the point. One must conclude that they don't actually interact with foreigners much, at least not ones who speak broken English.

The sentiment is a good example of what is often called a luxury belief: a conviction that doesn't serve the less fortunate or abled people it claims to support. Often the opposite. It merely helps privileged, upper-class people feel better about themselves, by demonstrating to everyone how sophisticated they are. That is, people who will never interact with immigrants or refugees unless they are already well integrated and wealthy enough.

By labeling it as "beginner English," they effectively demand an affirmation that the way a foreigner speaks is only temporary, that it will get better over time. But I can tell you, this isn't done out of charity. Because I have experienced the transition from speaking like a foreigner to speaking like one of them. People treat you and your ideas differently. In some ways, they cut you less slack. In other ways, it's only then that they finally start to take you seriously.

Let me illustrate this with an example that sophisticates will surely be allergic to. One time, while at a bar, when I still had my accent, I attempted to colloquially use a particular word. That word is "nigga." With an "a" at the end. In response, there was a proverbial record scratch, and my companions patiently and carefully explained to me that that was a word that polite people do not use.

No shit, Sherlock. You live on a continent that exports metric tons of gangsta rap. We can all hear and see it. It's really not difficult to understand the particular rules. Bitch, did I stutter?

Even though I had plenty of awareness of the linguistic sensitivities they were beholden to, in that moment, they treated me like an idiot, while playing the role of a more sophisticated adult. They saw themselves as empathetic and concerned, but actually demonstrated they didn't take me fully seriously. Not like one of them at all.

If you want people's unconditional respect, here's what did work for me: you go toe-to-toe with someone's alcoholic wine aunt at a party, as she tries to degrade you and your friend, who is the host. You effortlessly spit back fire in her own tongue and get the crowd on your side. Then you casually let them know you're not even one of them, not one bit. Jawdrops guaranteed.

This is what peak assimilation actually looks like.

Ethnic food

The Ethnic Aisle

In a similar vein, consider the following, from NYT Food:

"Why do American grocery stores still have an ethnic aisle?

The writer laments the existence of segregated foods in stores, and questions their utility. "Ethnic food" is a meaningless term, we are told, because everyone has an ethnicity. Such aisles even personify a legacy of white supremacy and colonialism. They are an anachronism which must be dismantled and eliminated wholesale, though it "may not be easy or even all that popular."

We do get other perspectives: shop owners simply put products where their customers are most likely to go look for them. Small brands tend to receive obscure placement, while larger brands get mixed in with the other foods, which is just how business goes. The ethnic aisle can also signal that the products are the undiluted original, rather than a version adapted to local palates. Some native shoppers explicitly go there to discover new ingredients or flavors, and find it convenient.

More so, the point about colonialism seems to be entirely undercut by the mention of "American aisles" in other countries, containing e.g. peanut butter, BBQ sauce and boxed cake mix. It cannot be colonialism on "our" part both when "we" import "their" products, as well as when "they" import "ours". That's just called trade.

Along the way, the article namedrops the exotic ingredients and foreign brands that apparently should just be mixed in with the rest: cassava flour, pomegranate molasses, dal makhani, jollof rice seasoning, and so on. We are introduced to a whole cast of business owners "of color," with foreign-sounding names. We are told about the "desire for more nuanced storytelling," including two sisters who bypassed stores entirely by selling online, while mocking ethnic aisles on TikTok. Which we all know is the most nuanced of places.

I find the whole thing preposterous. In order to even consider the premise, you already have to live in an incredibly diverse, cosmopolitan city. You need to have convenient access to products imported from around the world. This is an enormous luxury, enabled by global peace and prosperity, as well as long-haul and just-in-time logistics. There, you can open an app on your phone and have top-notch world cuisine delivered to your doorstep in half an hour.

For comparison, my parents are in their 70s and they first ate spaghetti as teenagers. Also, most people here still have no clue what to do with fish sauce other than throw it away as soon as possible, lest you spill any. This is fine. The expectation that every cuisine is equally commoditized in your local corner store is a huge sign of privilege, which reveals how provincial the premise truly is. It ignores that there are wide ranging differences between countries in what is standard in a grocery store, and what people know how to make at home.

Even chips flavors can differ wildly from country to country, from the very same multinational brands. Did you know paprika chips are the most common thing in some places, and not a hipster food?

paprika chips by lays

Crucially, in a different time, you could come up with the same complaints. In the past it would be about foods we now consider ordinary. In the future it would be about things we've never even heard of. While the story is presented as a current issue for the current times, there is nothing to actually support this.

To me, this ignorance is a feature, not a bug. The point of the article is apparently to waffle aimlessly while namedropping a lot of things the reader likely hasn't heard of. The main selling point is novelty, which paints the author and their audience as being particularly in-the-know. It lets them feel they are sophisticated because of the foods they cook and eat, as well as the people they know and the businesses they frequent. If you're not in this loop, you're supposed to feel unsophisticated and behind the times.

It's no coincidence that this is published in the New York Times. New Yorkers have a well-earned reputation for being oblivious about life outside their bubble: the city offers the sense that you can have access to anything, but its attention is almost always turned inwards. It's not hard to imagine why, given the astronomical cost of living: surely it must be worth it! And yes, I have in fact spent a fair amount of time there, working. It couldn't just be that life elsewhere is cheaper, safer, cleaner and friendlier. That you can reach an airport in less than 2 hours during rush hour. On a comfortable, modern train. Which doesn't look and smell like an ashtray that hasn't been emptied out since 1975.

But I digress.

"Ethnic aisles are meaningless because everyone has an ethnicity" is revealed to be a meaningless thought. It smacks headfirst into the reality of the food business, which is a lesson the article seems determined not to learn. When "diversity" turns out to mean that people are actually diverse, have different needs and wants, and don't all share the same point of view, they just think diversity is wrong, or at least, outmoded, a "necessary evil." Even if they have no real basis of comparison.

graffiti near school in New York

Negative Progress

I think both stories capture an underlying social affliction, which is about progress and progressivism.

The basic premise of progressivism is seemingly one of optimism: we aim to make the future better than today. But the way it often works is by painting the present as fundamentally flawed, and the past as irredeemable. The purpose of adopting progressive beliefs is then to escape these flaws yourself, at least temporarily. You make them other people's fault by calling for change, even demanding it.

What is particularly noticeable is that perceived infractions are often in defense of people who aren't actually present at all. The person making the complaint doesn't suffer any particular injury or slight, but others might, and this is enough to condemn in the name of progress. "If an [X] person saw that, they'd be upset, so how dare you?" In the story of "broken English," the original message doesn't actually refer to a specific person or incident. It's just a general thing we are supposed to collectively do. That the follow-up completely contradicts the premise, well, that apparently doesn't matter. In the case of the ethnic aisle, the contradictory evidence is only reluctantly acknowledged, and you get the impression they had hoped to write a very different story.

This too is a provincial belief masquerading as sophistication. It mashes together groups of people as if they all share the exact same beliefs, hang-ups and sensitivities. Even if individuals are all saying different things, there is an assumed archetype that overrules it all, and tells you what people really think and feel, or should feel.

To do this, you have to see entire groups as an "other," as people that are fundamentally less diverse, self-aware and curious than the group you're in. That they need you to stand up for them, that they can't do it themselves. It means that "inclusion" is often not about including other groups, but about dividing your own group, so you can exclude people from it. The "diversity" it seeks reeks of blandness and commodification.

In the short term it's a zero-sum game of mining status out of each other, but in the long run everyone loses, because it lets the most unimaginative, unworldly people set the agenda. The sense of sophistication that comes out of this is imaginary: it relies on imagining fault where there is none, and playing meaningless word games. It's not about what you say, but how you say it, and the rules change constantly. Better keep up.

Usually this is associated with a profound ignorance about the actual past. This too is a status-mining move, only against people who are long gone and can't defend themselves. Given how much harsher life was, with deadly diseases, war and famine regular occurences, our ancestors had to be far smarter, stronger and self-sufficient, just to survive. They weren't less sophisticated, they came up with all the sophisticated things in the first place.

When it comes to the more recent past, you get the impression many people still think 1970 was 30, not 51 years ago. The idea that everyone was irredeemably sexist, racist and homophobic barely X years ago just doesn't hold up. Real friendships and relationships have always been able to transcend larger social matters. Vice versa, the idea that one day, everyone will be completely tolerant flies in the face of evidence and human nature. Especially the people who loudly say how tolerant they are: there are plenty of skeletons in those closets, you can be sure of that.

* * *

There's a Dutch expression that applies here: claiming to have invented hot water. To American readers, I gotta tell you: it really isn't hard to figure out that America is a society stratified by race, or exactly how. I figured that out the first time I visited in 2001. I hadn't even left the airport in Philadelphia when it occurred to me that every janitor I had seen was both black and morbidly obese. Completely unrelated, McDonald's was selling $1 cheeseburgers.

Later in the day, a black security guard had trouble reading an old-timey handwritten European passport. Is cursive racist? Or is American literacy abysmal because of fundamental problems in how school funding is tied to property taxes? You know this isn't a thing elsewhere, right?

In the 20 years since then, nothing substantial has improved on this front. Quite the opposite: many American schools and universities have abandoned their mission of teaching, in favor of pushing a particular worldview on their students, which leaves them ill-equipped to deal with the real world.

Ironically this has created a wave of actual American colonialism, transplanting the ideology of intersectionality onto other Western countries where it doesn't apply. Each country has their own long history of ethnic strife, with entirely different categories. The aristocrats who ruled my ancestors didn't even let them get educated in our own language. That was a right people had to fight for in the late 1960s. You want to tell me which words I should capitalize and which I shouldn't? Take a hike.

Not a year ago, someone trying to receive health care here in Dutch was called racist for it, by a French speaker. It should be obvious the person who did so was 100% projecting. I suspect insecurity: Dutch speakers are commonly multi-lingual, but French speakers are not. When you are surrounded by people who can speak your language, when you don't speak a word of theirs, the moron is you, but the ego likes to say otherwise. So you pretend yours is the sophisticated side.

All it takes to pierce this bubble is to actually put the platitudes and principles to the test. No wonder people are so terrified.

September 09, 2021

It's been long overdue, but Planet Grep now does the https dance (i.e., if you try to use an unencrypted connection, it will redirect you to https). Thank you letsencrypt!

I hadn't previously done this because some blogs that we carry might link to http-only images; but really, that shouldn't matter, and we can make Planet Grep itself be a https site even if some of the content is http-only.

Enjoy!

September 08, 2021

Cette interdépendance que l’on essaie d’oublier afin de camoufler l’apport essentiel de l’oisiveté et de la réflexion ouverte.

En 2014, alors que je parlais beaucoup du prix libre, j’ai reçu un gros paiement d’un lecteur. Ce lecteur me remerciait, car les idées que je décrivais l’inspiraient pour son projet de site de jeu d’échecs en ligne. 6 années plus tard, un de mes étudiants a choisi, comme logiciel libre à présenter pour son examen, ce logiciel : Lichess. Il m’a décrit le modèle libre de développement de Lichess, la méthode de don et le prix libre. Lichess est l’un des plus importants sites d’échecs dans le monde et est fréquenté par des grands maitres comme Magnus Carlsen.

Outre une immense fierté de savoir que certaines des graines que j’ai semées ont contribué à de magnifiques forêts, cette anecdote illustre surtout un point très important que l’idéologie Randienne tente à tout prix de camoufler : le succès n’est pas la propriété d’un individu. Un individu n’est jamais productif tout seul, il ne peut pas « se faire tout seul » en dépit de l’image que l’on aime donner des milliardaires. Si les parents de Jeff Bezos ne lui avaient pas donné 300.000$ en lui faisant promettre de trouver un vrai travail une fois les 300.000$ dépensés, il n’y aurait pas d’Amazon aujourd’hui. Chacun d’entre nous utilise des routes, des moyens de communication, des hôpitaux, des écoles et a des échanges intellectuels fournis par la communauté. L’idéologie de la propriété intellectuelle et des brevets nous fait croire qu’il y’a un unique inventeur, un génie solitaire qui mérite de récolter le fruit de ses efforts. C’est entièrement complètement faux. Nous sommes dépendants les uns des autres et nos succès sont essentiellement des chances, saisies ou non, que nous offre la communauté.

De plus, les brevets sont une gigantesque arnaque intellectuelle. J’en ai fait l’expérience moi-même dans un article assez ancien qui a eu pas mal de retentissement sans jamais rencontrer de contradiction.

https://ploum.net/working-with-patents/

Brevets qui ne servent d’ailleurs que l’intérêt des riches et puissants. Amazon, par exemple, a développé une technique pour repérer ce qui se vend bien sur son site afin de le copier et d’en faire sa propre version. Même s’il y’a des brevets. Parce que personne n’a les ressources d’attaquer Amazon sur une histoire de brevets.

https://www.currentaffairs.org/2020/12/how-amazon-destroys-the-intellectual-justifications-for-capitalism

Les brevets sont une arnaque construite sur un concept entièrement fictif : celui de l’inventeur solitaire. Une fiction qui nie l’idée même de l’interdépendance sociale.

Une interdépendance sociale dont l’apport essentiel à la productivité individuelle a été illustré par un généticien, William Muir, qui a décidé de sélectionner les poules qui pondaient le plus d’œufs afin de créer un « super poulailler » qui serait hyper productif. Le résultat a été catastrophique. Les poules qui pondaient le plus d’œufs au sein d’un poulailler étaient en fait les plus agressives qui empêchaient les autres de pondre. Le super poulailler est devenu une boucherie d’ou presque aucun œuf ne sortait et dont la majorité des poules mourraient !

La conclusion est simple : même les poules qui pondent peu ont un rôle essentiel dans la productivité globale de la communauté. Le meilleur poulailler n’est pas composé des meilleures pondeuses, bien au contraire.

https://economicsfromthetopdown.com/2021/01/14/the-rise-of-human-capital-theory/

Grâce aux témoignages de mes lecteurs, je peux affirmer que mes billets de blog ont une influence sur la société à laquelle j’appartiens. Influence que j’estime essentiellement positive, voire très positive, selon mes propres critères. Lichess en est un exemple spectaculaire, mais je reçois des mails beaucoup plus intimes qui vont dans le même sens et qui me touchent beaucoup (même si j’ai pris la décision de ne plus y répondre systématiquement). Je peux donc affirmer que je suis utile à mon humble échelle.

Au cours de ma carrière, je ne peux trouver aucun exemple où mon travail salarié ait jamais eu le moindre impact et où mon utilité a été démontrée. Pire : je ne vois pas un seul impact positif des entreprises entières pour lesquelles j’ai travaillé. En étant très optimiste, je peux affirmer qu’on a amélioré la rentabilité de certains de nos clients. Mais ce n’est pas vraiment un impact sociétal positif. Et ce rendement est de toute façon noyé dans une gabegie de projets abscons et de procédures administratives. Pendant dix ans, j’ai été payé dans des super-poulaillers, dans des entreprises qui sont elles-mêmes en compétition. Pour un résultat soit nul, soit nocif pour l’humanité et la planète car augmentant la consommation globale.

À l’opposé, je vois directement l’impact des projets auxquels j’ai contribué sans rétribution, notamment les projets de logiciels libres. Le développeur Mike Williamson est arrivé à la même conclusion.

https://mike.zwobble.org/2021/08/side-projects-vs-industry/

Si vous cherchez mon nom sur Wikipedia, vous arriverez sur la page d’un projet auquel j’ai consacré plusieurs années de sommeil sans toucher le moindre centime.

https://fr.wikipedia.org/wiki/Getting_Things_Gnome

Revenu de base

C’est peut-être pour ça que le revenu de base me semble tellement essentiel. En 2013, je tentais de vous convaincre que le revenu de base était une bonne idée et de signer la pétition pour forcer les instances européennes d’étudier la question. Hélas, le nombre de signatures n’avait pas été atteint.

https://ploum.net/pourquoi-vous-etes-sans-le-savoir-favorable-au-revenu-de-base/

Huit ans plus tard, une nouvelle pétition vient de voir le jour. Si vous êtes citoyen européen, je vous invite vivement à la signer. C’est très facile et très officiel. Il faut mettre vos données personnelles, mais pas votre email. Il est nécessaire d’obtenir un minimum de signatures dans tous les pays d’Europe. N’hésitez pas à partager avec vos contacts internationaux.

https://eci.ec.europa.eu/014/public/#/screen/home

Les observables

Lorsqu’on vous parle de la productivité d’un individu ou du mérite des personnes riches, rappelez-vous l’histoire des poulaillers.

Mais pour les poules, c’est facile. Il suffit de mesurer les œufs pondus. Le problème avec le capitalisme moderne, c’est qu’on se plante tout le temps dans les métriques. Or, si on utilise une mauvaise métrique, on va optimiser tout le système pour avoir des mauvais résultats.

J’ai beaucoup glosé sur ce paradigme des métriques, que j’appelle des « observables ». Je tourne en rond autour du même thème : on mesure la productivité à l’aide des heures de travail (vu que le salarié moyen ne pond pas), donc on crée des heures de travail, donc les jobs servent à remplir le plus d’heures possible. Ce que j’appelle le principe d’inefficacité maximale. Au final, on passe 8h par jour à tenter de brûler la planète afin, une fois sorti du bureau, de pouvoir se payer des légumes bio en ayant l’impression de sauver la même planète.

https://ploum.net/le-principe-dinefficacite-maximale/

Outre les heures de travail, il y’a d’autres métriques absurdes comme les clics, les pages vues et ce genre de choses. Les métriques des gens qui font du marketing : faire le plus de bruit possible ! Le département marketing, c’est un peu un super-poulailler où on a mis tous les coqs les plus bruyants. Et on s’étonne de ne pas avoir un seul œuf. Mais beaucoup de bruit.

https://ploum.net/le-silence-au-milieu-du-bruit/

L’effet des métriques absurdes a un impact direct sur votre vie. Genre si vous utilisez Microsoft Team au travail. Car désormais, votre manager va pouvoir avoir des statistiques sur votre utilisation de Teams. Le programmeur hyper concentré qui a coupé Teams pour coder une super fonctionnalité va bien vite se faire virer à cause de mauvaises statistiques. Et votre vie privée ? Elle ne rentre pas dans les plans du superpoulailler !

https://www.zdnet.com/article/i-looked-at-all-the-ways-microsoft-teams-tracks-users-and-my-head-is-spinning/

Comme plus personne n’a le temps de réfléchir (vu qu’il n’y a pas de métriques sur le sujet et qu’au contraire réfléchir bousille d’autres métriques), l’avenir appartient à ceux qui arrivent à maximiser les métriques. Ou mieux : qui arrive à faire croire qu’ils sont responsables de métriques maximisées. Changer de travail régulièrement permet de ne jamais vraiment exposer son incompétence et de montrer en grade à chaque étape, augmentant ainsi son salaire jusqu’à devenir grand manager hyper bien payé dans un univers où les métriques sont de plus en plus floues. La compétence est remplacée par l’apparence de compétence, qui est essentiellement de la confiance en soi et de l’opportunisme politique. Cela rejoint un peu la thèse de Daniel Drezner développée dans « The Ideas Industry » : les idées simples, prémâchées, faciles à s’approprier (genre TED) prennent le pas sur les analyses profondes et plus subtiles. C’est également un constat fait par Cal Newport dans « A World Without Email » où il dénonce la mentalité de « ruche bourdonnante » de toute entreprise moderne.

Vous êtes entrepreneur ou indépendant ? C’est pareil : vous maximisez les métriques absurdes de vos clients. Si vous avez de la chance d’avoir des clients ! Sinon, vous passez votre temps à optimiser les métriques que vous offrent Facebook, Google Analytics ou Amazon en ayant l’impression de bosser à votre projet. Y’a même un métier entier qui ne fait qu’optimiser une métrique offerte par Google : le SEO.

Il y a quelques années, le simple fait d’avoir émis cette idée m’a valu que des professionnels du secteur s’organisent pour qu’une recherche à mon nom renvoie vers des injures de leur cru. Cette anecdote illustre bien le problème des métriques absurdes : il est impossible de faire comprendre qu’une métrique est absurde à ceux qui payent pour optimiser cette métrique et à ceux qui ont bâti leur carrière sur la même métrique. Une simple remise en question génère une violence complètement disproportionnée, religieuse.

Religion et violence

Le repli identitaire, la religiosité ou la plupart des opinions conservatrices sont générés par l’angoisse et le sentiment de ne pas comprendre. Ce n’est pas une analyse politique, mais bien neurologique. Il suffit de désactiver quelques neurones dans le cerveau pour que, soudainement, l’angoisse ne soit plus liée à ce repli. Comme on ne peut pas désactiver ces neurones chez tout le monde, il reste une solution qui a déjà fait ses preuves : l’éducation, qui permet de comprendre et d’être moins angoissé.

https://www.lemonde.fr/passeurdesciences/article/2015/10/21/moins-croire-en-dieu-avec-la-stimulation-magnetique_6001729_5470970.html

La religion n’est de toute façon qu’un prétexte. Ce ne sont pas les interprétations religieuses qui sont la cause de violences ou de repli, elles en sont au contraire le symptôme, l’excuse.

https://medium.com/incerto/religion-violence-tolerance-progress-nothing-to-do-with-theology-a31f351c729e

Le poulailler sans-tête !

En utilisant religieusement les mauvaises métriques, nous sommes en train de faire de la planète une sorte de super-poulailler où la bêtise et la stupidité sont optimisées. C’est d’ailleurs la définition même de la foi : croire sans poser de question, sans chercher à comprendre. La foi est la bêtise élevée au rang de qualité. L’invasion du capitole par les partisans de Trump en a été l’illustration suprême : des gens pas très malins, ayant la foi que l’un d’entre eux avait un plan et qu’ils allaient le suivre. Sauf qu’il n’y avait pas de plan, que cette invasion était un « meme » comme l’est Q : une simple idée lancée sur les réseaux sociaux qui s’est créé une auto-importance grâce à la rumeur et au bouche-à-oreille virtuel. D’ailleurs, une fois dans le capitole, personne ne savait quoi faire. Ils se sont assis sur les fauteuils pour se sentir importants, ont pris des selfies, ont tenté de trouver des complots croustillants, en quelques secondes, dans les centaines de pages de documents législatifs qui sont probablement disponibles sur le site du gouvernement. Quand votre culture politique est alimentée essentiellement par des séries d’actions sur Netflix, la révolution trouve vite ses limites.

Comme le souligne très bien Cory Doctorrow, les memes et les fake news ne sont pas la réalité, mais ils sont l’expression d’un fantasme. Les memes sur Internet ne sont pas créés pour décrire la réalité, mais pour tenter de faire plier la réalité à nos désirs.

https://locusmag.com/2019/07/cory-doctorow-fake-news-is-an-oracle/

Mais pas besoin d’aller aussi loin. Bien avant Trump, la Belgique avait connu le concept du « politicien-meme » avec le député Laurent Louis. Député tellement absurde que j’avais ironisé sur le fait qu’il n’était qu’une blague à travers un article satirique. Article qui avait d’ailleurs eu pour résultat que Laurent Louis lui-même avait posté son certificat de naissance sur les réseaux sociaux, pour prouver qu’il existait. Cette non-perception de l’ironie m’avait particulièrement frappé.

Comme Trump, Laurent Louis avait fini par trouver un créneau et des partisans. Assez pour foutre un peu le bordel, pas assez pour ne pas disparaitre dans l’oubli comme une parenthèse illustrant les faiblesses d’un système politique bien trop optimisé pour récompenser le marketing et la bêtise. Mais je tombe dans le pléonasme.

https://ploum.net/le-depute-qui-nexistait-pas/

S’évader du poulailler

J’achète un recueil de nouvelles de Valery Bonneau. Je le prête à ma mère avant même de le lire. Elle me dit de lire absolument la première nouvelle,  » Putain de cafetière « . Je me plonge. Je tombe de mon fauteuil de rire. Franchement, le coup du frigo américain avec un code PIN, j’en rigole encore.

Profitez-en ! (en version papier, c’est encore plus délectable !)

https://www.valerybonneau.com/romans/nouvelles-noires-pour-se-rire-du-desespoir/putain-de-cafetiere

Envie d’un roman gonflé à la vitamine ? Besoin de vous évader des confinements et couvre-feux à gogo ? Printeurs de Ploum est fait pour vous!

Ce n’est pas moi qui le dis, c’est une critique que je ne me lasse pas de relire :

https://albdoblog.com/2021/01/20/printeurs-ploum/

D’ailleurs, si vous avez lu Printeurs, n’hésitez pas à donner votre avis sur Senscritique et Babelio. Je déteste Senscritique, mais je n’ai pas encore trouvé d’alternative durable.

https://www.senscritique.com/livre/Printeurs/43808921

https://www.babelio.com/livres/Dricot-Printeurs-Science-fiction/1279338

Un autre plugin Firefox qui me sauve la vie et pour lequel j’ai souscrit un abonnement premium à prix libre :


https://ninja-cookie.com/

Fini de paramétrer les cookies. Le plugin les refuse automatiquement au maximum refus possible. C’est parfait et indispensable.

Ça en dit long sur l’état du web actuel. Quand on voit le nombre de protections qu’il faut avoir pour pouvoir tout simplement « lire » le contenu des pages web sans avoir le cerveau qui frit et sans être espionné de tous les côtés, on comprend mieux l’intérêt d’un protocole comme Gemini qui est conçu à la base pour être le moins extensible possible !

Conseil BD

Après les magnifiques « L’Autre Monde » et « Mary la Noire », je découvre une nouvelle facette de l’univers de Florence Magnin . « L’héritage d’Émilie ».

J’ai découvert Magnin par hasard, dans ma librairie préférée. L’Autre Monde m’a interpellé. Le dessin était magnifique, mais d’une naïveté particulière. Je n’étais pas certain d’aimer. Je n’ai pas aimé, j’ai littéralement été aspiré. Ce mélange de naïveté et d’univers pour adulte, de fantastique à la fois désuet et incroyablement moderne. L’héritage d’Émilie ne fait pas exception. En fait, il transcende même les deux autres en mélangeant le Paris des années folles et les légendes celtiques d’Irlande, le tout dans une œuvre de fantastique champêtre qui glisse brusquement dans le space opera intergalactique. Oui, c’est complètement incroyable. Et oui, j’adore.

Photo by Artem Beliaikin on Unsplash

Oubliez un instant les réseaux sociaux et abonnez-vous par mail ou par RSS (max 2 billets par semaine et rien d’autre). Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, offrez et partagez ses livres.

Ce texte est publié sous la licence CC-By BE.

I'm excited to announce that Acquia has signed a definitive agreement to acquire Widen, a digital asset management (DAM) and product information management (PIM) company.

The Acquia and Widen logos shown next to each other

It's not hard to understand how Widen fits Acquia's strategy. Our goal is to build the best Digital Experience Platform (DXP). Content is at the heart of any digital experience. By adding a DAM and PIM to our platform, our customers will be able to create better content, more easily. That will result in better customer experiences. Plain and simple.

Widen is for organizations with larger marketing teams managing one or more brands. These teams create thousands of "digital assets": images, videos, PDFs and much more. Those digital assets are used on websites, mobile applications, in-store displays, presentations, etc. Managing thousands of files, plus all the workflows to support them, is difficult without the help of a DAM.

For commerce purposes, marketers need to correlate product images with product information like pricing, sizing, or product specifications. To do so, Widen offers a PIM. Widen built their PIM on top of their DAM — an approach that is both clever and unique. Widen's PIM can ingest product content from proprietary systems, master data management (MDM) platforms, data lakes, and more. From there, marketers can aggregate, synthesize, and syndicate product content across digital channels.

In short, organizations need a lot of content to do business. And online commerce can't exist without product information. It's why we are so excited about Widen, and the ability to add a DAM and PIM to our product portfolio.

Because content is at the heart of any digital experience, we will build deep integrations between Widen and Acquia's DXP. So in addition to acquiring Widen, we are making a large investment in growing Widen's engineering team. That investment will go towards extending the existing Widen module for Drupal, and creating integrations with Acquia's products: Acquia Site Studio, Acquia Campaign Studio (Mautic), Acquia Personalization, and more. Digital asset management will be a core building block of our DXP.

Needless to say, we will continue to support and invest in Widen working with other Content Management Systems and Digital Experience Platforms. We are building an open DXP; one of our key principles is that customers should be able to integrate with their preferred technologies, and that might not always be ours. By growing the engineering team, we can focus on building Drupal and Acquia integrations without disrupting the existing roadmap and customer commitments.

A few other facts that might be of interest:

So last but not least, I'd like to welcome all of Widen's customers and employees to the Acquia family. I'm excited to see what we will accomplish together.

September 07, 2021

In this last post on the infrastructure domain, I cover the fifth and final viewpoint that is important for an infrastructure domain representation, and that is the location view. As mentioned in previous posts, the viewpoints I think are most representative of the infrastructure domain are:

Like with the component view, the location view is a layered approach. While I initially wanted to call it the network view, "location" might be a broader term that matches the content better. Still, it's not a perfect name, but the name is less important than the content, not?

September 04, 2021

I had been a happy user of the Nokia 6.1 I bought 3 and a half years ago, but with battery life slowly going down and both OS major updates and security-updates having stopped it was time to find a replacement.

Although the tech reporters and vloggers were underwhelmed by the screen (no Oled or Amoled, only 60Hz refresh rate) and CPU (the SM4350 Snapdragon 480 is considered too slow), I choose the Nokia X20 because:

  • bare-bones Android One experience
  • 3 years OS & security updates
  • 3 years warranty
  • 128GB memory & 8GB RAM
  • 5G

And you know what? This old man has no issues whatsoever with the screen or CPU. The only downside: the eco-friendly backcover is pretty ugly. But all in all pretty good hardware for a very reasonable price (€339 incl. VAT), so if all is well I won’t bother you with boring smartphone-news until 2024? :-)

September 02, 2021

I published the following diary on isc.sans.edu: “Attackers Will Always Abuse Major Events in our Lifes“:

All major events in our daily life are potential sources of revenue for attackers. When elections or major sports events are organized, attackers will surf on these waves and try to make some profit or collect interesting data (credentials). It’s the same with major meteorological phenomena. The hurricane “Ida” was the second most intense hurricane to hit the state of Louisiana on record, only behind “Katrina”… [Read more]

The post [SANS ISC] Attackers Will Always Abuse Major Events in our Lifes appeared first on /dev/random.

September 01, 2021

Blogging sometimes feels like talking to an imaginary friend. It's an interesting comparison because it could help me write more regularly. For example: I can picture myself going to dinner with my imaginary friend. Once we sit down, what would we talk about? What would I share?

I'd share that I've been doing well the past year.

Work is going well. I'm fortunate to help lead at a growing software company. We continue to hit record sales quarter after quarter, and hired more than 250 new employees in 2021 alone. Keeping up with all the work can be challenging but I continue to have fun and learn a lot, which is the most important part.

Most days I work from home. Working from home consists of 8 hours of Zoom meetings, followed by email, presentation and planning work. I finish most work days energized and drained at the same time.

Over the course of two years, I've created a home office setup that is more comfortable, more ergonomic, and more productive than my desk at the office. I invested in an ergonomic chair, standing desk, camera setup, a second screen, and even a third screen. Possibly an interesting topic for a future blog post.

Despite having a great home office setup, I'd like to work more from interesting locations. I'm writing this blog post from an island on Lake Winnipesaukee in New Hampshire where we have a management offsite. Working from an island is as awesome as it sounds. The new hybrid work arrangement provides that extra flexibility.

A chair with a view of Lake Winnipesaukee
Overlooking Lake Winnipesaukee in New Hampshire. Coffee and laptop for morning blogging.

When not working, I've been enjoying the summer in Boston. We moved from the suburbs to the city this year, and have been busy exploring our new neighborhood. We love it!

I've been very happy with our decision to move to the city, except for one thing: tennis. I love playing tennis with a coach, and that has been nearly impossible in the city. As a result I haven't played tennis for months — the lack of workout routine has been really bothering me. Because I love racket sports the most, I started to explore if there are good squash, pickleball or table tennis options in downtown Boston. Recommendations welcome!

Last but not least, we spent some time at Cape Cod this summer, and traveled to Iceland for a weekend. I'll tie off this blog post with a few photos of those trips.

An American flag waving in the light of the moon
A red moon over the water in Cape Cod.
Eating dinner outside overlooking the ocean
Dinner at Cape Cod.
A aarshmallow over a camp fire
S'mores on the beach.
Three people walking alongside dries lava
Hiking alongside the lava from the Gerlingadalur volcano in Iceland. The volcano was active, hence the smoke coming from the lava.

In my previous post, I started with the five different views that would support a good view of what infrastructure would be. I believe these views (component, location, process, service, and zoning) cover the breadth of the domain. The post also described the component view a bit more and linked to previous posts I made (one for services, another for zoning).

The one I want to tackle here is the most elaborate one, also the most enterprise-ish, and one that always is a balance on how much time and effort to put into it (as an architect), as well as hoping that the processes are sufficiently standardized in a flexible manner so that you don't need to cover everything again and again in each project.

So, let's talk about processes...

August 30, 2021

Alors que je déclipsais le pied de mes pédales après ma grande traversée du Massif central en VTT en compagnie de Thierry Crouzet, mon téléphone m’afficha un mail au titre à la fois évident et incompréhensible, inimaginable : « Roudou nous a quittés ».

Avec Internet est apparu une nouvelle forme de relation sociale, une nouvelle forme d’interaction voire, j’ose le terme, d’amitié. Une amitié envers des personnes avec qui on se découvre des affinités intellectuelles, mais qu’on ne verra pas souvent voire jamais. Une amitié tout de même. Une amitié qui peut mener sur une complicité, sur la création de projets communs. Une amitié qui dépasse bien des relations en chair et en os que la proximité nous impose quotidiennement.

Jean-Marc Delforge, Roudou pour les intimes, était pour moi de ces amitiés au long cours. Lecteur de mon blog depuis des années, utilisateur de logiciel libre et illustrateur amateur, il m’a envoyé le tout premier fan-art de Printeur et signera ensuite la couverture du premier EPUB Printeurs.

À force de discussions, nous créerons ensemble le webcomic « Les startupeurs » dont j’ai empilé les scénarios avant que, malheureusement, Roudou ne trouve plus le temps pour les dessiner. Des personnages d’employés un peu désabusés (dont l’un est ma parodie selon Roudou), rêvant de créer leurs startup et addicts de la machine à café (une trouvaille de Roudou !).

https://ploum.net/les-startupeurs-un-nouveau-webcomic/

On s’amusait comme des fous avec ces idées, s’essayant au cartoon politique, partageant, discutant et se découvrant une passion commune pour le VTT.

Car Roudou était plus qu’un passionné de VTT. C’était un meneur, un créateur de trace et le fondateur du forum VTTnet. Dans son sillage, impossible de ne pas pédaler.

En 2015, il m’invita à le rejoindre avec mon filleul Loïc pour 3 jours de VTT intensifs en compagnie des membres du forum.

Roudou, sa fille Noémie, mon filleul Loïc et les autres malades de VTTNet en 2015

Par le plus grand des hasards, Loïc et moi sommes repassés dans la région début juillet pour un trip bikepacking. Lorsque Roudou a découvert cela, il m’a immédiatement envoyé un message pour me dire qu’on s’était raté de peu. Alors que Loïc et moi nous prélassions au bord du lac de l’Eau d’Heure, lui était probablement en train d’y faire du bateau. Il rigolait en lisant l’itinéraire que nous avions pris, me disant qu’il aurait pu nous guider, qu’il habitait tout près.

Je me suis senti triste à l’idée d’avoir manqué une telle opportunité de pédaler ensemble. J’ai promis qu’on referait le trip l’année prochaine. Que ce serait vraiment chouette de se retrouver sur un vélo (même si, pour des raisons de santé qu’il ne voulait pas détailler, le VTT de Roudou était devenu électrique).

À un message un peu accusateur me demandant comment j’osais venir pédaler dans sa région sans le prévenir, je répondis que j’étais persuadé qu’il habitait bien plus à l’ouest.

La réponse de Roudou ne se fit pas attendre : « Ma femme aussi me dit souvent que je suis bien trop à l’ouest. »

Ce fut le dernier message que je reçus de lui. Le 16 juillet, j’embarquais pour 1000km de VTT essentiellement déconnectés, me promettant d’aller rouler avec Roudou l’été prochain.

Mais alors que je pédalais loin de tout, la mort l’a surpris, interrompant à jamais notre fil de discussion, plongeant les startupeurs, les vététistes, sa femme, ses filles et ses amis dans une tristesse infinie.

Roudou va me manquer. Ses crobards et ses photos humoristiques envoyés pour réagir à mes billets de blog et mes livres vont me manquer. Les startupeurs, même s’ils étaient en hibernation, vont me manquer (je n’ai d’ailleurs pas de copie de cette œuvre commune, peut-être perdue). Lorsque je me plongerai dans la suite de Printeurs, je sais que les personnages auront une pensée pour Roudou, ce lecteur qui leur faisait prendre corps sous sa tablette graphique.

Je garderai toujours en moi ce regret d’avoir oublié de le prévenir, d’avoir gâché cette dernière opportunité avant qu’il parte pédaler un peu plus à l’Ouest. Un peu trop à l’Ouest…

Salut l’artiste, salut Roudou ! Nous continuerons à suivre tes traces en pensant à toi.

Oubliez un instant les réseaux sociaux et abonnez-vous par mail ou par RSS (max 2 billets par semaine et rien d’autre). Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, offrez et partagez ses livres.

Ce texte est publié sous la licence CC-By BE.

I published the following diary on isc.sans.edu: “Cryptocurrency Clipboard Swapper Delivered With Love“:

Be careful if you’re a user of cryptocurrencies. My goal is not to re-open a debate about them and their associated financial risks. No, I’m talking here about technical risk. Wallet addresses are long strings of characters that are pretty impossible to use manually. It means that you’ll use your clipboard to copy/paste your wallets to perform payments. But some malware monitors your clipboard for “interesting data” (like wallet addresses) and tries to replace it with another one. If you perform a payment operation, it means that you will transfer some BTC or XMR to the wrong wallet, owned by the attacker… [Read more]

The post [SANS ISC] Cryptocurrency Clipboard Swapper Delivered With Love appeared first on /dev/random.

August 28, 2021

Cover Image

Making code reusable is not an art, it's a job

Extensibility of software is a weird phenomenon, very poorly understood in the software industry. This might seem strange to say, as you are reading this in a web browser, on an operating system, desktop or mobile. They are by all accounts, quite extensible and built out of reusable, shared components, right?

But all these areas are suffering enormous stagnation. Microsoft threw in the towel on browsers. Mozilla fired its engineers. Operating systems have largely calcified around a decades-old feature set, and are just putting up fortifications. The last big shift here was Apple's version of the mobile web and app store, which ended browser plug-ins like Flash or Java in one stroke.

Most users are now silo'd inside an officially approved feature set. Except for Linux, which is still figuring out how audio should work. To be fair, so is the web. There's WebAssembly on the horizon, but the main thing it will have access to is a creaky DOM and an assortment of poorly conceived I/O APIs.

It sure seems like the plan was to have software work much like interchangeable legos. Only it didn't happen at all, not as far as end-users are concerned. Worse, the HTTP-ificiation of everything has largely killed off the cow paths we used to have. Data sits locked behind proprietary APIs. Interchange doesn't really happen unless there is a business case for it. The default of open has been replaced with a default of closed.

This death of widespread extensibility ought to seem profoundly weird, or at least, ungrappled with.

We used to collect file types like Pokémon. What happened? If you dig into this, you work your way through types, but then things quickly get existential: how can a piece of code do anything useful with data it does not understand? And if two programs can interpret and process the same data the same way, aren't they just the same code written twice?

Most importantly: does this actually tell us anything useful about how to design software?

Mallard ducks
The Birds of America, John James Audubon (1827)

Well?

Let's start with a simpler question.

If I want a system to be extensible, I want to replace a part with something more specialized, more suitable to my needs. This should happen via the substitution principle: if it looks like a duck, walks like a duck and quacks like a duck, it's a duck, no matter which kind. You can have any number of sub-species of ducks, and they can do things together, including making weird new little ducks.

So, consider:

If I have a valid piece of code that uses the type Animal, I should be able to replace Animal with the subtype Duck, Pig or Cow and still have valid code.

True or False? I suspect your answer will depend on whether you've mainly written in an object-oriented or functional style. It may seem entirely obvious, or not at all.

This analogy by farm is the usual intro to inheritance: Animal is the supertype. When we call .say(), the duck quacks, but the cow moos. The details are abstracted away and encapsulated. Easy. We teach inheritance and interfaces this way to novices, because knowing what sounds your objects make is very important in day-to-day coding.

But, seriously, this obscures a pretty important distinction. Understanding it is crucial to making extensible software. Because the statement is False.

So, the farmer goes to feed the animals:

type GetAnimal  = () => Animal;
type FeedAnimal = (animal: Animal) => void;

How does substitution apply here? Well, it's fine to get ducks when you were expecting animals. Because anything you can do to an Animal should also work on a Duck. So the function () => Duck can stand-in for an () => Animal.

But what about the actions? If I want to feed the ducks breadcrumbs, I might use a function feedBread which is a Duck => void. But I can't feed that same bread to the cat and I cannot pass feedBread to the farmer who expects an Animal => void. He might try to call it on the wrong Animal.

This means the allowable substitution here is reversed depending on use:

  • A function that provides a Duck also provides an Animal.
  • A function that needs an Animal will also accept a Duck.

But it doesn't work in the other direction. It seems pretty obvious when you put it this way. In terms of types:

  • Any function () => Duck is a valid substitute for () => Animal.
  • Any function Animal => void is a valid substitute for Duck => void.

It's not about using a type T, it's about whether you are providing it or consuming it. The crucial distinction is whether it appears after or before the =>. This is why you can't always replace Animal with Duck in just any code.

This means that if you have a function of a type T => T, then T appears on both sides of =>, which means neither substitution is allowed. You cannot replace the function with an S => S made out of a subtype or supertype S, not in general. It would either fail on unexpected input, or produce unexpected output.

This shouldn't be remarkable at all among people who code in typed languages. It's only worth nothing because intros to OO inheritance don't teach you this, suggesting the answer is True. We use the awkward words covariant and contravariant to describe the two directions, and remembering which is which is hard.

I find this quite strange. How is it people only notice one at first?

let duck: Duck = new Duck();
let animal: Animal = duck;
class Duck extends Animal {
  method() {
    // ...
  } 
}

Here's one explanation. First, you can think of ordinary values as being returned from an implicit getter () => value. This is your default mental model, even if you never really thought about it.

Second, it's OO's fault. When you override a method in a subclass, you are replacing a function (this: Animal, ...) => with a function (this: Duck, ...) => . According to the rules of variance, this is not allowed, because it's supposed to be the other way around. To call it on an Animal, you must invoke animal.say() via dynamic dispatch, which the language has built-in.

Every non-static method of class T will have this: T as a hidden argument, so this constrains the kinds of substitutions you're allowed to describe using class methods. Because when both kinds of variance collide, you are pinned at one level of abstraction and detail, because there, T must be invariant.

This is very important for understanding extensibility, because the common way to say "neither co- nor contravariant" is actually just "vendor lock-in".

birds

The Mirage of Extensibility

The goal of extensibility is generally threefold:

  • Read from arbitrary sources of data
  • Perform arbitrary operations on that data
  • Write to arbitrary sinks of data

Consider something like ImageMagick or ffmpeg. It operates on a very concrete data type: one or more images (± audio). These can be loaded and saved in a variety of different formats. You can apply arbitrary filters as a processing pipeline, configurable from the command line. These tools are swiss army knives which seem to offer real extensibility.

type Input<T> = () => T;
type Process<T> = T => T;
type Output<T> = T => void;
functions of one or two Ts

Formally, you decode your input into some shared representation T. This forms the glue between your processing blocks. Then it can be sent back to any output to be encoded.

It's crucial here that Process has the same input and output type, as it enables composition of operations like lego. If it was Process<A, B> instead, you would only be able to chain certain combinations (A → B, B → C, C → D, ...). We want to have a closed, universal system where any valid T produces a new valid T.

Of course you can also define operators like (T, T) => T. This leads to a closed algebra, where every op always works on any two Ts. For the sake of brevity, operators are implied below. In practice, most blocks are also configurable, which means it's an options => T => T.

This seems perfectly extensible, and a textbook model for all sorts of real systems. But is it really? Reality says otherwise, because it's engineering, not science.

Consider a PNG: it's not just an image, it's a series of data blocks which describe an image, color information, physical size, and so on. To faithfully read a PNG and write it out again requires you to understand and process the file at this level. Therefor any composition of a PNGInput with a PNGOutput where T is just pixels is insufficient: it would throw away all the metadata, producing an incomplete file.

Now add in JPEG: same kind of data, very different compression. There are also multiple competing metadata formats (JFIF, EXIF, ...). So reading and writing a JPEG faithfully requires you to understand a whole new data layout, and store multiple kinds of new fields.

This means a swiss-army-knife's T is really some kind of wrapper in practice. It holds both data and metadata. The expectation is that operations on T will preserve that metadata, so it can be reattached to the output. But how do you do that in practice? Only the actual raw image data is compatible between PNG and JPEG, yet you must be able to input and output either.

meta = {
  png?: {...}
  jpeg?: {...}
}

If you just keep the original metadata in a struct like this, then a Process<T> interested in metadata has to be aware of all the possible image formats that can be read, and try them all. This means it's not really extensible: adding a new format means updating all the affected Process blocks. Otherwise Input<T> and Process<T> don't compose in a useful sense.

meta = {
  color: {...},
  physical: {...},
  geo: {...},
}

If you instead harmonize all the metadata into a single, unified schema, then this means new Input<T> and Output<T> blocks are limited to metadata that's already been anticipated. This is definitely not extensible, because you cannot support any new concepts faithfully.

If you rummage around inside ImageMagick you will in fact encounter this. PNG and JPEG's unique flags and quirks are natively supported.

meta = {
  color: {...},
  physical: {...},
  geo: {...},
  x-png?: {...}
  x-jpeg?: {...}
}

One solution is to do both. You declare a standard schema upfront, with common conventions that can be relied on by anyone. But you also provide the ability to extend it with custom data, so that specific pairs of Input/Process/Output can coordinate. HTTP and e-mail headers are X-able.

meta = {
  img?: {
    physical?: {...},
    color?: {...},
  },
  fmt?: {
    png?: {...},
    jfif?: {...},
    exif?: {...},
  },
},

The problem is that there is no universal reason why something should be standard or not. Standard is the common set of functionality "we" are aware of today. Non-standard is what's unanticipated. This is entirely haphazard. For example, instead of an x-jpeg, it's probably better to define an x-exif because Exif tags are themselves reusable things. But why stop there?

Mistakes stick and best practices change, so the only way to have a contingency plan in place is for it to already exist in the previous version. For example, through judicious use of granular, optional namespaces.

The purpose is to be able to make controlled changes later that won't mess with most people's stuff. Some breakage will still occur. The structure provides a shared convention for anticipated needs, paving the cow paths. Safe extension is the default, but if you do need to restructure, you have to pick a new namespace. Conversion is still an issue, but at least it is clearly legible and interpretable which parts of the schema are being used.

One of the smartest things you can do ahead of time is to not version your entire format as v1 or v2. Rather, remember the version for any namespace you're using, like a manifest. It allows you to define migration not as a transaction on an entire database or dataset at once, but rather as an idempotent process that can be interrupted and resumed. It also provides an opportunity to define a reverse migration that is practically reusable by other people.

This is how you do it if you plan ahead. So naturally this is not how most people do it.

birds

N+1 Standards

X- fields and headers are the norm and have a habit of becoming defacto standards. When they do, you find it's too late to clean it up into a new standard. People try anyway, like with X-Forwarded-For vs Forwarded. Or -webkit-transform vs transform. New software must continue to accept old input. It must also produce output compatible with old software. This means old software never needs to be updated, which means new software can never ditch its legacy code.

Let's look at this story through a typed lens.

What happens is, someone turns an Animal => Animal into a Duck => Duck without telling anyone else, by adding an X- field. This is fine, because Animal ignores unknown metadata, and X- fields default to none. Hence every Animal really is a valid Duck, even though Duck specializes Animal.

Slowly more people replace their Animal => Animal type with Duck => Duck. Which means ducks are becoming the new defacto Animal. But then someone decides it needed to be a Chicken => Chicken instead, and that chickens are the new Animal. Not everyone is on-board with that.

So you need to continue to support the old Duck and the new Chicken on the input side. You also need to output something that passes as both Duck and Chicken, that is, a ChickenDuck. Your signature becomes:

(Duck | Chicken | ChickenDuck) => ChickenDuck

This is not what you wanted at all, because it always lays two eggs at the same time, one with an X and one without. This is also a metaphor for IPv4 vs IPv6.

If you have one standard, and you make a new standard, now you have 3 standards: the old one, the theoretical new one, and the actual new one.

replacing one T=>T in a chain with S=>S

Invariance pops up again. When you have a system built around signature T => T, you cannot simply slot in a S => S of a super or sub S. Most Input and Output in the wild still only produces and consumes T. You have to slot in an actual T => S somewhere, and figure out what S => T means.

Furthermore, for this to do something useful in a real pipeline, T already has to be able to carry all the information that S needs. And S => T cannot strip it out again. The key is to circumvent invariance: neither type is really a subtype or supertype of the other. They are just different views and interpretations of the same underlying data, which must already be extensible enough.

Backwards compatibility is then the art of changing a process Animal => Animal into a Duck => Duck while avoiding a debate about what specifically constitutes quacking. If you remove an X- prefix to make it "standard", this stops being true. The moment you have two different sources of the same information, now you have to decide whose job it is to resolve two into one, and one back into two.

This is particularly sensitive for X-Forwarded-For, because it literally means "the network metadata is wrong, the correct origin IP is ..." This must come from a trusted source like a reverse proxy. It's the last place you want to create compatibility concerns.

sideways birds

If you think about it, this means you can never be sure about any Animal either: how can you know there isn't an essential piece of hidden metadata traveling along that you are ignoring, which changes the meaning significantly?

Consider what happened when mobile phones started producing JPEGs with portrait vs landscape orientation metadata. Pictures were randomly sideways and upside down. You couldn't even rely on something as basic as the image width actually being, you know, the width. How many devs would anticipate that?

The only reason this wasn't a bigger problem is because for 99% of use-cases, you can just apply the rotation once upfront and then forget about it. That is, you can make a function RotatableImage => Image aka a Duck => Animal. This is an S => T that doesn't lose any information anyone cares about. This is the rare exception, only done occasionally, as a treat.

If you instead need to upgrade a whole image and display pipeline to support, say, high-dynamic range or P3 color, that's a different matter entirely. It will never truly be 100% done everywhere, we all know that. But should it be? It's another ChickenDuck scenario, because now some code wants images to stay simple, 8-bit and sRGB, while other code wants something else. Are you going to force each side to deal with the existence of the other, in every situation? Or will you keep the simplest case simple?

A plain old 2D array of pixels is not sufficient for T in the general case, but it is too useful on its own to simply throw it out. So you shouldn't make an AbstractImage which specializes into a SimpleImage and an HDRImage and a P3Image, because that means your SimpleImage isn't simple anymore. You should instead make an ImageView with metadata, which still contains a plain Image with only raw pixels. That is, a SimpleImage is just an ImageView<Image, NoColorProfile>. That way, there is still a regular Image on the inside. Code that provides or needs an Image does not need to change.

It's important to realize these are things you can only figure out if you have a solid idea of how people actually work with images in practice. Like knowing that we can just all agree to "bake in" a rotation instead of rewriting a lot of code. Architecting from the inside is not sufficient, you must position yourself as an external user of what you build, someone who also has a full-time job.

If you want a piece of software to be extensible, that means the software will become somebody else's development dependency. This puts huge constraints on its design and how it can change. You might say there is no such thing as an extensible schema, only an unfinished schema, because every new version is really a fork. But this doesn't quite capture it, and it's not quite so absolute in practice.

Colonel Problem

Interoperability is easy in pairs. You can model this as an Input<T> "A" connecting to an Output<T> "B". This does not need to cover every possible T, it can be a reduced subset R of T. For example, two apps exchange grayscale images (R) as color PNGs (T). Every R is also a T, but not every T is an R. This means:

  • Any function () => grayscaleImage is a valid substitute for () => colorImage.
  • Any function (colorImage) => void is a valid substitute for (grayscaleImage) => void.

This helps A, which is an => R pretending to be a => T. But B still needs to be an actual T =>, even if it only wants to be an R =>. Turning an R => into a T => is doable as long as you have a way to identify the R parts of any T, and ignore the rest. If you know your images are grayscale, just use any of the RGB channels. Therefore, working with R by way of T is easy if both sides are in on it. If only one side is in on it, it's either scraping or SEO.

But neither applies to arbitrary processing blocks T => T that need to mutually interoperate. If A throws away some of the data it received before sending it to B, and then B throws away other parts before sending it back to A, little will be left. For reliable operation, either A → B → A or B → A → B ought to be a clean round-trip. Ideally, both. Just try to tell a user you preserve <100% of their data every time they do something.

photoshop layer panel

Consider interoperating with e.g. Adobe Photoshop. A Photoshop file isn't just an image, it's a collection of image layers, vector shapes, external resources and filters. These are combined into a layer tree, which specifies how the graphics ought to be combined. This can involve arbitrary nesting, with each layer having unique blend modes and overlaid effects. Photoshop's core here acts like a kernel in the OS sense, providing a base data model and surrounding services. It's responsible for maintaining the mixed raster/vector workspace of the layered image. The associated "user space" is the drawing tools and inspectors.

Being mutually compatible with Photoshop means being a PSD => PSD back-end, which is equivalent to re-implementing all the "kernel space" concepts. Changing a single parameter or pixel requires re-composing the entire layer stack, so you must build a kernel or engine that can do all the same things.

Also, let's be honest here. The average contemporary dev eyes legacy desktop software somewhat with suspicion. Sure, it's old and creaky, and their toolbars are incredibly out of fashion. But they get the job done, and come with decades of accumulated, deep customizability. The entrenched competition is stiff.

This reflects what I call the Kernel Problem. If you have a processing kernel revolving around an arbitrary T => T block, then the input T must be more like a program than data. It's not just data and metadata, it's also instructions. This means there is only one correct way to interpret them, aside from differences in fidelity or performance. If you have two such kernels which are fully interoperable in either direction, then they must share the same logic on the inside, at least up to equivalence.

If you are trying to match an existing kernel T => T's features in your S => S, your S must be at least as expressive as their original T. To do more, every T must also be a valid S. You must be the Animal to their Duck, not a Duck to their Animal, which makes this sort of like reverse inheritance: you adopt all their code but can then only add non-conflicting changes, so as to still allow for real substitution. A concrete illustration is what "Linux Subsystem for Windows" actually means in practice: put a kernel inside the kernel, or reimplement it 1-to-1. It's also how browsers evolved over time, by adding, not subtracting.

Therefor, I would argue an "extensible kernel" is in the broad sense an oxymoron, like an "extensible foundation" of a building. The foundation is the part that is supposed to support everything else. Its purpose is to enable vertical extension, not horizontal.

If you expand a foundation without building anything on it, it's generally considered a waste of space. If you try to change a foundation underneath people, they rightly get upset. The work isn't done until the building actually stands. If you keep adding on to the same mega-building, maintenance and renewal become impossible. The proper solution for that is called a town or a city.

Naturally kernels can have plug-ins too, so you can wonder if that's actually a "motte user-land" or not. What's important is to notice the dual function. A kernel should enable and support things, by sticking to the essentials and being solid. At the same time, it needs to also ship with a useful toolset working with a single type T that behaves extensibly: it must support arbitrary programs with access to processes, devices, etc.

If extensibility + time = kitchen sink bloat, how do you counter entropy?

motte and bailey

You must anticipate, by designing even your core T itself to be decomposable and hence opt-in à la carte. A true extensible kernel is therefor really a decomposable kernel, or perhaps a kernel generator, which in the limit becomes a microkernel. This applies whether you are talking about Photoshop or Linux. You must build it so that it revolves around an A & B & C & ..., so that both A => A and B => B can work directly on an ABC and implicitly preserve the ABC-ness of the result. If all you need to care about is A or B, you can use them directly in a reduced version of the system. If you use an AB, only its pertinent aspects should be present.

Entity-Component Systems are a common way to do this. But they too have a kernel problem: opting in to a component means adopting a particular system that operates on that type of component. Such systems also have dependency chains, which have to be set up in the right order for the whole to behave right. It is not really A & B but A<B> or B<A> in practice. So in order for two different implementations of A or B to be mutually compatible, they again have to be equivalent. Otherwise you can't replace a Process<T> without replacing all the associated input, or getting unusably different output.

The main effect of à la carte architecture is that it never seems like a good idea to force anyone else to turn their Duck into a Chicken, by adopting all your components. You should instead try to agree on a shared Animal<T>. Any ChickenDuck that you do invent will have a limited action radius. Because other people can decide for themselves whether they truly need to deal with chickens on their own time.

None of this is new, I'm just recapturing old wisdom. It frankly seems weird to use programming terminology to have described this problem, when the one place it is not a big deal is inside a single, comfy programming environment. We do in fact freely import modules à la carte when we code, because our type T is the single environment of our language run-time.

But it's not so rosy. The cautionary tale of Python 2 vs 3: if you mess with the internals and standard lib, it's a different language, no matter how you do it. You still have a kernel everyone depends on and it can take over a decade to migrate a software ecosystem.

Everyone has also experienced the limits of modularity, in the form of overly wrapped APIs and libraries, which add more problems than they solve. In practice, everyone on a team must still agree on one master, built incrementally, where all the types and behavior is negotiated and agreed upon. This is either a formal spec, or a defacto one. If it is refactored, that's just a fork everyone agrees to run with. Again, it's not so much extensible, just perpetually unfinished.

À la carte architecture is clearly necessary but not sufficient on its own. Because there is one more thing that people tend to overlook when designing a schema for data: how a normal person will actually edit the data inside.

birds

Oil and Water

Engineering trumps theoretical models, hence the description of PSD above actually omits one point deliberately.

It turns out, if all you want to do is display a PSD, you don't need to reimplement Photoshop's semantics. Each .PSD contains a pre-baked version of the image, so that you don't need to interpret it. A .PSD is really two file formats in one, a layered PSD and something like a raw PNG. It is not a ChickenDuck but a DuckAnimal. They planned ahead so that the Photoshop format can still work if all you want to be is MSPaint => MSPaint. For example, if you're a printer.

This might lead you to wonder.

Given that PNG is itself extensible, you can imagine a PNG-PSD that does the same thing as a PSD. It contains an ordinary image, with all the Photoshop specific data embedded in a separate PSD section. Wouldn't that be better? Now any app that can read PNG can read PSD, and can preserve the PSD-ness. Except, no. If anyone blindly edits the PNG part of the PNG-PSD, while preserving the PSD data, they produce a file where both are out of sync. What you see now depends on which app reads it. PNG-PSDs would be landmines in a mixed ecosystem.

It's unavoidable: if some of the data in a schema is derived from other data in it, the whole cannot be correctly edited by a "dumb", domain-agnostic editor, because of the Kernel Problem. This is why "single source of truth" should always be the end-goal.

A fully extensible format is mainly just kicking the can down the road, saving all the problems for later. It suggests a bit of a self-serving lie: "Extensibility is for other people." It is a successful business recipe, but a poor engineering strategy. It results in small plug-ins, which are not first class, and not fundamentally changing any baked in assumptions.

But the question isn't whether plug-ins are good or bad. The question is whether you actually want to lock your users of tomorrow into how your code works today. You really don't, not unless you've got something battle-hardened already.

birds

If you do see an extensible system working in the wild on the Input, Process and Output side, that means it's got at least one defacto standard driving it. Either different Inputs and Outputs have agreed to convert to and from the same intermediate language... or different middle blocks have agreed to harmonize data and instructions the same way.

This must either flatten over format-specific nuances, or be continually forked to support every new concept being used. Likely this is a body of practices that has mostly grown organically around the task at hand. Given enough time, you can draw a boundary around a "kernel" and a "user land" anywhere. To make this easier, a run-time can help do auto-conversion between versions or types. But somebody still has to be doing it.

This describes exactly what happened with web browsers. They cloned each other's new features, while web developers added workarounds for the missing pieces. Not to make it work differently, but to keep it all working exactly the same. Eventually people got fed up and just adopted a React-like.

That is, you never really apply extensibility on all three fronts at the same time. It doesn't make sense: arbitrary code can't work usefully on arbitrary data. The input and output need to have some guarantees about the process, or vice versa.

Putting data inside a free-form key/value map doesn't change things much. It's barely an improvement over having a unknownData byte[] mix-in on each native type. It only pays off if you actually adopt a decomposable model and stick with it. That way the data is not unknown, but always provides a serviceable view on its own. Arguably this is the killer feature of a dynamic language. The benefit of "extensible data" is mainly "fully introspectable without recompilation."

You need a well-defined single type T that sets the ground rules for both data and code, which means T must be a common language. It must be able to work equally well as both an A, B and C, which are needs that must have been anticipated. Yet they should be built such that you can just use a D of your own, without inconvenient dependency. The key quality to aim for is not creativity but discipline.

If you can truly substitute a type with something else everywhere, it can't be arbitrarily extended or altered, it must retain the exact same interface. In the real world, that means it must actually do the same thing, only marginally better or in a different context. A tool like ffmpeg only exists because we invented a bajillion different ways to encode the same video, and the only solution is to make one thing that supports everything. It's the Unicode of video.

If you extend something into a new type, it's not actually a substitute, it's a fork trying to displace the old standard. As soon as it's used, it creates a data set that follows a new standard. Even when you build your own parsers and/or serializers, you are inventing a spec of your own. Somebody else can do the same thing to you, and that somebody might just be you 6 months from now. Being a programmer means being an archivist-general for the data your code generates.

* * *

If you actually think about it, extensibility and substitution are opposites in the design space. You must not extend, you must decompose, if you wish to retain the option of substituting it with something simpler yet equivalent for your needs. Because the other direction is one-way only, only ever adding complexity, which can only be manually refactored out again.

If someone is trying to sell you on something "extensible," look closely. Is it actually à la carte? Does it come with a reasonable set of proven practices on how to use it? If not, they are selling you a fairy tale, and possibly themselves too. They haven't actually made it reusable yet: if two different people started using it to solve the same new problem, they would not end up with compatible solutions. You will have 4 standards: the original, the two new variants, and the attempt to resolve all 3.

Usually it is circumstance, hierarchy and timing that decides who adapts their data and their code to whom, instead of careful consideration and iteration. Conway's law reigns, and most software is shaped like the communication structure of the people who built it. "Patch," "minor" or "major" release is just the difference between "Pretty please?", "Right?" and "I wasn't asking."

We can do a lot better. But the mindset it requires at this point is not extensibility. The job at hand is salvage.

August 27, 2021

IT architects try to use views and viewpoints to convey the target architecture to the various stakeholders. Each stakeholder has their own interests in the architecture and wants to see their requirements fulfilled. A core role of the architect is to understand these requirements and make sure the requirements are met, and to balance all the different requirements.

Architecture languages or meta-models often put significant focus on these views. Archimate has a large annex on Example Viewpoints just for this purpose. However, unless the organization is widely accustomed to enterprise architecture views, it is unlikely that the views themselves are the final product: being able to translate those views into pretty slides and presentations is still an important task for architects when they need to present their findings to non-architecture roles.

August 23, 2021

In the 15 years that I've been blogging, it's never been this quiet on my blog.

Blogging is an important part of me. It's how I crystallize ideas, reflect, and engage with thousands of people around the world. Blogging encourages me to do research; it improves my understanding of different topics. Blogging sometimes forces me to take sides; it helps me find myself.

I miss blogging. Unfortunately, I've lost my blogging routine.

At first, COVID-19 was to blame for that. I'd write many of my blog posts on the train to work. My train ride was one hour each way and that gave me plenty of time to write. Once in the office, there is zero time for blogging. COVID-19 interrupted my blogging routine and took away my protected writing time.

Then earlier this year, we moved from the suburbs of Boston to the city. Renovating our new condo, selling our old condo, and moving homes consumed much of my personal time — blogging time included. And now we live in the city, I no longer commute by train.

Admittedly, I've also felt blocked. I've been waiting to blog until I had something interesting to say, but nothing seemed interesting enough.

So I'm eager to find a new blogging routine. I'm a fan of routines. Routines add productivity and consistency to my life. Without a good blogging routine, I'm worried about the future of this blog.

To get back into a blogging routine, I made two decisions: (1) to target one blog post per week and (2) to balance my inner critic. I will no longer wait for something interesting to come along (as this blog post illustrates).

When you break out of any habit, it can be hard to get back into it. To get back into a routine, it's better to write something regularly than to write nothing at all. These seem achievable goals and I'm hopeful they get me blogging more frequently again.

August 20, 2021

published the following diary on isc.sans.edu: “Waiting for the C2 to Show Up“:

Keep this in mind: “Patience is key”. Sometimes when you are working on a malware sample, you depend on online resources. I’m working on a classic case: a Powershell script decodes then injects a shellcode into a process. There are plenty of tools that help you to have a good idea of a shellcode behavior (like scdbg)… [Read more]

The post [SANS ISC] Waiting for the C2 to Show Up appeared first on /dev/random.

August 19, 2021

Dear IoT manufacturers,

Yes, I admit: I like your products and my Geekness does not help! I like to play with them. If some are “gadgets” that finally land in a drawer amongst others with cables and connectors, some of them are really useful and I use them daily. You can probably imagine that, when I receive a new device, it’s not connected “in the wild” just after unboxing it. Wait… You do?

First, I read the provided documentation (you know the famous “RTFM”), I google for some forum or blog articles to see if other people already played with it. Finally, it is connected to a dedicated network without a full Internet connection (Read: “most of the egress traffic being blocked”). And then, the first problems arise… The initial setup fails, I need to restore to factory settings and try again. Basically, the process is:

Connect > Setup > Fail > Check firewall logs > Open ports > Reset > Try again > Drink more coffee

This discovery process is so annoying! Why your customers should sometimes perform something like “network reverse engineering” to understand how your product is working?

I’ve a message for you:

Please be transparent!

Why don’t you provide all the documentation for a safe setup? By example:

  • Required TCP/UDP port(s)
  • Hardcoded IP addresses/hostnames (yes, not everybody is using 8.8.8.8 as a DNS resolver or pool.ntp.org as NTP server!)
  • Specific services (VPN) and their purpose(s)
  • Avoid obscure protocols and stick to standard ones
  • Sometimes, even the device MAC address is not written on the box. To get the MAC you need to boot it a first timeç

Also avoid advices like:

If it does not work, disable the security feature <x>!

Why don’t you allow some critical settings to be customised like DNS, NTP, etc… or use proxy?

Why don’t you explain why this suspicious VPN connection is required? What are the data exchanged? What do you do with them (Ok, I’m dreaming)

I see you coming… You are right on one point: Such very technical pieces of information can’t be understood by many of your customers and your business model requires addressing the largest customers base. I don’t ask you to include this information in the documentation provided with the device but why not keep technical specs online for those that need to review them? The right balance must be found between usability and security and it’s up to me, your customer, to decide where to slide the cursor. If I disable this option, a sexy feature won’t be available? ok, fair enough but it’s my choice.

No, your FAQ does not contain relevant information for me. “Have you tried turning it off and on again” is not the right answer!

Thank you!

The post Public Message to IoT Manufacturers appeared first on /dev/random.

August 16, 2021

August 08, 2021

pi cluster

Last year I got a raspberry pi 4 to play with and installed Manjaro on it.

The main reason I went with Manjaro was that the ArchLinux Arm image/tgz for the Raspberry Pi 4 was still 32 bits, or you needed to create-your-own kernel.

But started to like Manjaro Linux, it provided a stable base with regular updates. This year I upgraded my setup with 2 additional Raspberry Pi 4 to provide clustering for my k3s (Kubernetes) setup. I used virtual machines on the Raspberry Pi to host the k3s nodes. Also because want to the Pi for other tasks and virtual machines makes it easier to split the resources. It’s also an “abstraction layer” if you want to combine the cluster with other ARM64 systems in the future.

I always (try to) to full disk encryption, when you have multiple nodes it’s important to be able to unlock the encryption remotely.

Install Manjaro on an encrypted filesystem

Manjaro will run an install script after the RPI is booted to complete the installion.

  • We have two options boot the pi from the standard non-encrypted image and extract/move it to an encrypted filesystem.
  • Extract the installation image and move the content to an encrypted filesystem.

You’ll find my journey of the second option below. The setup is mainly the same as I did last year, but with support to unlock the encryption with ssh.

The host system to extract/install the image is an x86_64 system running Archlinux.

Download and copy

Download and verify the Manjaro image from: https://www.manjaro.org/download/#raspberry-pi-4.

Copy the image to keep the original intact.

[root@vicky manjaro]# cp Manjaro-ARM-xfce-rpi4-20.06.img image

Create tarball

Verify the image

Verify the image layout with fdisk -l.

[root@vicky manjaro]# fdisk -l image
Disk image: 4.69 GiB, 5017436160 bytes, 9799680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x090a113e

Device     Boot  Start     End Sectors   Size Id Type
image1           62500  500000  437501 213.6M  c W95 FAT32 (LBA)
image2          500001 9799679 9299679   4.4G 83 Linux
[root@vicky manjaro]# 

We’ll use kpartx to map the partitions in the image so we can mount them. kpartx is part of the multipath-tools.

Map the partitions in the image with kpartx -ax, the “-a” option add the image, “-v” makes it verbose so we can see where the partitions are mapped to.

[root@vicky manjaro]# kpartx -av image
add map loop1p1 (254:10): 0 437501 linear 7:1 62500
add map loop1p2 (254:11): 0 9299679 linear 7:1 500001
[root@vicky manjaro]#

Create the destination directory.

[root@vicky manjaro]# mkdir /mnt/chroot

Mount the partitions.

[root@vicky manjaro]# mount /dev/mapper/loop1p2 /mnt/chroot
[root@vicky manjaro]# mount /dev/mapper/loop1p1 /mnt/chroot/boot
[root@vicky manjaro]#

Create the tarball.

[root@vicky manjaro]# cd /mnt/chroot/
[root@vicky chroot]# tar czvpf /home/staf/Downloads/isos/manjaro/Manjaro-ARM-xfce-rpi4-21.07.tgz .

Umount.

[root@vicky ~]# umount /mnt/chroot/boot 
[root@vicky ~]# umount /mnt/chroot
[root@vicky ~]# cd /home/staf/Downloads/isos/manjaro/
[root@vicky manjaro]# kpartx -d image
loop deleted : /dev/loop1
[root@vicky manjaro]# 

Partition and create filesystems

Partition

Partition your harddisk delete all partitions if there are partition on the harddisk.

I’ll create 3 partitions on my harddisk

  • boot partitions of 500MB (Type c ‘W95 FAT32 (LBA)’
  • root partitions of 50G
  • rest
[root@vicky ~]# fdisk /dev/sdh

Welcome to fdisk (util-linux 2.35.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x49887ce7.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-976773167, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-976773167, default 976773167): +500M

Created a new partition 1 of type 'Linux' and of size 500 MiB.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2): 2
First sector (1026048-976773167, default 1026048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (1026048-976773167, default 976773167): +50G

Created a new partition 2 of type 'Linux' and of size 50 GiB.

Command (m for help): n
Partition type
   p   primary (2 primary, 0 extended, 2 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (3,4, default 3): 
First sector (105883648-976773167, default 105883648): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (105883648-976773167, default 976773167): 

Created a new partition 3 of type 'Linux' and of size 415.3 GiB.

Command (m for help): t
Partition number (1-3, default 3): 1
Hex code (type L to list all codes): c

Changed type of partition 'Linux' to 'W95 FAT32 (LBA)'.

w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
Command (m for help):  

Create the boot file system

The raspberry pi uses a FAT filesystem for the boot partition.

[root@vicky tmp]# mkfs.vfat /dev/sdh1
mkfs.fat 4.2 (2021-01-31)
[root@vicky tmp]# 

Create the root filesystem

Overwrite the root partition with random data

[root@vicky tmp]# dd if=/dev/urandom of=/dev/sdg2 bs=4096 status=progress
53644914688 bytes (54 GB, 50 GiB) copied, 682 s, 78.7 MB/s 
dd: error writing '/dev/sdg2': No space left on device
13107201+0 records in
13107200+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 687.409 s, 78.1 MB/s
[root@vicky tmp]# 

Encrypt the root filesystem

Benchmark

I booted the RPI4 from a sdcard to verify the encryption speed by executing the cryptsetup benchmark.

[root@minerva ~]# cryptsetup benchmark
# Tests are approximate using memory only (no storage IO).
PBKDF2-sha1       398395 iterations per second for 256-bit key
PBKDF2-sha256     641723 iterations per second for 256-bit key
PBKDF2-sha512     501231 iterations per second for 256-bit key
PBKDF2-ripemd160  330156 iterations per second for 256-bit key
PBKDF2-whirlpool  124356 iterations per second for 256-bit key
argon2i       4 iterations, 319214 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
argon2id      4 iterations, 321984 memory, 4 parallel threads (CPUs) for 256-bit key (requested 2000 ms time)
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b        23.8 MiB/s        77.7 MiB/s
    serpent-cbc        128b               N/A               N/A
    twofish-cbc        128b        55.8 MiB/s        56.2 MiB/s
        aes-cbc        256b        17.4 MiB/s        58.9 MiB/s
    serpent-cbc        256b               N/A               N/A
    twofish-cbc        256b        55.8 MiB/s        56.1 MiB/s
        aes-xts        256b        85.0 MiB/s        74.9 MiB/s
    serpent-xts        256b               N/A               N/A
    twofish-xts        256b        61.1 MiB/s        60.4 MiB/s
        aes-xts        512b        65.4 MiB/s        57.4 MiB/s
    serpent-xts        512b               N/A               N/A
    twofish-xts        512b        61.3 MiB/s        60.3 MiB/s
[root@minerva ~]# 
Create the Luks volume

The aes-xts cipher seems to have the best performance on the RPI4.

[root@vicky ~]# cryptsetup luksFormat --cipher aes-xts-plain64 --key-size 256 --hash sha256 --use-random /dev/sdh2

WARNING!
========
This will overwrite data on /dev/sdh2 irrevocably.

Are you sure? (Type 'yes' in capital letters): YES
Enter passphrase for /dev/sdh2: 
Verify passphrase: 
WARNING: Locking directory /run/cryptsetup is missing!
[root@vicky ~]# 
Open the Luks volume
[root@vicky ~]# cryptsetup luksOpen /dev/sdh2 cryptroot
Enter passphrase for /dev/sdh2: 
[root@vicky ~]# 

Create the root filesystem

[root@vicky tmp]# mkfs.ext4 /dev/mapper/cryptroot
mke2fs 1.46.3 (27-Jul-2021)
Creating filesystem with 13103104 4k blocks and 3276800 inodes
Filesystem UUID: 65c19eb5-d650-4d8a-8335-ef792604009d
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (65536 blocks): done
Writing superblocks and filesystem accounting information: done   

[root@vicky tmp]# 

Mount and extract

Mount the root filesystem.

[root@vicky ~]# mount /dev/mapper/cryptroot /mnt/chroot
[root@vicky ~]# mkdir -p /mnt/chroot/boot
[root@vicky ~]# mount /dev/sdh1 /mnt/chroot/boot
[root@vicky ~]# 

And extract the tarball.

[root@vicky manjaro]# cd /home/staf/Downloads/isos/manjaro/
[root@vicky manjaro]# tar xpzvf Manjaro-ARM-xfce-rpi4-21.07.tgz -C /mnt/chroot/
[root@vicky manjaro]# sync

chroot

To continue the setup we need to boot or chroot into the operating system. It possible to run ARM64 code on a x86_64 system with qemu - qemu will emulate an arm64 CPU -.

Install qemu-arm-static

Install the qemu-arm package. It not in the main Archlinux distribution but it’s available as a AUR.

[staf@vicky ~]$ yay -S qemu-arm-static 

copy qemu-arm-static

Copy the qemu-arm-static into the chroot.

[root@vicky manjaro]# cp /usr/bin/qemu-arm-static /mnt/chroot/usr/bin/
[root@vicky manjaro]# 

mount proc & co

To be able to run programs in the chroot we need the proc, sys and dev filesystems mapped into the chroot.

For dns resolution to work you need to mount /run into the chroot, and start the systemd-resolved.service.

[root@vicky ~]# mount -t proc none /mnt/chroot/proc
[root@vicky ~]# mount -t sysfs none /mnt/chroot/sys
[root@vicky ~]# mount -o bind /dev /mnt/chroot/dev
[root@vicky ~]# mount -o bind /dev/pts /mnt/chroot/dev/pts
[root@vicky ~]# mount -o bind /run /mnt/chroot/run/
[root@vicky ~]# 

dns

Start the systemd-resolved.service on you host system. This is required to have dns available in your chroot during the installation.

Alternativaly you can use a proxy during the installation.

[root@vicky ~]# systemctl start systemd-resolved.service 

chroot

Chroot into ARM64 installation.

LANG=C chroot /mnt/chroot/

Set the PATH.

[root@vicky /]# export PATH=/sbin:/bin:/usr/sbin:/usr/bin

And verify that we are running aarch64.

[root@vicky /]# uname -a 
Linux vicky 5.12.19-hardened1-1-hardened #1 SMP PREEMPT Tue, 20 Jul 2021 17:48:41 +0000 aarch64 GNU/Linux
[root@vicky /]# 

Update and install vi

Update the public keyring

[root@vicky /]# pacman-key --init
gpg: /etc/pacman.d/gnupg/trustdb.gpg: trustdb created
gpg: no ultimately trusted keys found
gpg: starting migration from earlier GnuPG versions
gpg: porting secret keys from '/etc/pacman.d/gnupg/secring.gpg' to gpg-agent
gpg: migration succeeded
==> Generating pacman master key. This may take some time.
gpg: Generating pacman keyring master key...
gpg: key DC12547C06A24CDD marked as ultimately trusted
gpg: directory '/etc/pacman.d/gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/etc/pacman.d/gnupg/openpgp-revocs.d/55620B2ED4DE18F5923A2451DC12547C06A24CDD.rev'
gpg: Done
==> Updating trust database...
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
[root@vicky /]# 
[root@vicky /]# pacman-key --refresh-keys
[root@vicky /]#
[root@vicky etc]# pacman -Qs keyring
local/archlinux-keyring 20210616-1
    Arch Linux PGP keyring
local/archlinuxarm-keyring 20140119-1
    Arch Linux ARM PGP keyring
local/manjaro-arm-keyring 20200210-1
    Manjaro-Arm PGP keyring
local/manjaro-keyring 20201216-1
    Manjaro PGP keyring
[root@vicky etc]# 
[root@vicky etc]# pacman-key --populate archlinux manjaro archlinuxarm manjaro-arm

Update all packages to the latest version.

[root@vicky etc]# pacman -Syyu
:: Synchronizing package databases...
 core                                   237.4 KiB   485 KiB/s 00:00 [#####################################] 100%
 extra                                    2.4 MiB  2.67 MiB/s 00:01 [#####################################] 100%
 community                                6.0 MiB  2.74 MiB/s 00:02 [#####################################] 100%
:: Some packages should be upgraded first...
resolving dependencies...
looking for conflicting packages...

Packages (4) archlinux-keyring-20210802-1  manjaro-arm-keyring-20210731-1  manjaro-keyring-20210622-1
             manjaro-system-20210716-1

Total Installed Size:  1.52 MiB
Net Upgrade Size:      0.01 MiB

:: Proceed with installation? [Y/n] 

We need an editor.

root@vicky /]# pacman -S vi
resolving dependencies...
looking for conflicting packages...

Packages (1) vi-1:070224-4

Total Download Size:   0.15 MiB
Total Installed Size:  0.37 MiB

:: Proceed with installation? [Y/n] y
:: Retrieving packages...
 vi-1:070224-4-aarch64                         157.4 KiB  2.56 MiB/s 00:00 [##########################################] 100%
(1/1) checking keys in keyring                                             [##########################################] 100%
(1/1) checking package integrity                                           [##########################################] 100%
(1/1) loading package files                                                [##########################################] 100%
(1/1) checking for file conflicts                                          [##########################################] 100%
(1/1) checking available disk space                                        [##########################################] 100%
:: Processing package changes...
(1/1) installing vi                                                        [##########################################] 100%
Optional dependencies for vi
    s-nail: used by the preserve command for notification
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
[root@vicky /]# 

Unlock the encryption remotely

When you have multiple systems it’s handy to be able to unlock the encryption remotely.

Install the required mkinitcpio packages

[root@vicky /]# pacman -S mkinitcpio-utils mkinitcpio-netconf mkinitcpio-dropbear

mkinitcpio

HOOKS

Add netconf dropbear encryptssh to HOOKS before filesystems in /etc/mkinitcpio.conf.

Don’t include the encrypt as this will cause the boot image to try the unlock the encryption twice and will your system will fail to boot.

[root@vicky /]#  vi /etc/mkinitcpio.conf
HOOKS=(base udev plymouth autodetect modconf block netconf dropbear encryptssh filesystems keyboard fsck)

Create the boot image

[root@vicky /]# ls -l /etc/mkinitcpio.d/
total 4
-rw-r--r-- 1 root root 246 Jun 11 11:06 linux-rpi4.preset
[root@vicky /]# 
[root@vicky /]# mkinitcpio -p linux-rpi4
==> Building image from preset: /etc/mkinitcpio.d/linux-rpi4.preset: 'default'
  -> -k 5.10.52-1-MANJARO-ARM -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.10.52-1-MANJARO-ARM
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [plymouth]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [netconf]
  -> Running build hook: [dropbear]
There is no root key in /etc/dropbear/root_key existent; exit
  -> Running build hook: [encryptssh]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
[root@vicky /]# 

Don’t include the encrypt hooks, as this will cause the boot image to try the unlock the encryption twice and will your system will fail to boot. We need to have ssh host created to continue with the configuration. We’ll continue with the unlock encryption configuration after the first boot of the Raspberry Pi.

update boot settings…

Get the UUID for the boot and the root partition.

[root@vicky boot]# ls -l /dev/disk/by-uuid/ | grep -i sdh
lrwxrwxrwx 1 root root 12 Jul  8 11:42 xxxx-xxxx -> ../../sdh1
lrwxrwxrwx 1 root root 12 Jul  8 12:44 xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -> ../../sdh2
[root@vicky boot]# 

The Raspberry PI uses cmdline.txt to specify the boot options.

[root@vicky ~]# cd /boot
[root@vicky boot]# 
[root@vicky boot]# cp cmdline.txt cmdline.txt_org
[root@vicky boot]# 

Remove splash and add plymouth.enable=0. Set console=tty1.

cryptdevice=/dev/disk/by-uuid/43c2d714-9af6-4d60-91d2-49d61e93bf3e:cryptroot root=/dev/mapper/cryptroot rw rootwait console=s
erial0,115200 console=tty1 selinux=0 plymouth.enable=0 quiet plymouth.ignore-serial-consoles smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 
kgdboc=serial0,115200 elevator=noop usbhid.mousepoll=8 snd-bcm2835.enable_compat_alsa=0 audit=0

fstab

[root@vicky etc]# cp fstab fstab_org
[root@vicky etc]# vi fstab
[root@vicky etc]# 
# Static information about the filesystems.
# See fstab(5) for details.

# <file system> <dir> <type> <options> <dump> <pass>
UUID=xxxx-xxxx  /boot   vfat    defaults        0       0

Finish your setup

Set the root password.

[root@vicky etc]# passwd

Set the timezone.

[root@vicky etc]# ln -s /usr/share/zoneinfo/Europe/Brussels /etc/localtime

Generate the required locales.

[root@vicky etc]# vi /etc/locale.gen 
[root@vicky etc]# locale-gen

Set the hostname.

[root@vicky etc]# vi /etc/hostname

clean up

Exit chroot

[root@vicky etc]# 
exit
[root@vicky ~]# uname -a
Linux vicky 5.12.19-hardened1-1-hardened #1 SMP PREEMPT Tue, 20 Jul 2021 17:48:41 +0000 x86_64 GNU/Linux
[root@vicky ~]# 

Make sure that there are no processes still running from the chroot.

[root@vicky ~]# ps aux | grep -i qemu
root       29568  0.0  0.0 151256 15900 pts/4    Sl   08:42   0:00 /usr/bin/qemu-aarch64-static /bin/bash -i
root       46057  0.1  0.0 152940 13544 pts/4    Sl+  08:50   0:05 /usr/bin/qemu-aarch64-static /usr/bin/ping www.google.be
root      151414  0.0  0.0   7072  2336 pts/5    S+   09:52   0:00 grep -i qemu

Kill the processes from the chroot.

root@vicky ~]# kill 29568 46057
[root@vicky ~]# 

Umount the chroot filesystems.

[root@vicky ~]# umount -R /mnt/chroot
[root@vicky ~]# 

Close the luks volume…

[root@vicky ~]# cryptsetup luksClose cryptroot
[root@vicky ~]# sync
[root@vicky ~]# 

Boot

Connect the usb disk to the raspberry pi and power it on. If you are lucky the PI will boot from the USB device and ask you to type the password to decrypt the root filesystem.

Configure unlock with ssh

Configure dropbear

Set the root key

Copy your public ssh key to /etc/dropbear/root_key

[root@staf-pi002 dropbear]# cat > root_key
[root@staf-pi002 dropbear]# chmod 600 root_key 
[root@staf-pi002 dropbear]# 

Convert the ssh rsa host key to pem

Dropbear can only handle the PEM format. We’ll need to convert the host key to the PEM format.

[root@staf-pi002 dropbear]# ssh-keygen -m PEM -p -f /etc/ssh/ssh_host_rsa_key
Key has comment 'root@stafwag-pi002'
Enter new passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved with the new passphrase.
[root@staf-pi002 dropbear]# 

Recreate the boot image.

[root@staf-pi002 dropbear]# mkinitcpio -p linux-rpi4
==> Building image from preset: /etc/mkinitcpio.d/linux-rpi4.preset: 'default'
  -> -k 5.10.52-1-MANJARO-ARM -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 5.10.52-1-MANJARO-ARM
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [plymouth]
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [netconf]
  -> Running build hook: [dropbear]
Key is a ssh-rsa key
Wrote key to '/etc/dropbear/dropbear_rsa_host_key'
Error: Error parsing OpenSSH key
Error reading key from '/etc/ssh/ssh_host_dsa_key'
Error: Unsupported OpenSSH key type
Error reading key from '/etc/ssh/ssh_host_ecdsa_key'
dropbear_rsa_host_key : sha1!! 2a:a4:d5:a0:00:ce:1e:9f:88:84:72:f2:03:ce:ac:4a:27:11:da:09
  -> Running build hook: [encryptssh]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux.img
==> Image generation successful
[root@staf-pi002 dropbear]# 

Configure a static ip address

Update /boot/cmdline with your ip configuration.

cryptdevice=/dev/disk/by-uuid/43c2d714-9af6-4d60-91d2-49d61e93bf3e:cryptroot root=/dev/mapper /cryptroot rw rootwait console=serial0,115200 console=tty1 selinux=0 plymouth.enable=0 quiet plymouth.ignore-serial-consoles smsc95xx.turbo_mode=N dwc_otg.lpm_enable=0 kgdboc=serial0,115 200 elevator=noop usbhid.mousepoll=8 snd-bcm2835.enable_compat_alsa=0 audit=0 ip=xxx.xxx.xxx.xxx::yyy.yyy.yyy.254:255.255.255.0:staf-pi002:eth0:none

Reboot and test.

Have fun!

Links

August 06, 2021

I published the following diary on isc.sans.edu: “Malicious Microsoft Word Remains A Key Infection Vector“:

Despite Microsoft’s attempts to make its Office suite more secure and disable many automatic features, despite the fact that users are warned that suspicious documents should not be opened, malicious Word documents remain a key infection vector today. One of our readers (thanks Joel!) shared a sample that he received and, unfortunately, opened on his computer. The document was delivered to him via a spoofed email (sent by a known contact)… [Read more]

The post [SANS ISC] Malicious Microsoft Word Remains A Key Infection Vector appeared first on /dev/random.

August 04, 2021

Occasionally (about 4% of people contacting me) I get a job offer for somewhere in another country.

This is a list of places outside of Belgium where people are apparently interested in having me. 😀

  • India (Hyderabad)
  • Germany (Stuttgart, Wiesbaden)
  • United Kindom (London)
  • France (Paris)
  • Italy (Turin)
  • Spain (Madrid)
  • Netherlands (Amsterdam, Rotterdam, The Hague, Eindhoven, Almere, Arnhem, Deventer, Delft)
  • Sweden (Stockholm)
  • Austria (Graz)
  • Switzerland (Zurich)
  • Norway (Stavanger)
  • Luxembourg (Luxembourg City)

I have never considered moving permanently to another country for work, and I wouldn’t feel comfortable to move to a country where I don’t speak the language. Even if the company language is English, I would still need to communicate with people in everyday life, for example going to the shop. So from the list above, only France and the Netherlands would remain.

Besides the language, there is still the matter of being cut off from the people who matter to me. Yes there is the internet, and during the pandemic there was virtually no other way to stay in touch, but still… it’s not the same. I already have some friends in the Netherlands, so (hypothetically) I would feel less alone there. But there are still plenty of interesting local companies to work for, so no thanks for now.

Have you ever been invited to work abroad? If yes, what was your motivation for doing so? What were your experiences? Feel free to share in the comments!

The post Working abroad? appeared first on amedee.be.

August 03, 2021

After reading a few hundred emails from recruiters, I see a couple of trends popping up. I’m being contacted for job offers that really aren’t relevant or interesting for me. Some of them may be attributed to automatic keyword scanning. But still. If possible, I would kindly ask everyone not to contact me for any of the following:

  • Freelance: I have never done freelance before. Working freelance means that I would first have to start all the paperwork to become self-employed, and at this moment I’m not interested in doing all that. Maybe that could change in the faraway future, but at this point in my life I prefer permanent positions.
  • C/C++ embedded development: At one of my previous jobs, I did testing on the embedded software of a smart printer. Testing. Not development. I have never written a single line of C or C++ in my life. I would probably be able to read and understand other people’s code, but I’m sure that there are plenty of people who are really fluent in C/C++.
  • Drupal development: A long, long time ago, I made and maintained a few small Drupal sites. I have also been to one or two Drupal Dev Days in the early 2000s. I think I still have a T-shirt somewhere. But in all that time, I only did Drupal admin, I never went into the itty-gritty PHP to write custom Drupal code. And I’m pretty sure that my Drupal skills are quite rusty now.
  • Node.js development: Oh dear. I did a few tiny Node.js projects: some “glue code”, some rapid prototyping. Nothing fancy, nothing production quality, never more than 100 lines of code. Let’s not do that.
  • SharePoint development: With the eternal words of William Shakespeare:

Fie on’t! ah fie! ’tis an unweeded garden,
That grows to seed; things rank and gross in nature
Possess it merely. That it should come to this!

Hamlet, Act I, Scene ii

  • Quality Control Operator: This is typically a case of blindly searching for keywords and not verifying the results. I have worked as a Software Quality Engineer, so if you search only for “quality”, you’ll end up with jobs where you do actual physical inspection of physical products. Rule of thumb: if I can’t test it with an Assert-statement in some kind of programming language, then it’s probably not the kind of “quality” that I’m looking for.
  • Production / “blue collar jobs”: Yeah well let’s not do that at all, shall we? With all due respect for the people who do this type of work, and some of it is really essential work, but I don’t think that this would ever make me happy.
  • First line tech support: Been there, done that, got the battle scars. Never again, thank you very much.

Benefits for not contacting me for any of these: you don’t waste time chasing a dead-end lead, and I can spend more time on reading and reacting to job offers that actually are relevant, interesting and even exciting. Everybody happy! 🙂

The post Thanks, but no thanks appeared first on amedee.be.

July 30, 2021

The public cloud is a different beast than an on-premise environment, and that also reflects itself on how we (should) look at the processes that are actively steering infrastructure designs and architecture. One of these is the business continuity, severe incident handling, and the hopefully-never-to-occur disaster recovery. When building up procedures for handling disasters (DRP = Disaster Recovery Procedure or Disaster Recover Planning), it is important to keep in mind what these are about.

I published the following diary on isc.sans.edu: “Infected With a .reg File“:

Yesterday, I reported a piece of malware that uses archive.org to fetch its next stage. Today, I spotted another file that is also interesting: A Windows Registry file (with a “.reg” extension). Such files are text files created by exporting values from the Registry (export) but they can also and can also be used to add or change values in the Registry (import). Being text files, they don’t look suspicious… [Read more]

The post [SANS ISC] Infected With a .reg File appeared first on /dev/random.

July 29, 2021

I published the following diary on isc.sans.edu: “Malicious Content Delivered Through archive.org“:

archive.org, also known as the “way back machine” is a very popular Internet site that allows you to travel back in time and browse old versions of a website (like the ISC website). It works like regular search engines and continuously crawls the internet via bots. But there is another way to store content on archive.org: You may create an account and upload some content by yourself… [Read more]

The post [SANS ISC] Malicious Content Delivered Through archive.org appeared first on /dev/random.

July 27, 2021

Autoptimize 2.9 was released earlier today. It features:

  • New: per page/ post Autoptimize settings so one can disable specific optimizations (needs to be enabled on the main settings page under “Misc Options”).
  • New: “defer inline JS” as sub-option of “do not aggregate but defer” allowing to defer (almost) all JS
  • Improvement: Image optimization now automatically switches between AVIF & WebP & Jpeg even if lazyload is not active (AVIF has to be explicitly enabled).
  • Improvement: re-ordering of “JavaScript optimization” settings
  • Misc. other minor fixes, see the GitHub commit log

This release coincides with my father’s 76th birthday, who continues to be a big inspiration to me. He’s a mechanical engineer who after retirement focused his technical insights, experience and never-ending inquisitiveness on fountain pen design and prototyping, inventing a new bulkfiller mechanism in the process. Search the web for Fountainbel to find out more about him (or read this older blogpost I wrote in Dutch). Love you pops!

July 19, 2021

It’s been a long time since I last looked for a job myself. At job[-1] (7 years) and job[-2] (2 years), the employers contacted me while I was already working somewhere else, and at job[-3] I worked for 5 years, so all added up, that makes more than 14 years since I last did anything like this.

Job sites

I started with creating or updating a profile on a couple of job sites:

There are a couple more job sites that I know of but haven’t done anything with. Please leave a comment if you think any of them offer benefits over those listed above.

  • Viadeo (mostly French, so probably less useful)
  • Xing (I think they are mostly German-based)
  • StepStone
  • Facebook Job Search (I can’t imagine that any employer on Facebook Job Search wouldn’t also be on LinkedIn, but maybe I’ll try it to see if the search works better there)

I have also updated my CV and I’ve put it online: https://amedee.be/cv.

A torrent of messages

But then — I think — I made a mistake. The weather was nice, I wanted to be outdoors, trying to unwind a bit from the unusual times of the past months, and I disconnected.

Meanwhile the messages started pouring in, via email, LinkedIn (messages and connection requests), and occasionally a phone call from an unknown number. First just a few, then dozens, and just a few weeks later, already a couple of hundred. Oops.

The thing is, while I was technically available, I wasn’t yet mentally available. I still had to disconnect from the previous job, where I worked for more than 7 years, and I needed to think about what I really want to do next. Should I do something similar as before, because I already have the experience? Or should I try to find something that truly sparks joy? More on that later.

Strategies

Anyway, I had to come up with some strategies to deal with these high volumes of communication. First of all, not to get completely crazy, I defined a schedule, because otherwise I’d be responding to messages 24/7. There are other important activities too, like actively browsing through the job listings on various sites, or keeping up to date with current technology, or reaching out to my network, or having a social media presence (like this blog), or, you know, being social, having hobbies, and life in general.

One thing I noticed right away in many messages, is that people ask me for a CV — even though my LinkedIn profile is current. But I get it. And a separate document doesn’t confine me to the format of one specific website, and it helps me to emphasize what I think is important. So I made sure that my CV is available on an easy to reach URL: https://amedee.be/cv.

Then I made two short template messages, one in Dutch and one in English, to thank people for contacting me, where they can find my CV, and — for the LinkedIn people — what my email address is. That’s because I find it easier to track conversations in my mailbox. I can also give labels and flags to conversations, to help me in identifying the interesting ones.

LinkedIn

On LinkedIn, it went like this:

  • Read message.
  • Copy contact details to a spreadsheet.
  • Copy/paste the Dutch or English template message, so that they have my CV and email address.
  • If their message was really interesting(*), add an additional message that I’ll get back to them, and close the conversation. That’ll move it to the top of the message queue.
  • If their message wasn’t interesting or unclear, archive the conversation. If they come back after reading my CV, they’ll either end up in my mailbox, or if they use LinkedIn again, they’ll pop back up at the top of the message queue. But I don’t want to worry about the kind of recruiters that are just “fishing”.

This way I reduced my LinkedIn messages from about 150 to about 20. That’s 20 job offers that I want to give a second, more detailed look. Wow. And that’s just LinkedIn.

(*) What makes a message interesting?

  • It’s relevant.
  • The job isn’t too far to commute.
  • They clearly read my LinkedIn profile.
  • There is a detailed job description.
  • My gut feeling.

Email

Email is another huge source of messages. Fortunately Gmail gives me some tools there to help me. One of the first things I had to do, was to clean out my mailbox. Seriously. It was a dumpster fire. My Inbox had thousands (!) of unread emails. I used rules, filters, deleted emails (I think I deleted more than 100 000 emails), archived emails, and unsubscribed from many, many newsletters that had accumulated over the years. I am now at the point where there are currently 3 emails in my Primary Inbox, all 3 of them actionable items that I expect to finish in the next two weeks, and then those emails will be archived too.

Then, for any recent(ish) email about job offers, I labeled them as “jobhunt” and moved them to the Updates Inbox. That’s the Inbox that Gmail already used automatically for most of these emails, so that was convenient. (For those who don’t know: Gmail has 5 inboxes: Primary, Social, Promotions, Updates and Forums.) At this moment, there are 326 emails labeled “jobhunt”. I’m sure that there will be some overlap with LinkedIn, but still. That’s a lot.

I’ll be using Gmail’s stars, “Important” flag, and archive, to classify emails. Again, just like with LinkedIn, if an email isn’t really interesting at first glance, it’ll go to the archive after I’ve send them a short default message.

Phone

I get it. Really, I do. For some of you, talking on the phone comes naturally, you do it all the time, and it’s your preferred way of communication. For you it’s the fastest way to do your job.

But for me it’s a tough one. I wouldn’t say that I have outright phone phobia, but phone really is my least favorite communication channel. I need some time to charge myself up for a planned phone call, and afterwards I need some time to process it. Even if it is just writing down some notes about what was discussed and looking up some stuff.

It also has to do with how I process information. Speech is in one direction, always forward, and always at the same speed. You can’t rewind speech. But that’s not how my brain works. I want to read something again and again, or skip a paragraph, or first jump to a conclusion and then jump back to see how we got to that conclusion. Sometimes my thoughts go faster than how I express them, and putting it in writing helps me to see the gaps.

Calls out of the blue? I prefer to avoid those. Really. Especially the ones where people just want to get to know me. In the time it takes for me to do one such phone call (and I do take them seriously), I’m able to process several emails. So I very much prefer to focus first on contacts who have something concrete and actionable.

As mentioned above, I record contact information in a spreadsheet. I then import that information into Google Contacts, so that when someone calls me, I see their name on the screen of my phone, and not just a number. That also helps me to decide to pick up the phone or let it go to voicemail. I will get back to those that go to voicemail, but it’ll just be at my own pace.

Social media presence

I’m starting to put myself a bit more out there, by engaging in conversations on LinkedIn. I have also picked up blogging again, and I’m sharing links to my posts on LinkedIn, Facebook and Twitter. Besides my Facebook profile, I also have a Facebook page, but I’m not using that fanatically, because for myself at this point I don’t see Facebook as a professional tool.

On Twitter I have two accounts: @amedee and @AmedeeVanGasse. The former is mostly for personal stuff, and is mostly in Dutch. The latter is one that I created to tweet at tech conferences, but we all know how many tech conferences there were in the last 1.5 years… 🙂 Most tweets there will be in English.

Epilogue

I feel like this has become a very long blog post. Maybe too long, I don’t know. Maybe I should have split it up in several parts? But for me it felt like one story I had to tell.

If any of you social media gurus out there have some opinions to share, that’s what the comment box below is for. 🙂

The post So, how is the jobhunt going? appeared first on amedee.be.

In my job as domain architect for "infrastructure", I often come across stakeholders that have no common understanding of what infrastructure means in an enterprise architecture. Since then, I am trying to figure out a way to easily explain it - to find a common, generic view on what infrastructure entails. If successful, I could use this common view to provide context on the many, many IT projects that are going around.

July 15, 2021

Ik schrijf al meer dan twintig jaar over allerlei technische onderwerpen en ik heb daarbuiten nog heel wat andere interesses. Maar het is nog maar enkele jaren geleden dat ik in een epifanie plots de rode draad in al die onderwerpen zag: decentralisatie.

  • Ik geloof in de kracht van opensourcesoftware omdat de gebruikers daardoor niet alleen consument maar ook producent zijn.

  • Ik ben een groot voorstander van self-hosted software zodat je niet afhankelijk bent van clouddiensten van grote bedrijven.

  • Tien jaar geleden al schreef ik over Bitcoin en ik vind blockchains en andere vormen van distributed ledgers fascinerende technologie om zonder centrale controle transacties te kunnen uitvoeren.

  • In mijn strijd voor meer privacy vind ik het vooral belangrijk dat mensen controle over hun eigen data hebben, en daarom ben ik een grote fan van technologieën zoals Nextcloud, Solid en end-to-end encryptie.

  • Ik vind het belangrijk dat je zelf dingen kunt maken en programmeren, en ik ben dan ook blij dat ik deel uitmaak van de DIY/makersbeweging die dit democratiseert met de Raspberry Pi, ESP32, Arduino, 3d-printers, fablabs enzovoort.

  • Ik vind dat huishoudens zoveel mogelijk zelfvoorzienend zouden moeten zijn, niet alleen voor energie, maar ook voor voedsel.

More Equal Animals - The Subtle Art of True Democracy (Bron: Dan Larimer)

Toen ik dan in het begin van dit jaar Dan Larimers (gratis te downloaden) boek More Equal Animals - The Subtle Art of True Democracy ontdekte, las ik dit bijna in één ruk uit. Op bijna elke pagina las ik wel inzichten die verweven waren met de rode draad in mijn interesses.

In essentie is een blockchain een techniek om zonder centrale aansturing tot consensus te komen. Bij Bitcoin is dat consensus over transacties, maar volgens Larimer kun je de principes van blockchains ook op onze samenleving zelf toepassen. Een succesvolle samenleving implementeert volgens Larimer een proces dat zoveel mogelijk tot consensus leidt.

Democratie is in die ogen een proces om geschillen te beslechten met meerdere partijen en dat is iets waarin blockchains zo goed zijn. Echte democratie gaat volgens Larimer dan ook over het coördineren met andere personen terwijl je nog je persoonlijke autonomie en macht behoudt. En de ideale vorm van de samenleving bestaat dan uit gedecentraliseerde, autonome gemeenschappen op alle niveaus.

Voor PC-Active schreef ik uitgebreider over de visie die Larimer in zijn boek beschreef. Als onderwerpen zoals persoonlijke autonomie, gedecentraliseerd bestuur, een antifragiele samenleving en de gevaren van moral hazard je nauw aan het hart liggen, dan is het boek More Equal Animals een aanrader.

July 12, 2021

Quick public service announcement; Autoptimize 2.9 is almost ready to be released but given the planned release of WordPress 5.8 (July 20th) and the risk of support requests mixing up WordPress core update related issues with the Autoptimize update related issues, Autoptimize 2.9 will probably be released one week after WordPress 5.8, so on or around Tuesday 27th.

If you’re eager to use 2.9 (with better image optimization, improved JS optimization and per page/ post Autoptimize settings) you can off course download the beta here immediately.

July 10, 2021

Ik heb de titel van deze blog gewijzigd naar ‘Paepe Thoon‘, omdat dat mijn favoriete Leuvenaar is.

Récit de 3 jours de bikepacking pas toujours entre les gouttes à travers le Hainaut, le nord de la France et le Namurois

Je tiens ma formation initiale et ma philosophie du bikepacking de Thierry Crouzet, auteur du livre « Une initiation au bikepacking » (dans lequel je fais un peu de figuration) : Partir en autonomie, mais le plus léger possible, éviter les routes à tout prix,préférer l’aventure et la découverte à la performance ou à la distance.

Élève appliqué de Thierry, je me transforme en professeur pour initier mon filleul Loïc. Anecdote amusante : la différence d’âge entre Thierry et moi est la même qu’entre moi et Loïc. L’enseignement se propage, de génération en génération.

Après plusieurs virées dans les magasins de camping et une très grosse sortie de préparation de 112km, rendez-vous est pris pour notre premier trip de bikepacking sur une trace que j’ai dessinée pour traverser la province du Hainaut du Nord au sud, couper à travers la France dans la région de Givet avant de remonter le Namurois.

Jour 1 : le Hainaut sauvage, 103km, 1250d+

Nous nous retrouvons le vendredi matin sur le Ravel de Genappe. Je suis en retard : je connais tellement ce parcours que j’étais persuadé qu’il faisait 10km. Mon compteur indique déjà 15km lorsque je trouve Loïc qui piaffe d’impatience.

Le temps de me présenter sa config bikepack (il a notamment troqué le Camelbak sur le dos pour une ceinture porte-gourde) et nous voilà partis. À peine sorti des routes de Genappe et nous sommes confrontés à des chemins qui viennent de vivre deux mois de pluie quasi permanente. Cela signifie d’énormes flaques et une végétation plus qu’abondante. J’avais été témoin, sur mes sentiers habituels, de chemins se refermant complètement en trois ou quatre jours de beau temps après des semaines de pluie.

De tout côté, nous sommes entourés par les ronces, les orties. Mes bras deviennent un véritable dictionnaire des différentes formes de piqures et de lacérations. Il y’a les pointues, les griffues, celles qui se boursouflent, celles qui grattent, celles qui saignent. Loïc se marre en m’entendant hurler. Car je suis de ceux qui hurlent avant d’avoir mal, un cri rauque à mi-chemin entre banzaï et le hurlement de douleur. Loïc, lui, préfère garder son énergie et souffre en silence.

Le contournement des flaques s’avère parfois acrobatique et, moins agile que Loïc, je glisse sur un léger éperon de boue, les deux pieds et les fesses dans une énorme mare de gadoue.

Le soleil nous aide à prendre l’essorage de chaussettes à la rigolade sous la caméra amusée de Loïc qui filme. Je ne le sais pas encore, mais l’eau sera le thème central de notre épopée.

Nous dépassons enfin Fleurus pour traverser la banlieue de Charleroi par Chatelineau et Châtelet. À travers des rues peu engageantes qui serpentent entre des façades borgnes, nous suivons la trace qui s’engouffre sous un pont d’autoroute, nous conduit entre deux maisons pour nous faire déboucher soudainement sur de magnifiques sentiers à travers les champs. Comme si les habitants tenaient à cacher la beauté de leur région aux citadins et aux automobilistes.

Après des kilomètres assez plats, le dénivelé se fait brusquement sentir. Nous atteignons les bois de Loverval pour continuer parmi la région boisée contournant Nalinnes. Si les paysages sont loin d’être époustouflants, la trace est un véritable plaisir, verte, physique et nous fait déboucher dans le chouette village de Thy-le-Château.

Nous nous arrêtons pour un sandwich dans une boucherie. Le boucher nous explique sillonner la région en VTT électrique et est curieux de savoir quelle application nous utilisons pour nos itinéraires. Il note le nom « Komoot » sur un papier avant de s’offusquer lorsque je lui explique que nous nous relayons pour passer les commandes afin d’avoir toujours quelqu’un près des vélos.

« On ne vole pas à Thy-le-Château ! » nous assène-t-il avec conviction. Le sandwich est délicieux et nous continuons à travers des montées et des descentes abruptes, inondées de flaques ou de torrents. Les passages difficiles se succèdent et j’ai le malheur de murmurer que je rêve d’un kilomètre tout plat sur une nationale.

J’ai à peine terminé ma prière que mon mauvais génie m’exauce. Arrivant au pied de Walcourt, étrange village qui flanque une colline abrupte, la trace nous propose de suivre 500m d’une route nationale. Mais celle-ci se révèle incroyablement dangereuse. Une véritable autoroute ! Pour l’éviter, nous devrions remonter toute la pente que nous venons de descendre et faire une boucle de plusieurs kilomètres. Loïc propose de rouler le long de la nationale, derrière le rail de sécurité. « Ça se tente ! » me fait-il.

Nous sommes de cette manière à plusieurs mètres des véhicules et protégés par la barrière. Cependant, ce terre-plein est envahi de ronces, d’orties et des détritus balancés par les automobilistes. Les 500m dans le hurlement des camions et des voitures lancées à vive allure sont très éprouvants. Moi qui suis parfois réveillé par l’autoroute à plus de 3km de mon domicile, je me dis qu’on sous-estime complètement la pollution sonore du transport automobile.

Cette épreuve terminée, nous attaquons la dernière colline avant d’arriver aux Lacs de l’Eau d’Heure, objectif assumé pour notre première pause.

Juste avant le barrage de la Plate Taille, nous bifurquons vers une zone de balade autour du lac. Nous nous planquons dans un petit bosquet où, malgré les panneaux d’interdiction, j’enfile un maillot pour profiter d’une eau délicieuse à 19°C. Sur la rive d’en face, je pointe l’endroit où Loïc a fait son baptême de plongée en ma compagnie.

Le cuissard renfilé, je remonte sur ma selle et nous repartons. La trace nous conduit dans des petits sentiers qui longent la route du barrage. Nous arrivons sur le parking du spot de plongée où nous sommes censés retrouver la route, séparée de nous par une barrière fermée. Nous continuons un peu au hasard dans les bois avant de tomber sur le village de Cerfontaine.

Nous quittons désormais la civilisation. Plusieurs kilomètres de sentiers escarpés nous attendent. Loïc voit passer un sanglier. Je vois plusieurs biches. La région est sauvage. Deux choses inquiètent Loïc. Le risque d’orage et la question de trouver à manger. Hein chef ?

Heureusement, nous débouchons sur Mariembourg où une terrasse accueillante nous tend les bras au centre du village. Nous mangeons bercés par les cris de quelques villageois se préparant pour le match de foot du soir à grand renfort de canettes de bière.

Nous étudions la trace, occupation principale d’un bikepacker en terrasse. J’avais prévu un zigzag à proximité de Couvin pour aller découvrir le canyon « Fondry des Chiens ». Étant donné l’heure avancée, je suggère de couper à travers la réserve naturelle de Dourbes.

Nous sommes à peine sortis de Mariembourg que Loïc reconnait la gare. Nous sommes sur les terres où Roudou nous avait emmenés lors d’un mémorable week-end VTTnet en 2015.

La réserve naturelle de Dourbes est tout sauf plate. Un régal de vététiste. Un peu moins avec près de 100bornes dans les pattes. Ça fait partie du bikepacking : parler de régal pour ce qui te fait pester au moment même.

Nous arrivons sur les berges du Viroin. La trace nous fait monter vers le château de Haute-Roche, véritable nid d’aigle qui semble inaccessible. La pente est tellement abrupte qu’il faut escalader d’une main en tirant les vélos de l’autre. Loïc vient m’aider pour les derniers mètres.

Les ruines de la tour moyenâgeuse se dressent devant nous. Après cet effort, Loïc décide qu’il a bien mérité de contempler la vue. Il contourne la tour par un étroit sentier qui nécessite même un mètre d’escalade sur le mur médiéval. J’hésite à le suivre puis me laisse gagner par son enthousiasme.

Loïc a découvert une terrasse qui surplombe la vallée de manière majestueuse. Derrière nous, la tour, devant le vide et la vue. C’est magnifique.

Loïc a soudain une idée : » Et si on plantait la tente ici ? »

J’hésite. Nous sommes sur une propriété privée. L’à-pic n’est pas loin. Les sardines ne se planteront peut-être pas dans la terre fine de la terrasse. Mais je vois les yeux de Loïc pétiller. Je propose de tester de planter une sardine pour voir si c’est faisable. Loïc propose une manière de disposer les deux tentes sur la terrasse de manière à être le plus éloigné possible du trou. Nous finissons par retourner aux vélos, décrocher tous les sacs pour les amener sur notre terrasse. Il reste à faire passer les vélos eux-mêmes par le même chemin. C’est acrobatique, mais nous y arrivons et bénéficions d’un coucher de soleil sublime alors que nous montons nos tentes.

J’utilise un peu d’eau de mon Camelbak pour improviser une douche rapide. Je tends mes fesses à toute la vallée. Vue pour vue, paysage pour paysage.

De la vallée, les faibles cris nous informent que les Belges perdent le match de foot. Nous nous couchons à l’heure où les multiples camps scouts qui parsèment la vallée décident de se lancer dans des chants qui relèvent plus du cri permanent. Au bruit du matelas pneumatique, je devine que Loïc se retourne et ne trouve pas le sommeil.

Jour 2 : la brousse française, 80km, 1500d+

Les supporters et les scouts ont à peine achevé leur tintamarre que les coqs de la vallée prennent le relais. Il n’est pas encore 7h que j’émerge de ma tente. Loïc a très mal dormi et est abasourdi par l’humidité qui dégouline dans sa tente. J’espérais que l’altitude nous protégerait de l’humidité du Viroin, il n’empêche que tout est trempé. Mon Camelbak, mal fermé, s’est vidé dans mon sac de cadre qui, parfaitement étanche, m’offre le premier vélo avec piscine intérieure, comble du luxe.

Heureusement, il fait relativement beau. J’avais prévenu Loïc de compter une grosse heure pour le remballage des affaires, surtout la première fois. Le fait de devoir repasser les vélos en sens inverse le long de la tour complique encore un peu plus la tâche. Nous pratiquons la philosophie « no trace » et Loïc en profite même pour ramasser des vieilles canettes. Au final, il nous faut plus d’1h30 pour être enfin prêts à pédaler. Nous traversons les bois, descendons le long d’une route où nous aidons un scout flamand un peu perdu à s’orienter avant d’accomplir la courte, mais superbe escalade des canons de Vierves. Escalade que nous avions accomplie en 2015 avec Roudou et sa bande sans que j’en aie le moindre souvenir. En pensée, Loïc et moi envoyons nos amitiés et nos souvenirs aux copains de VTTnet.

La trace nous fait ensuite longer la route par un single escarpé avant de nous conduire à Treignes où nous déjeunons sur le parking d’un Louis Delhaize. Je constate que la trace fait un gros détour pour éviter 3km de route et nous fais escalader un énorme mamelon pour en redescendre un peu plus loin en France. La route étant peu fréquentée, je propose d’avancer par la route pour gagner du temps. L’avenir devait révéler ce choix fort judicieux.

Une fois en France, je m’arrange pour repiquer vers la trace. Nous faisons une belle escalade en direction du fort romain du Mont Vireux. Comme le fort en lui-même est au bout d’un long cul-de-sac, nous décidons de ne pas le visiter et de descendre immédiatement sur Vireux où nous traversons la Meuse.

Nous escaladons la ville. Je m’arrête à la dernière maison avant la forêt pour me ravitailler en eau auprès d’habitants absolument charmants et un peu déçus de ne pas pouvoir faire plus pour moi que de me donner simplement de l’eau.

Nous quittons désormais la civilisation pour nous enfoncer dans les plateaux au sud de Givet. Les chemins forestiers sont magnifiques, en montée permanente. Quelques panneaux indiquent une propriété privée. Nous croisons cependant un 4×4 dont le conducteur nous fait un signe amical qui me rassure sur le fait que le chemin soit public. Mais, au détour d’un sentier, une grande maison se dresse, absurde en un endroit aussi reculé. La trace la contourne et nous fait arriver devant une barrière un peu bringuebalante. Je me dis que nous sommes sur le terrain de la maison, qu’il faut en sortir. Nous passons donc la barrière, prenant soin de la refermer, et continuons une escalade splendide et très physique.

Au détour d’un tournant, je tombe sur une harde de sangliers. Plusieurs adultes protègent une quinzaine de marcassins. Les adultes hésitent en me voyant arriver. L’un me fait face avant de changer d’avis et emmener toute la troupe dans la forêt où je les vois détaler. Loïc arrive un peu après et nous continuons pour tomber sur une harde d’un autre type : des humains. Un patriarche semble faire découvrir le domaine à quelques adultes et une flopée d’enfants autour d’un pick-up. Il nous arrête d’un air autoritaire et nous demande ce que nous faisons sur cette propriété privée.

Je lui explique ma méprise à la barrière et la trace GPS en toute sincérité. Il accepte avec bonne grâce mes explications et tente de nous indiquer un chemin qui nous conviendrait. Je promets de tenter de marquer le chemin comme privé sur Komoot (sans réfléchir au fait que c’est en fait sur OpenStreetMap qu’il faut le marquer et que je n’ai pas encore réussi à le faire). Finalement, il nous indique la barrière la plus proche pour sortir du domaine qui se révèle être exactement le chemin indiqué par notre trace. Nous recroisons la harde de sangliers et de marcassins.

Nous escaladons la barrière en remarquant l’immensité de la propriété privée que nous avons traversée et sommes enfin sur un chemin public qui continue sur un plateau avant de foncer vers le creux qui nous sépare de la Pointe de Givet, Pointe que nous devons escalader à travers un single beaucoup trop humide et trop gras pour mes pneus. J’en suis réduit à pousser mon vélo en regardant Loïc escalader comme un chamois. Au cours du périple, les descentes et les montées trop grasses seront souvent à la limite du petit torrent de montagne. Une nouvelle discipline est née : le bikepack-canyoning.

Le sommet nous accueille sous forme de vastes plaines de hautes graminées où le chemin semble se perdre. La trace descend dans une gorge sensée déboucher sur la banlieue est de Givet. Mais la zone a été récemment déboisée. Nous descendons au milieu des cadavres de troncs et de branches dans un paysage d’apocalypse sylvestre. La zone déboisée s’arrête nette face à un mur infranchissable de ronces et de buissons. La route n’est qu’à 200m d’après le GPS, mais ces 200m semblent infranchissables. Nous remontons péniblement à travers les bois pour tenter de trouver un contournement.

Loïc fait remarquer que le paysage ressemble à une savane africaine. Nous roulons à l’aveuglette. Parfois, un souvenir de chemin semble nous indiquer une direction. Nous regagnons l’abri de quelques arbres avant de déboucher sur une vaste prairie de très hautes graminées, herbes et fleurs. Comme nous sommes beaucoup trop à l’ouest, je propose de piquer vers l’est. Une légère éclaircie dans un taillis nous permet de nous faufiler dans une pente boisée que je dévale sur les fesses, Loïc sur les pédales. Le pied de cette raide descente nous fait déboucher sur un champ de blé gigantesque. Du blé à perte de vue et aucun chemin, aucun dégagement. Nous nous résignons à la traverser en suivant des traces de tracteur afin de ne pas saccager les cultures. Les traces nous permettent de traverser le champ en largeur avant de s’éloigner vers l’ouest où la limite du champ n’est même pas visible.

À travers une haie d’aubépines particulièrement touffue, nous apercevons une seconde prairie. Avec force hurlements de douleur et de rage, nous faisons traverser la haie à nos vélos avant de suivre le même passage. De la prairie de pâturage, il devient facile de regagner un chemin desservant l’arrière des jardins de quelques maisons.

Après plusieurs heures de galère et très peu de kilomètres parcourus, nous regagnons enfin la civilisation. Loïc vient de faire son baptême de cet élément essentiel du bikepacking : l’azimut improvisé (autrement connu sous le nom de « On est complètement paumé ! »).

On pourrait croire qu’avec les GPS et la cartographie moderne, se perdre est devenu impossible. Mais la réalité changeante et vivante de la nature s’accommode mal avec la fixité d’une carte. L’état d’esprit du bikepacker passera rapidement du « Trouver le chemin le plus engageant pour arriver à destination » à « Trouver un chemin pour arriver à destination » à « Trouver un chemin praticable » pour finir par un « Mon royaume pour trouver n’importe quoi qui me permet tout simplement de passer ». Après des passages ardus dans les ronces ou les aubépines, après avoir dévalé des pentes particulièrement raides, l’idée de faire demi-tour n’est même plus envisageable. Il faut lutter pour avancer, pour survivre.

Un aphorisme me vient spontanément aux lèvres : « L’aventure commence lorsque tu as envie qu’elle s’arrête ».

Nous pénétrons alors dans Givet par l’ouest alors que j’avais prévu d’éviter la ville. Nous avons faim, nous sommes fatigués et nous n’avons fait qu’une vingtaine de kilomètres. Loïc a du mal de se rendre compte du temps perdu.

Sur une placette un peu glauque où se montent quelques maigres attractions foraines, nous enfilons un sandwich. Pour ma part, un sandwich que je viens d’acheter, mais pour Loïc, un sandwich particulièrement savoureux, car acheté le matin en Belgique et qui a fait toute l’aventure accroché au vélo. Se décrochant même avant une violente descente, emportant la veste de Loïc au passage et nécessitant une réescalade de la pente pour récupérer ses biens.

Hors de Givet, la nature reprend ses droits. Les montées boueuses succèdent aux singles envahis de flaques. Nous retrouvons la Belgique au détour d’un champ. Après quelques patelins typiquement namurois (les différences architecturales entre les bleds hennuyers, français et namurois me sautent aux yeux), nous enchainons de véritables montagnes russes jouant sur les berges de la Lesse.

Alors que j’ai un excellent rythme, une pause impromptue s’impose, le lieu me subjuguant par la beauté un peu irréelle d’une petite cascade. Je m’arrête et m’offre un bain de pieds tandis que Loïc prend des photos. Une fois sortis des gorges de la Lesse, nous nous arrêtons pour étudier la situation.

J’avais prévu un itinéraire initial de 330km, mais, Loïc devant être absolument rentré le 4 au soir, j’ai également concocté un itinéraire de secours de 270km pour le cas où nous aurions du retard. Les itinéraires divergeaient un peu après le retour en Belgique, l’un faisant une boucle par Rochefort, l’autre revenant en droite ligne vers Waterloo.

Par le plus grand des hasards, je constate que je me suis arrêté littéralement au point de divergence. Étant donné le temps perdu le matin, il me semble beaucoup plus sage de prendre l’itinéraire court, au grand dam de Loïc, très motivé, mais très conscient de la deadline.

Le seul problème est que mon itinéraire court ne passe par aucune ville digne de ce nom avant le lendemain, que je n’ai repéré aucun camping. Loïc me demande d’une petite voix inquiète si on va devoir se coucher le ventre vide. Parce qu’il y’aura aussi la question de trouver à manger. Hein chef ?

Je propose d’aviser un peu plus loin. Sur le chemin, quelques moutons échappés de leur enclos me regardent méchamment. Le mâle dominant commence même à gratter du sabot. Je leur crie dessus en fonçant, ils s’écartent.

Arrivés à un croisement, nous consultons les restaurants disponibles dans les quelques villages aux alentours. Un détour par Ciney me semble la seule solution pour s’assurer un restaurant ouvert. Nous sommes au milieu de nos hésitations lorsqu’un vététiste en plein effort s’arrête à notre hauteur. Tout en épongeant la sueur qui l’inonde, il nous propose son aide. Sa connaissance du lieu est bienvenue : il nous conseille d’aller à Spontin pour être sûrs d’avoir à manger puis d’aller dans un super camping au bord du Bocq. Par le plus grand des hasards, il est justement en train de flécher un parcours VTT qui passe tout prêt.

Nous le remercions et nous mettons à suivre ses instructions et ses flèches. Un petit détour assez pittoresque qui nous fait passer dans des singles relativement techniques par moment. C’est vallonné et la journée commence à se faire sentir. Psychologiquement, l’idée d’être presque arrivés rend ces 15km particulièrement éprouvants. Après une grande descente nous débouchons sur un carrefour au milieu de Spontin, carrefour orné, ô miracle, d’une terrasse de restaurant. Nous nous installons sans hésiter. Je commande une panna cotta en entrée.

Loïc est très inquiet à l’idée de ne pas avoir de place au camping recommandé par notre confrère. Le téléphone ne répond pas. De plus, ce camping est à une dizaine de kilomètres, dans un creux qu’il faudra escalader au matin. Alors que nous mangeons, j’aperçois derrière Loïc un panneau au carrefour qui indique un camping à seulement 2km. Un coup d’oeil sur la carte m’apprend que ce camping est à quelques centaines de mètres de notre trace. Je téléphone et le gérant me répond qu’il n’y a aucun souci de place.

Après le repas, nous sautons sur nos montures pour gravir ces 2 derniers kilomètres, le camping étant sur une hauteur. Rasséréné par la certitude d’avoir un logement et le ventre plein, Loïc me lâche complètement dans la côte. Son enthousiasme est multiplié, car il reconnait le camping. C’est une constante de ce tour : alors que je cherche à lui faire découvrir des choses, il reconnait sans cesse les lieux et les paysages pour y être venu à l’une ou l’autre occasion. Parfois même avec moi.

L’emplacement de camping est magnifique, aéré, calme avec une vue superbe. Par contre, les douches sont bouillantes sans possibilité de régler la température, les toilettes sont « à terrasse », sans planche ni papier. Je préfère encore chier dans les bois, mais la douche fait du bien.

La nuit est ponctuée d’épisode de pluie. Je croise les doigts pour qu’il fasse sec au moment de remballer la tente. Je n’ai encore jamais remballé le matériel sous la pluie.

Jour 3 : l’aquanamurois, 97km, 1200d+ 

À 7h30, je commence à secouer la tente de Loïc. Je l’appelle. Pas un bruit. Je recommence, plus fort. Je secoue son auvent. J’espère qu’il est toujours vivant. À ma cinquième tentative, un léger grognement me répond : « Gnnn… »

Loïc a dormi comme un bébé. Il émerge. Nous remballons paisiblement sous un grand soleil et faisons sécher les tentes.

La grande inquiétude de la journée, ce sont les menaces d’orage. Jusqu’à présent, nous sommes littéralement passés entre les gouttes. Nous précédons les gros orages de quelques heures, roulant toujours dans des éclaircies.

Sous un soleil très vite violent, nous nous échappons dans une série de petits singles envahis de végétation avant de commencer l’escalade pour sortir de Crupet. Nous escaladons un magnifique chemin à plus de 15%. Sur la gauche, la vue vers la vallée est absolument à couper le souffle avec des myriades de fleurs bleues au premier plan. À moins que ce ne soit la pente qui coupe le souffle. Des randonneurs nous encouragent, je suis incapable de répondre. Le sommet se profile au bord d’un camp scout. Après quelques centaines de mètres sur la route, un panneau indiquant une église médiévale attire mon attention. Cette fois-ci, c’est moi qui reconnais l’endroit ! Nous sommes à 1km du lieu de mon mariage. J’entraine Loïc dans un bref aller-retour pour envoyer une photo souvenir à mon épouse.

À partir de là, je connais l’endroit pour y être venu de multiples fois à vélo. Après des traversées de champs, nous nous enfonçons dans les forêts du Namurois, forêts aux chemins dévastés par les orages et les torrents de boue. Au village de Sart-Bernard, j’interpelle un habitant pour savoir s’il y’a un magasin ou une boulangerie dans les environs. À sa réponse, je comprends que j’aurais pu tout aussi bien lui demander un complexe cinéma 15 salles, un parc d’attractions et un centre d’affaires.

Nous nous enfonçons donc dans la forêt, zigzaguant entre les chemins privés, pour déboucher finalement sur Dave. Un kilomètre de nationale malheureusement incontournable nous permet d’aller traverser la Meuse sur une écluse juste au moment où celle-ci commence à se remplir pour laisser passer un bateau. Nous continuons le long du fleuve pour aller déguster une crêpe à Wépion. Le temps se couvre, mais reste sec.

La crêpe engloutie, il est temps de sortir du lit de la Meuse. Ma trace passe par une côte que j’ai déjà eu le plaisir d’apprécier : le Fonds des Chênes. Jamais trop pentue ni technique, la côte est cependant très longue et se durcit vers la fin, alors même qu’on a l’impression de sortir du bois et d’arriver dans un quartier résidentiel.

J’arrive au sommet lorsque les premières gouttes commencent à tomber. J’ai à peine le temps d’enfiler ma veste que le déluge est sur nous. Abrité sous un arbre, j’attends Loïc qui, je l’apprendrai après, a perdu beaucoup de temps en continuant tout droit dans une propriété privée.

À partir de ce moment-là, nous allons rouler sous des trombes d’eau incessantes. À travers les bois, nous descendons sur Malonne dont nous escaladons le cimetière à travers des lacets dignes d’un col alpin. La trace traverse littéralement le cimetière au milieu des tombes. Loïc s’étonne. Je réponds que, au moins, on ne dérange personne. C’est ensuite la descente sur Seneffe avant de longer la Sambre.

Lors de notre journée de préparation, nous sommes passés par là dans l’autre sens. Nous sommes en terrain connu, le côté exploration du bikepacking s’estompe pour laisser la place à la douleur psychologique du retour. Étant donné la pluie, je suis heureux de rentrer. Je n’ose imaginer installer une tente sous la pluie, renfiler des vêtements trempés le lendemain.

Nous n’essayons même plus de contourner les flaques qui se sont, de toute façon, transformées en inévitables marigots. Nous roulons des mètres et des mètres avec de l’eau jusqu’aux moyeux, chaque coup de pédale remplissant les chaussures d’eau comme une noria.

Loïc m’a plusieurs fois expliqué être motivé par la pluie. Sous la pluie, il pédale mieux. J’ai en effet observé qu’il supporte assez mal la chaleur alors que, pour moi, rien n’est aussi délectable que d’escalader un col en plein cagnard.

Ses explications se confirment. Loïc fonce, escalade. J’ai de plus en plus de mal à le suivre. L’eau me mine, ma nouvelle selle me torture les fesses. Nous traversons Spy, les plaines de Ligny, probablement tout aussi inondées qu’en 1815 et le golf de Rigenée. La trace traverse le bois Pigeolet, mais je me souviens avoir été bloqué au château de Cocriamont lors d’une de mes aventures antérieures. J’impose un demi-tour et nous gagnons Sart-Dames-Avelines par la route.

Alors que nous arrivons à Genappe, la pluie qui s’était déjà un peu calmée s’arrête tout à fait. Nous en profitons pour prendre un dernier verre en terrasse avant de nous dire au revoir. Nous avons le sentiment d’être à la maison.

Il me reste néanmoins encore 15km à faire. 15km essentiellement de Ravel. Mes chaussures sont presque sèches, l’optimisme est de mise.

C’est sans compter que le Ravel est inondé par endroit, traversé de coulées de boue. Certaines maisons se sont barricadées avec des sacs de sable. Des arbres arrachés rendent le passage compliqué. Alors que je traverse une flaque que je croyais étendue, mais peu profonde, le Ravel étant en théorie essentiellement plat, je m’enfonce jusqu’au moyeu. Je suis recouvert, ainsi que mon vélo et mes sacs, d’une boue jaune, grasse, épaisse et collante.

Il était dit que je ne pouvais pas arriver sec à Louvain-la-Neuve…

280km, près de 4000m de d+ et une expérience mémorable. Je suis enchanté d’avoir pu condenser en 3 jours toutes les expériences d’un trip de bikepacking : camping sauvage, heures perdues à pousser le vélo dans une brousse sans chemin, découragements suivis d’espoirs, pauses imprévues et terrasses délectables.

Maintenant que Loïc a gouté aux joies du bikepacking « extreme », je n’ai qu’une envie : qu’on reparte pour explorer d’autres régions. J’ai une attirance toute spéciale pour les Fagnes… Par contre, cette expérience de la pluie me fait renoncer au rêve de parcourir l’Écosse en bikepacking.

Alors qu’une Grande Traversée du Massif Central (GTMC pour les intimes) se profile avec Thierry, deux inquiétudes restent vives : mes fesses me font toujours autant souffrir (peut-être devrais-je passer le cap du tout suspendu) et je ne me sens pas psychologiquement armé pour affronter un bivouac sous la pluie.

Mais, après tout, l’aventure ne commence-t-elle pas au moment où tu as envie qu’elle s’arrête ?

Oubliez un instant les réseaux sociaux et abonnez-vous par mail ou par RSS (max 2 billets par semaine et rien d’autre). Dernier livre paru : Printeurs, thriller cyberpunk. Pour soutenir l’auteur, offrez et partagez ses livres.

Ce texte est publié sous la licence CC-By BE.

July 08, 2021

I published the following diary on isc.sans.edu: “Using Sudo with Python For More Security Controls“:

I’m a big fan of the Sudo command. This tool, available on every UNIX flavor, allows system administrators to provide access to certain users/groups to certain commands as root or another user. This is performed with a lot of granularity in the access rights and logging/reporting features. I’m using it for many years and I’m still learning great stuff about it. Yesterday, at the Pass-The-Salt conference, Peter Czanik presented a great feature of Sudo (available since version 1.9): the ability to extend features using Python modules… [Read more]

The post [SANS ISC] Using Sudo with Python For More Security Controls appeared first on /dev/random.

As I mentioned in An IT services overview I try to keep track of the architecture and designs of the IT services and solutions in a way that I feel helps me keep in touch with all the various services and solutions out there. Similar to how system administrators try to find a balance while working on documentation (which is often considered a chore) and using a structure that is sufficiently simple and standard for the organization to benefit from, architects should try to keep track of architecturally relevant information as well.

So in this post, I'm going to explain a bit more on how I approach documenting service and solution insights for architectural relevance.

July 07, 2021

I did not write any wrap-up for a while because we are all stuck at home and most conference organizers still decided to cancel live events (even if it seems to change by the end of 2021 where some nice events are already scheduled). For the second time, Pass-The-Salt was converted to a virtual event. I like this small event with a great atmosphere. This edition in a few numbers: they received 27 proposals and selected 16 amongst them. They were presented spread across three half-days. Let’s review what was presented.

The first day started with Eloi Benoist-Vanderbeken who talked about jailbreak detection mechanisms in iOS and how to bypass them. iOS is a closed operating system and Apple really takes this seriously. Indeed, when a device has been “jailbroken”, all security measures are bypassed and the user gains full control of the device. Some apps really take this as a major problem and try to detect this. Think about banking applications but also games. To detect how the jailbreak detection is implemented, it’s mandatory to reverse the app to read the code. Also, Apple restricts a lot of operations: only signed code can be executed, side loading is not allowed (you remember the Epic Game problem?) and apps may only use public API’s. One of the best tools to help is debugging an app is Frida. How to debug an iOS app? Without a jailbreak in place, Frida is injected using ptrace but the app must be repackaged, with a lot of side effects. With a jailbreak already in place, Frida can just attach to the process. As a case study, a banking app was taken as an example. When the app is launched on a jailbroken device, it just crashes. A first set of checks did not reveal anything special. Check deeper, the app tried to trigger an invalid entry error. Some Frida usage examples were reviewed, based on hooks. Frida can hook syscalls to get information about the parameters and/or return values or completely replace the syscall with another function. It can also execute a function before any syscall. This is very interesting to follow what’s happening (example: to keep an eye on decrypted data). Another technique is the classic usage of breakpoints. Some techniques to detect jailbreaks: access specific files with open(), utime(), stats(), …), try to detect the presence of a debugger, check the parent PID, is the root filesystem writable?

The second presentation was the one of Esther Onfroy. She’s an Android expert, hacktivist and speaker. The title of Esther’s presentation was “Pithus: let’s open the Android pandora’s box”. Pithus is a project that focus on analyzing Android applications (APK). Why a new project? According to Esther, threat intelligence should not be the proprietary of a private company. For some, the price is a subscription is crazy high so the idea to develop an open-source tool based on free software and self-hostable. Pithus is able to check domains, certificates, the application time line, behavioral analysis and is linked to 3rd party services like MalwareBazaar. A specific function was presented deeper: The CFG (“Control Flow Graph”) dissection using AndroGuard. This is an interesting feature to search for share code:

Benoit Forgette presented “Hook as you want it“. Android is very popular in the mobile landscape. It provides a standard bootloader, an API for hardware, an ARM trustzone, High-level language (easy to write apps) and IPC. Review of the Android permission enforcement mechanism. AST is a tree and a hook is a programming trick that helps to intercept system calls. Once again here Frida will be used.

The next talk was “PatrowlHears and Survival tips for prioritizing threats” by Nicolas Mattiocco. It starts with a quick overview of Patrowl with themotto “demotivate attackers”. In vulnerability management, process is key: “How to manage vulnerabilities?”. It’s not only running a tool at regular interval:

One of the problems is the changing landscape, a lack of skilled people, budget pressure, … Automation is also a key point. And when running the tool more often, you get more findings and more alerts How to improve this? Then Nicolas explained how to prioritise findings. Some ideas: check the CVSS score, are we vulnerable? Is it exposed to the Internet? Is it a critical asset? Functional exploit? Patch or work around? etc… Most important: does it have a nice logo? 🙂 To help in this process, PatrolHears was introduced. It provides scalable, free and open-source solutions for orchestrating Security Operations and providing Threat Intelligence feeds. PatrowlHears is an advanced and real-time Vulnerability Intelligence platform, including CVE, exploits and threats news. Its also a feed aggregator (Packetstorm, Tenable Nessus DB, Metasploit, Exploitdb, ZDI, … Great tool to check if you’re dealing with vulnerability management.

To wrap-up the first day, we had a pretty funny presentation by Michael Hamm from circl.lu. The Luxembourg CERT produces always nice content and it was, once again, the case. Michael presented three live demos (yes, live) of USB devices alterations from a forensic point of view. The first one was about changing the content of a USB disk mounted in “read-only” mode. Yes, it’s possible. That’s why using a physical write-blocker is always highly recommended. The second one, was a kind of DoS attack: By modifying only a few bytes on the filesystem, a Linux system will, once the modified disk is inserted, mount it up to 250 times 😉 Finally, the last demo was even more impressive: by altering the MDR (Master Boot Record) and creating a “Polyglot boot sector”, Michael demonstrated how different files can be seen by different operating systems. Mounting the drive on Linux, some files will be listed, on Windows, other files will be listed.

Day two started with “Fedora CoreOS, a container focused OS to securely deploy and run applications” by Timothée Ravier. Intro to CoreOS: automatic updates, automatic provisioning (Ignition is used to provision nodes on 1st boot), immutable infrastructure and finally huge use of containers. What about security? “Softwares have bugs” so reduce the OS footprint (-> reduce the attack surface). Next, use safer languages: Go & Rust (memory safe languages). They use rpm-ostree to update the system (think about a “git for operating system”. Uses read-only filesystems and a clear split across important directories (/usr, /etc, /var). Everything runs in containers (podman, docker). Confinement with SELinux. Demos! The first one was to run a Matrix server More about CoreOS here.

The second talk was provided by Clément Oudot, a regular PTS speaker: “Hosting Identity in the Cloud with free softwares“. After a nice live song, Clément did a quick introduction/recap about IAM (“Identity and Access Management”) and a review of the market: big players are expensive – pay per user – and complex to install/manage.

So what about FOSS? A lot of tools exist but don’t cover the complete set of IAM features. FusionIAM is a project to propose a unified platform based on different tools: OpenLDAP, LDAP ToolBox, Fusion Directory, LDAP Synchronization Connector, and LemonLDAP:NG. From a features point of view, it provides whites pages, access manager, sync connector, service desk, directory server, directory manager. A great solution to investigate if you need to deploy IAM in a cloud environment.

The next presentation was “Biscuit: pubkey signed token with offline attenuation and Datalog authz policies” by Geoffroy Couprie. Geoffroy works for a hosting company and is often facing authentication issues (example for API’s). A classic way to authenticate users is a JWT (JSON Web Token). They are usually signed using a public key and contain a lot of data. They are not perfect therefore Geoffroy developed a new authentication and authorization token called Biscuit. You can find more information about this project here (code) and here.

Ange Albertini presented “Generating Weird Files“. Ange is known to play a lot with file formats. He already presented several pieces of research where he demonstrated how files can be built in a way that, depending on the program used to open them, they will present different types of content. Today, Ange presented his tool called Mitra. The open-source tool takes 2 files, checks their types, and generates possible polyglots.

What are the strategies used by Mitra?

  • Concatenation: easy because some file format do not need to start at offset 0
  • Cavities (filling empty spaces)
  • Parasite (abusing comments in metadata)
  • Zipper

Ange performed some demos. It was funny to see a reference to 1 file that identified up to 190 different file formats! The conclusion to this? These techniques could be prevented by creating new file formats that starts at offset 0 (magic bytes).

The next time slot was assigned to … me! I presented “Home-Made Distributed Blocklist“. More to come soon on this blog!

We wrapped the afternoon with a very interesting talk by Peter Czanik: “Security alerting made easy using Python“. Peter is a also a regular contributor to the conference and is leading open source products like syslog-ng or Sudo. Even if I’m an old Sudo user (probably for more than 15 years), I’m always learning new stuff about this wonderful tool and it was again the case. Peter explained how to interconnect Sudo and Python. Sudo is a modular tool that accepts plugins in C but also in Python to extend features and controls. You can filter data, alert, sanitize them etc… Peter reviewed some examples of integration.

Day 3 started with some Kunernetes security stuff: “ATT&CKing Kubernetes: A technical deep dive into the new ATT&CK for Containers” by Magno Login. Magno is the maintainer of the following github project: awesome-k8s-security. Then, he talked about the MITRE ATT&CK and more precisely, the matrix dedicated to containers (link). This matrix was created for a while but was updated recently with plenty of new content (thanks to the contribution of volunteers)

The second part of the talk was a suite of attacks scenarios based on a vulnerable Drupal CMS instance. Magno presented many techniques from the MITRE matrix. Scripts described in demos were collected from honeypots or found on C2 servers.

Then we switched to the wonderful world of shellcodes with Yashdeep Saini and Harpreet Singh who presented “Revisiting the Art of Encoder-Fu for novel shellcode obfuscation techniques“. Back to the roots with a review of the differences between x86, x64 and ARM registers and instructions (comparisons, shifting, control flows, stack, data movement and arithmetic operations.

They reviewed how encoders affect the shell code (XOR, Shikata_ga_nai, etc). Another comparison was to show comparative graphs based on the distributions of instructions:

Other observations: encoder types adding laters and branches(mode control, call changes). Other add transformation (more data movements). Why encode? The shell code can be transformed as per transport supported by the target applications. Bad characters can be replaced and more obfuscation.

In Search of Lost Time: A Review of JavaScript Timers in Browsers” by Thomas Rokicki. Timing attacks exploit timing differences to infer secrets from the sanbox.Attacks can be classified in 4 classes: Harare-contention-base attack (Ex: rawhammer.js), Transcient execution attacks (Spectre), Attacks based on system resources (Keystroke attacks, memory deduplication attacks ) and attacks based on browser resources(History sniffing or fingerprinting). This is a topic that remains always difficult to follow for me… I just kept this in mind as a recap to the talk: “If you are concerned by timing attacks, use the TOR browser, they are the strictest on timers countermesures

Then Nils Amiet and Tommaso Gagliardoni presented “ORAMFS: Achieving Storage-Agnostic Privacy“. The important word here is “privacy” or how to store data on a 3rd party service in a safe way. The classic control is to encrypt the complete set of data. But even if data can’t be read more info can be leaked: The size of data, some fields of database will change, etc… Think about an employees db. Example with the good old Truecrypt software: Create a TC volume, encrypted then mounted by the user. A cool feature was the “hidden volume” but can be detected even without the passphrase:

Keep in mind that “Encryption alone does not hide access patterns“. That’s why oramfs (Oblivious Random Access Machines) has been developed. Storage agnostic, written in Rust, resizing supported, multiple encryption cyphers, the tool was demonstrated live. Nice demo!

Finally, to wrap-up day 3, Damien Cauquil presented “Meet Piotr, a firmware emulation tool for trainers and researchers“. Damien performs a lot of IoT trainings and was looking for a nice solution to setup labs. Indeed, even more with the pandemic, it has been challenging to prepare labs. For example, some IoT device might be discontinued or their firmware upgraded to a non-vulnerable version. Another requirement was the size of the lab (to not download gigabytes of data). That’s why Piotr has been developed. It is based on Qemu, Firmadyne and ARM-X. After describing the architecture (how those components are interconnected), Damien performed some demos and booted/exploited some IoT devices. It looks to be a great solution also to researchers. If interested, the code is available here.

And that’s over for this virtual edition of Pass-The-Salt! Let’s hope that the next one will be in-person. Thanks to the organisers for keeping the event up’n’running! All slides are available online as well as videos.

The post Pass-The-Salt 2021 Virtual Wrap-Up appeared first on /dev/random.

I published the following diary on isc.sans.edu: “Python DLL Injection Check“:

They are many security tools that inject DLL into processes running on a Windows system. The classic examples are anti-virus products. They like to inject plenty of code that, combined with API hooking, implements security checks. If DLLs are injected into processes, they can be detected and it’s a common anti-debugging or evasion technique implemented by many malware samples. If you’re interested in such techniques, they are covered in the FOR610 training. The detection relies on a specific API call GetModuleFileName()… [Read more]

The post [SANS ISC] Python DLL Injection Check appeared first on /dev/random.

July 05, 2021

Axl staring at the Boston skyline, waiting for the Fourth of July fireworks to start.
Axl staring at the Boston skyline, waiting for the Fourth of July fireworks to start. It had been years since he saw the fireworks so it was extra special, despite the fog and rain.
Cover Image

Live

It is actually pretty easy to build a mediocre headless React today, i.e. an implementation of React that isn't hooked directly into anything else.

react-reconciler is an official package that lets you hook up React to anything already. That's how both React-DOM and React-Native share a run-time.

Most third-party libraries that use it (like react-three-fiber) follow the same approach. They are basically fully wrapped affairs: each notable Three.js object (mesh, geometry, material, light, ...) will tend to have a matching node in the React tree. Three.js has its own scene tree, like the browser has a DOM, so react-reconciler will sync up the two trees one-to-one.

The libraries need to do this, because the target is a retained in-memory model. It must be mutated in-place, and then re-drawn. But what would it look like to target an imperative API directly, like say 2D Canvas?

You can't just call an imperative API directly in a React component, because the idea of React is to enable minimal updates. There is no guarantee every component that uses your imperative API will actually be re-run as part of an update. So you still need a light-weight reconciler.

Implementing your own back-end to the reconciler is a bit of work, but entirely doable. You build a simple JS DOM, and hook React into that. It doesn't even need to support any of the fancy React features, or legacy web cruft: you can stub it out with no-ops. Then you can make up any <native /> tags you like, with any JS value as a property, and have React reconcile them.

Then if you want to turn something imperative into something declarative, you can render elements with an ordinary render prop like this:

<element render={(context) => {
  context.fillStyle = "blue";
  context.drawRect(/*...*/);
}} />

This code doesn't run immediately, it just captures all the necessary information from the surrounding scope, allowing somebody else to call it. The reconciler will gather these multiple "native" elements into a shallow tree. They can then be traversed and run, to form a little ad-hoc program. In other words, it's an Effect-like model again, just with all the effects neatly arranged and reconciled ahead of time. Compared to a traditional retained library, it's a lot more lightweight. It can re-paint without having to re-render any Components in React.

You can also add synthetic events like in React-DOM. These can be forwarded with conveniences like event.stopPropagation() replicated.

I've used this with great success before. Unfortunately I can't show the results here—maybe in the future—but I do have something else that should demonstrate the same value proposition.

React works hard to synchronize its own tree with a DOM-like tree, but it's just a subset of the tree it already has. If you remove that second tree, what's left? Does that one tree still do something useful by itself?

I wagered that it would and built a version of it. It's pretty much just a straight up re-implementation of React's core pattern, from the ground up. It has some minor tweaks and a lot of omissions, but all the basics of hook-driven React are there. More importantly, it has one extra superpower: it's designed to let you easily collect lambdas. It's still an experiment, but the parts that are there seem to work fine already. It also has tests.

Yeet Reduce

As we saw, a reconciler derives all its interesting properties from its one-way data flow. It makes it so that the tree of mounted components is also the full data dependency graph.

So it seems like a supremely bad idea to break it by introducing arbitrary flow the other way. Nevertheless, it seems clear that we have two very interesting flavors just asking to be combined: expanding a tree downstream to produce nodes in a resumable way, and yielding values back upstream in order to aggregate them.

Previously I observed that trying to use a lambda in a live DFG is equivalent to potentially creating new outputs out of thin air. Changing part of a graph means it may end up having different outputs than before. The trick is then to put the data sinks higher up in the tree, instead of at the leaves. This can be done by overlaying a memoized map-reducer which is only allowed to pass things back in a stateless way.

yeet reduce

The resulting data flow graph is not in fact a two-way tree, which would be a no-no: it would have a cycle between every parent and child. Instead it is a DFG consisting of two independent copies of the same tree, one forwards, one backwards, glued together. Though in reality, the second half is incomplete, as it only needs to include edges and nodes leading back to a reducer.

chain of fibers in the forwards direction turns down and back to yield values in the backwards direction

Thus we can memoize both the normal forward pass of generating nodes and their sinks, as well as the reverse pass of yielding values back to them. It's two passes of DFG, one expanding, one contracting. It amplifies input in the first half by generating more and more nodes. But it will simultaneously install reducers as the second half to gather and compress it back into a collection or a summary.

When we memoize a call in the forward direction, we will also memoize the yield in the other direction. Similarly, when we bust a cache on the near side, we also bust the paired cache on the far side, and keep busting all the way to the end. That's why it's called Yeet Reduce. Well that and yield is a reserved keyword.

when 'yeeting' a value, you throw it into a bin on the far side and knock everything down after it

What's also not obvious is that this process can be repeated: after a reduction pass is complete, we can mount a new fiber that receives the result as input. As such, the data flow graph is not a single expansion and contraction, but rather, many of them, separated by a so-called data fence.

This style of coding is mainly suited for use near the top of an application's data dependency graph, or in a shallow sub-tree, where the number of nodes in play is typically a few dozen. When you have tons of tiny objects instead, you want to rely on data-level parallelism rather than mounting each item individually.

Horse JS

I used to think a generalized solution for memoized data flow would be something crazy and mathematical. The papers I read certainly suggested so, pushing towards the equivalent of automatic differentiation of any code. It would just work. It would not require me to explicitly call memo on and in every single Component. It should not impose weird rules banning control flow. It would certainly not work well with non-reactive code. And so on.

There seemed to be an unbridgeable gap between a DFG and a stack machine. This meant that visual, graph-based coding tools would always be inferior in their ability to elegantly capture Turing-complete programs.

Neither seems to be the case. For one, having to memoize things by hand doesn't feel wrong in the long run. A minimal recomputation doesn't necessarily mean a recomputation that is actually small and fast. It feels correct to make it legible exactly how often things will change in your code, as a substitute for the horrible state transitions of old. Caching isn't always a net plus either, so fully memoized code would just be glacial for real use cases. That's just how the memory vs CPU trade-off falls these days.

That said, declaring dependencies by hand is annoying. You need linter rules for it because even experienced engineers occasionally miss a dep. Making a transpiler do it or adding it into the language seems like a good idea, at least if you could still override it. I also find <JSX> syntax is only convenient for quickly nesting static <Components> inside other <Components>. Normal JS {object} syntax is often more concise, at least when the keys match the names. Once you put a render prop in there, JSX quickly starts looking like Lisp with a hangover.

When your Components are just resources and effects instead of widgets, it feels entirely wrong that you can't just write something like:

live (arg) => {
  let [service, store] = mount [
     Service(...),
     Store(...),
  ];
}

Without any JSX or effect-like wrappers. Here, mount would act somewhat like a reactive version of the classic new operator, with a built-in yield, except for fiber-mounted Components instead of classes.

I also have to admit to being sloppy here. The reason you can think of a React component as an Effect is because its ultimate goal is to create e.g. an HTML DOM. Whatever code you run exists, in theory, mostly to generate that DOM. If you take away that purpose, suddenly you have to be a lot more conscious of whether a piece of code can actually be skipped or not, even if it has all the same inputs as last time.

This isn't actually as simple as merely checking if a piece of code is side-effect free: when you use declarative patterns to interact with stateful code, like a transaction, it is still entirely contextual whether that transaction needs to be repeated, or would be idempotent and can be skipped. That's the downside of trying to graft statelessness onto legacy tech, which also requires some mutable water in your immutable wine.

I did look into writing a Babel parser for a JS/TS dialect, but it turns out the insides are crazy and it takes three days just to make it correctly parse live / mount with the exact same rules as async / await. That's because it's a chain of 8 classes, each monkey patching the previous one's methods, creating a flow that's impractical to trace step by step. Tower of Babel indeed. It's the perfect example to underscore this entire article series with.

It also bothers me that each React hook is actually pretty bad from a garbage collection point of view:

const memoized = useMemo(() => slow(foo), [foo]);

This will allocate both a new dependency array [foo] and a new closure () => slow(foo). Even if nothing has changed and the closure is not called. This is unavoidable if you want this to remain a one-liner JS API. An impractical workaround would be to split up and inline useMemo into into its parts which avoid all GC:

// One useMemo() call
let memoized;
{
  useMemoNext();
  useMemoPushDependency(foo);
  memoized = useMemoSameDependencies() ? useMemoValue() : slow(foo);
}

But a language with a built-in reconciler could actually be quite efficient on the assembly level. Dependencies could e.g. be stored and checked in a double buffered arrangement, alternating the read and write side.

I will say this: React has done an amazing job. It got popular because its Virtual DOM finally made HTML sane to work with again. But what it actually was in the long run, was a Trojan horse for Lisp-like thinking and a move towards Effects.

No-API

So, headless React works pretty much exactly as described. Except, without the generators, because JS generators are stateful and not rewindable/resumable. So for now I have to write my code in the promise.then(…) style instead of using a proper yield.

I tried to validate it by using WebGPU as a test case, building out a basic set of composable components. First I hid the uglier parts of the WebGPU API inside some pure wrappers (the makeFoo(...) calls below) for conciseness. Then I implemented a blinking cube like this:

export const Cube: LiveComponent<CubeProps> = memo((fiber) => (props) => {
  const {
    device, colorStates, depthStencilState,
    defs, uniforms, compileGLSL
  } = props;

  // Blink state, flips every second
  const [blink, setBlink] = useState(0);
  useResource((dispose) => {
    const timer = setInterval(() => {
      setBlink(b => 1 - b);
    }, 1000);
    dispose(() => clearInterval(timer));
  });

  // Cube vertex data
  const cube = useOne(makeCube);
  const vertexBuffers = useMemo(() =>
    makeVertexBuffers(device, cube.vertices), [device]);

  // Rendering pipeline
  const pipeline = useMemo(() => {
    const pipelineDesc: GPURenderPipelineDescriptor = {
      primitive: {
        topology: "triangle-list",
        cullMode: "back",
      },
      vertex: makeShaderStage(
        device,
        makeShader(compileGLSL(vertexShader, 'vertex')),
        {buffers: cube.attributes}
      ),
      fragment: makeShaderStage(
        device,
        makeShader(compileGLSL(fragmentShader, 'fragment')),
        {targets: colorStates}
      ),
      depthStencil: depthStencilState,
    };
    return device.createRenderPipeline(pipelineDesc);
  }, [device, colorStates, depthStencilState]);

  // Uniforms
  const [uniformBuffer, uniformPipe, uniformBindGroup] = useMemo(() => {
    const uniformPipe = makeUniforms(defs);
    const uniformBuffer = makeUniformBuffer(device, uniformPipe.data);
    const entries = makeUniformBindings([{resource: {buffer: uniformBuffer}}]);
    const uniformBindGroup = device.createBindGroup({
      layout: pipeline.getBindGroupLayout(0),
      entries,
    });
    return ([uniformBuffer, uniformPipe, uniformBindGroup]
         as [GPUBuffer, UniformDefinition, GPUBindGroup]);
  }, [device, defs, pipeline]);

  // Return a lambda back to parent(s)
  return yeet((passEncoder: GPURenderPassEncoder) => {
    // Draw call
    uniformPipe.fill(uniforms);
    uploadBuffer(device, uniformBuffer, uniformPipe.data);

    passEncoder.setPipeline(pipeline);
    passEncoder.setBindGroup(0, uniformBindGroup);
    passEncoder.setVertexBuffer(0, vertexBuffers[0]);
    passEncoder.draw(cube.count, 1, 0, 0);
  });
}

This is 1 top-level function, with zero control flow, and a few hooks. The cube has a state (blink), that it decides to change on a timer. Here, useResource is like a sync useEffect which the runtime will manage for us. It's not pure, but very convenient.

All the external dependencies are hooked up, using the react-like useMemo hook and its mutant little brother useOne (for 0 or 1 dependency). This means if the WebGPU device were to change, every variable that depends on it will be re-created on the next render. The parts that do not (e.g. the raw cube data) will be reused.

This by itself is remarkable to me: to be able to granularly bust caches like this deep inside a program, written in purely imperative JS, that nevertheless is almost a pure declaration of intent. When you write code like this, you focus purely on construction, not on mutation. It also lets you use an imperative API directly, which is why I refer to this as "No API": the only wrappers are those which you want to add yourself.

Notice the part at the end: I'm not actually yeeting a real draw command. I'm just yeeting a lambda that will insert a draw command into a vanilla passEncoder from the WebGPU API. It's these lambdas which are reduced together in this sub-tree. These can then just be run in tree order to produce the associated render pass.

What's more, the only part of the entire draw call that actually changes regularly is the GPU uniform values. This is why uniforms is not an immutable object, but rather an immutable reference with mutable registers inside. In react-speak it's a ref, aka a pointer. This means if only the camera moves, the Cube component does not need to be re-evaluated. No lambda is re-yeeted, and nothing is re-reduced. The same code from before would keep working.

Therefor the entirety of Cube() is wrapped in a memo(...). It memoizes the entire Component in one go using all the values in props as the dependencies. If none of them changed, no need to do anything, because it cannot have any effect by construction. The run-time takes advantage of this by not re-evaluating any children of a successfully memoized node, unless its internal state changed.

The very top of the (reactive) part is:

export const App: LiveComponent<AppProps> = () => (props) => {
  const {canvas, device, adapter, compileGLSL} = props;

  return use(AutoCanvas)({
    canvas, device, adapter,
    render: (renderContext: CanvasRenderingContextGPU) => {

      const {
        width, height, gpuContext,
        colorStates, colorAttachments,
        depthStencilState, depthStencilAttachment,
      } = renderContext;

      return use(OrbitControls)({
        canvas,
        render: (radius: number, phi: number, theta: number) =>

          use(OrbitCamera)({
            canvas, width, height,
            radius, phi, theta,
            render: (defs: UniformAttribute[], uniforms: ViewUniforms) =>

              use(Draw)({
                device, gpuContext, colorAttachments,
                children: [

                  use(Pass)({
                    device, colorAttachments, depthStencilAttachment,
                    children: [

                      use(Cube)({device, colorStates, depthStencilState, compileGLSL, defs, uniforms}),

                    ]
                  })

                ],
              })
          })
      });
    }
  });
};

This is a poor man's JSX, but also not actually terrible. It may not look like much, but, pretty much everyone who's coded any GL, Vulkan, etc. has written a variation of this.

This tree composes things that are completely heterogeneous: a canvas auto-sizer, interactive controls, camera uniforms, frame buffer attachments, and more, into one neat, declarative structure. This is quite normal in React-land these days. The example above is static to keep things simple, but it doesn't need to be, that's the point.

The nicest part is that unlike in a traditional GPU renderer, it is trivial for it to know exactly when to re-paint the image or not. Even those mutable uniforms come from a Live component, the effects of which are tracked and reconciled: OrbitCamera takes mutable values and produces an immutable container ViewUniforms.

You get perfect battery-efficient sparse updates for free. It's actually more work to get it to render at a constant 60 fps, because for that you need the ability to independently re-evaluate a subtree during a requestAnimationFrame(). I had to explicitly add that to the run-time. It's around 1100 lines now, which I'm happy with.

Save The Environment

If it still seems annoying to have to pass variables like device into everything, there's the usual solution: context providers, aka environments, which act as invisible skip links across the tree:

export const GPUDeviceContext = makeContext();
export const App: LiveComponent<AppProps> = () => (props) => {
  const {canvas, device, adapter, compileGLSL} = props;

  return provide(GPUDeviceContext, device,
    use(AutoCanvas)({ /*...*/ })
  );
}

export const Cube: LiveComponent<CubeProps> = memo((fiber) => (props) => {
  const device = useContext(GPUDeviceContext);
  /* ... */
}

You also don't need to pass one variable at a time, you can pass arbitrary structs.

In this situation it is trickier for the run-time to track changes, because you may need to skip past a memo(…) parent that didn't change. But doable.

Yeet-reduce is also a generalization of the chunking and clustering processes of a modern compute-driven renderer. That's where I got it from anyway. Once you move that out, and make it a native op on the run-time, magic seems to happen.

This is remarkable to me because it shows you how you can wrap, componentize and memoize a completely foreign, non-reactive API, while making it sing and dance. You don't actually have to wrap and mount a <WebGPUThingComponent> for every WebGPUThing that exists, which is the popular thing to do. You don't need to do O(N) work to control the behavior of N foreign concepts. You just wrap the things that make your code more readable. The main thing something like React provides is a universal power tool for turning things off and on again: expansion, memoization and reconciliation of effects. Now you no longer need to import React and pretend to be playing DOM-jot either.

The only parts of the WebGPU API that I needed to build components for to pull this off, were the parts I actually wanted to compose things with. This glue is so minimal it may as well not be there: each of AutoSize, Canvas, Cube, Draw, OrbitCamera, OrbitControls and Pass is 1 reactive function with some hooks inside, most of them half a screen.

I do make use of some non-reactive WebGPU glue, e.g. to define and fill binary arrays with structured attributes. Those parts are unremarkable, but you gotta do it.

If I now generalize my Cube to a generic Mesh, I have the basic foundation of a fully declarative and incremental WebGPU toolkit, without any OO. The core components look the same as the ones you'd actually build for yourself on the outside. Its only selling point is a supernatural ability to get out of your way, which it learnt mainly from React. It doesn't do anything else. It's great when used to construct the outside of your program, i.e. the part that loads resources, creates workloads, spawns kernels, and so on. You can use yeet-reduce on the inside to collect lambdas for the more finicky stuff, and then hand the rest of the work off to traditional optimized code or a GPU. It doesn't need to solve all your problems, or even know what they are.

I should probably reiterate: this is not a substitute for typical data-level parallelism, where all your data is of the exact same type. Instead it's meant for composing highly heterogeneous things. You will still want to call out to more optimized code inside to do the heavy lifting. It's just a lot more straightforward to route.

For some reason, it is incredibly difficult to get this across. Yet algorithmically there is nothing here that hasn't been tried before. The main trick is just engineering these things from the point of view of the person who actually has to use it: give them the same tools you'd use on the inside. Don't force them to go through an interface if there doesn't need to be one.

The same can be said for React and Live, naturally. If you want to get nerdy about it, the reconciler can itself be modeled as a live effect. Its actions can themselves become regular nodes in the tree. If there were an actual dialect with a real live keyword, and WeakMaps on steroids, that would probably be doable. In the current implementation, it would just slow things down.

Throughout this series, I've used Javascript syntax as a lingua franca. Some might think it's insane to stretch the language to this point, when more powerful languages exist where effects fit more natively into the syntax and the runtime. I think it's better to provide stepping stones to actually get from here to there first.

I know that once you have gone through the trouble of building O(N2) lines of code for something, and they work, the prospect of rewriting all of them can seem totally insane. It probably won't be as optimized on the micro-level, which in some domains does actually still matter, even in this day and age. But how big is that N? It may actually be completely worth it, and it may not take remotely as long as you think.

As for me, all I had to do was completely change the way I structure all my code, and now I can finally start making proper diagrams.

Source code on GitLab.