Planet Grep

Planet'ing Belgian FLOSS people

Planet Grep is maintained by Wouter Verhelst. All times are in UTC.

August 23, 2019

Knowledge.z

“Sorry but if sincerely defending your beliefs is clearly and easily distinguishable from trolling, your beliefs are basic.”


🍌

Education is the art of conveying a sense of truth by telling a series of decreasing lies.

Stories are templates for ideas our brains most easily absorb. Shape your idea into a classic story, and it will go right in, even if it's made up.

Performance is a natural part of communication. Embrace it and you can become a superhero to others, as long as you don't actually believe it.

Truly understanding people means understanding all their little vanities, self-deceptions and flaws. Start with your own.

Only you can truly motivate yourself. Find your goal and never stop working towards it. It may take 20 years.

Dare to dream and push on, even through personal hell. Sometimes it takes a friend or a crowd. Sometimes the only way out is through.

Don't just think outside the box. Learn to exit the world around it, and observe it from all angles simultaneously. Choose the most optimal move for all possible games you find yourself in.

Always do it with style, even if you have to copy someone else first. Practice.

Social graphs pass along proof-of-work. You can win by producing it and racing the crowd. The people who validate it have competing and often low or wrong standards.

Social harmony depends on an accurate simulation of the other party. People who see a gift of information will continue to grow. People who see threats in every message will never feel safe.

Rituals and tradition are the deep protocols of society, supporting its glacial macro-evolution. We retain and perform them in obfuscated form, often mistaking our lack of understanding for obsolescence.

Foundational myths tell us to learn from the past and avoid it, that we know who's good or evil, what's virtuous or sinful. Yet every generation creates demons from its own ranks, rarely the same as the last.

Don't teach them to look for swastikas, only the jackboot stomping on a face, and mind your own step.

Ghosts are real, they are called egregores and live inside groups of people. We can reverse engineer them to figure out how they work.

Civilization is the art of compressing lessons and erasing mistakes, to fit increasing knowledge into a fixed biological substrate. Decay is the opposite, when mistakes are compressed and lessons are erased.

winzip trial

I published the following diary on isc.sans.edu: “Simple Mimikatz & RDPWrapper Dropper“:

Let’s review a malware sample that I spotted a few days ago. I found it interesting because it’s not using deep techniques to infect its victims. The initial sample is a malicious VBScript. For a few weeks, I started to hunt for more Powershell based on encoded directives. The following regular expression matched on the file… [Read more]

[The post [SANS ISC] Simple Mimikatz & RDPWrapper Dropper has been first published on /dev/random]

August 19, 2019

An Easy Tutorial

Graphics programming can be intimidating. It involves a fair amount of math, some low-level code, and it's often hard to debug. Nevertheless I'd like to show you how to do a simple "Hello World" on the GPU. You will see that there is in fact nothing to be afraid of.

Most environments offer you a printf-equivalent and a string type, but that's not how we do things in GPU land. We like the raw stuff, and we work with pixels themselves. So we're going to draw our text to the console directly. I'll show you the general high level flow, and then wrap up some of the details.

a window saying hello world

First, we're going to define our alphabet.

let alphabet = allocate_values(
  &['H', 'E', 'L', 'O', 'W', 'R', 'D', '🌎'],
  AlphabetFormat::Letter);

Next we define our message by encoding it from this alphabet.

let message = allocate_indices(
//  H  E  L  L  O  W  O  R  L  D
  &[0, 1, 2, 2, 3, 4, 3, 5, 2, 6, 7],
  IndexFormat::Uint8);

We'll also need to assemble this alphabet soup into positioned text. Don't worry, I precalculated the horizontal X offsets:

let xs = allocate_values(
  &[0.0, 49.0, 130.0, 195.0, 216.0, 238.0, 328.0, 433.0, 496.0, 537.0, 561.0, 667.0],
  AlphabetFormat::Float);

The font is loaded as glyphs, a map of glyph images:

let glyphs = console.load_glyphs("helvetica.ttf");

We now have everything we need to print it pixel-by-pixel to the top of the console, which we call 'rasterizing':

fn hello_world(
  line: Line,
  message: Vec<Index>,
  alphabet: Vec<Letter>,
  xs: Vec<Float>,
  args: Vec<Argument>,
) {
  // Get glyph library
  let glyphs = args[0];

  // Loop over all the indices in the message
  for i in 0..message.len() {

    // Retrieve the x position for this index.
    let x = xs[i];

    // Retrieve the letter in the alphabet
    let letter = alphabet[message[i]];
    
    // Retrieve the glyph image for this letter
    let glyph  = glyphs[letter];

    // Rasterize it to the line
    rasterize(line, x, glyph.image, glyph.width, glyph.height);
  }
}

rasterize() is provided for you, but if you're curious, this is what it looks like on the inside:

fn rasterize(
  line: Line,
  offset: Float,
  image: Frame,
  width: Int,
  height: Int
) {

  // Iterate over rows and columns
  for y in 0..height {
    for x in 0..width {

      // Get target position
      let tx = x + offset;
      let ty = y;

      // Get image pixel color
      let source = get(image, x, y);

      // Get current target pixel color
      let destination = get(line, tx, ty);

      // Blend source color with destination color
      let blended = blend(source, destination);

      // Save new color to target
      set(target, tx, ty, blended);
    }
  }
};

It's just like blending pixels in Photoshop, with a simple nested rows-and-columns loop.

Okay so I did gloss over an important detail.

The thing is, you can't just call hello_world(...) to run your code. I know it looks like a regular function, just like rasterize(), but it turns out you can only call built-in functions directly. If you want to call one of your own functions, you need to do a little bit extra, because the calling convention is slightly different. I'll just go over the required steps so you can follow along.

First you need to actually access the console you want to print to.

So you create a console instance:

let instance = Console::Instance::new();

and get an adapter from it:

let adapter =
  instance.get_adapter(
    &AdapterDescriptor {
      font_preference: FontPreference::Smooth,
    });

so you can get an actual console:

let console =
  adapter.create_console(
    &ConsoleDescriptor {
      extensions: Extensions {
        subpixel_antialiasing: true,
      },
    });

But this console doesn't actually do anything yet. You need to create an interactive window to put it in:

let events_loop = EventsLoop::new();

let window = WindowBuilder::new()
  .with_dimensions(LogicalSize {
    width: 1280.0, height: 720.0
  })
  .with_title("Console")
  .build(&events_loop).unwrap();

and then make a surface to draw to:

let surface = instance.create_surface();

Now if you want to print more than one line of text, you need to set up a line feed:

let descriptor =
  LineFeedDescriptor {
    usage: LineUsageFlags::OUTPUT_ATTACHMENT,
    format: TextFormat::UTF8,
    width: 120,
    height: 50,
  };

let line_feed = console.create_line_feed(&surface, &descriptor);

let next_line = line_feed.get_next_line();

And if you want emoji, which we do, you need a separate emoji buffer too:

let images =
  console.create_emoji(
    EmojiDescriptor {
      size: Extent2d {
        width: 256,
        height: 256,
      },
      array_size: 1024,
      dimension: ImageDimension::D2,
      format: ImageFormat::RGBA8,
      usage: ImageUsageFlags::OUTPUT_ATTACHMENT,
    });

let emoji_buffer = images.create_default_view();

Okay, we're all set!

Now we just need to encode the call, using a call encoder:

let encoder = console.create_call_encoder();

We begin by describing the special first argument (line), a combo of next_line and the emoji_buffer. We also have to provide some additional flags and parameters:

let call =
  encoder.encode_function_call(
    FunctionCallDescriptor {
      console_attachments: &[
        ConsoleAttachmentDescriptor {
          attachment: &next_line,
          load_op: LoadOp::Clear,
          store_op: StoreOp::Store,
          clear_letter: ' ',
        }
      ],
      emoji_attachment: Some(
        ConsoleEmojiAttachmentDescriptor {
          attachment: &emoji_buffer,
          load_op: LoadOp::Clear,
          store_op: StoreOp::Store,
          clear_color: "rgba(0, 0, 0, 0)",
        })
    });

The message of type Vec<Index> is added using a built-in convention for indices:

call.set_index_buffer(message);

The alphabet: Vec<Letter> and the xs: Vec<Float> can also be directly passed in, because they are accessed 1-to-1 using our indices, as numbered arguments:

call.set_alphabet_buffers(&[
  (&alphabet, 0), (&xs, 1)
]);

However, the glyph images are a bit trickier, as they are a custom keyword argument.

To make this work, we need to create an argument group layout, which describes how we'll pass the arguments to sample our glyph images:

let argument_group_layout =
  console.create_argument_group_layout(
    &ArgumentGroupLayoutDescriptor {
      bindings: &[
        ArgumentGroupLayoutBinding {
            binding: 0,
            visibility: Visibility::PIXEL,
            ty: BindingType::SampledText,
        },
        ArgumentGroupLayoutBinding {
            binding: 1,
            visibility: Visibility::PIXEL,
            ty: BindingType::Sampler,
        },
      ]
    });

We then put it into a larger function call layout, in case we have multiple groups of keyword arguments:

let function_call_layout =
  console.create_function_call_layout(
    FunctionCallLayoutDescriptor {
      argument_group_layouts: &[argument_group_layout],
    });

We also need to create bindings to match this layout, to actually bind our argument values:

let glyph_view = glyphs.create_default_view();

let sampler = console.create_sampler(
  &TextSamplerDescriptor {
    address_mode: AddressMode::ClampToEdge,
    text_filter: FilterMode::TypeHinted,
    hint_clamp: 100.0,
    max_anisotropy: 4,
    compare_function: CompareFunction::Always,
    border_color: BorderColor::TransparentBlack,
  });

let argument_group =
  console.create_argument_group(
    &BindGroupDescriptor {
      layout: argument_group_layout,
      bindings: &[
        Binding {
          binding: 0,
          resource: BindingResource::ImageView(&glyph_view),
        },
        Binding {
          binding: 1,
          resource: BindingResource::Sampler(&sampler),
        },
      ]
    });

And add it to our call:

call.set_argument_group(0, argument_group);

Alright! We're pretty much ready to make the call now. Just one more thing. The function call descriptor.

We need to pass the raw code for hello_world as a string to console.create_code_module, and annotate it with a few extra bits of information:

let function_call =
  console.create_function_call(
    &FunctionCallDescriptor {
      layout: &function_call_layout,
      call_stage: CallStageDescriptor {
        module: console.create_code_module(&hello_world),
        entry_point: "hello_world",
      },
      rasterization_state: RasterizationStateDescriptor {
        emoji_alignment: Alignment::Middle,
        emoji_bias: 0,
        emoji_scale: 1.5,
      },
      text_topology: Topology::Letters,
      console_states: &[
        ConsoleStateDescriptor {
          format: TextFormat::UTF8,
          color: BlendDescriptor {
            src_factor: BlendFactor::SrcAlpha,
            dst_factor: BlendFactor::OneMinusSrcAlpha,
            operation: BlendOperation::Add,
          },
          alpha: BlendDescriptor {
            src_factor: BlendFactor::OneMinusDstAlpha,
            dst_factor: BlendFactor::One,
            operation: BlendOperation::Add,
          },
          write_mask: ColorWriteFlags::ALL,
        },
      ],
      emoji_state: Some(EmojiStateDescriptor {
        format: ImageFormat::RGBA8,
        emoji_enabled: true,
        emoji_variant: CompareFunction::LessEqual,
      }),
      index_format: IndexFormat::Uint8,
      alphabet_buffers: &[
        AlphabetBufferDescriptor {
          stride: 1,
          step_mode: InputStepMode::Letter,
          attributes: AlphabetAttributeDescriptor {
            attribute_index: 0,
            format: AlphabetFormat::Letter,
            offset: 0,
          },
        },
        AlphabetBufferDescriptor {
          stride: 1,
          step_mode: InputStepMode::Letter,
          attributes: AlphabetAttributeDescriptor {
            attribute_index: 1,
            format: AlphabetFormat::Number,
            offset: 0,
          },
        },
      ],
      sample_count: 1,
    });

Which we add to the call:

call.set_function_call(&function_call);

Well, you actually have to do this first, but it was easier to explain it last.

Now all that's left is to submit the encoded command to the console queue, and we're already done:

console
  .get_queue()
  .submit(&[encoder.finish()]);

a black window

Hm.

Damn, and I was going to show you how to make a matrix letter effect as an encore. You can pass a letter_shader to rasterizeWithLetterFX(...). It's easy, takes a couple hundred lines tops, all you have to do is call a function on a GPU.

(All code in this post is real, but certain names and places have been changed to protect the innocent. If you'd like to avoid tedious bureaucracy in your code, why not read about how the web people are trying to tame similar lions?)

12
Objects created
Low code no code

A version of this article was originally published on Devops.com.

Twelve years ago, I wrote a post called Drupal and Eliminating Middlemen. For years, it was one of the most-read pieces on my blog. Later, I followed that up with a blog post called The Assembled Web, which remains one of the most read posts to date.

The point of both blog posts was the same: I believed that the web would move toward a model where non-technical users could assemble their own sites with little to no coding experience of their own.

This idea isn't new; no-code and low-code tools on the web have been on a 25-year long rise, starting with the first web content management systems in the early 1990s. Since then no-code and low-code solutions have had an increasing impact on the web. Examples include:

While this has been a long-run trend, I believe we're only at the beginning.

Trends driving the low-code and no-code movements

According to Forrester Wave: Low-Code Development Platforms for AD&D Professionals, Q1 2019, In our survey of global developers, 23% reported using low-code platforms in 2018, and another 22% planned to do so within a year..

Major market forces driving this trend include a talent shortage among developers, with an estimated one million computer programming jobs expected to remain unfilled by 2020 in the United States alone.

What is more, the developers who are employed are often overloaded with work and struggle with how to prioritize it all. Some of this burden could be removed by low-code and no-code tools.

In addition, the fact that technology has permeated every aspect of our lives — from our smartphones to our smart homes — has driven a desire for more people to become creators. As the founder of Product Hunt Ryan Hoover said in a blog post: As creating things on the internet becomes more accessible, more people will become makers..

But this does not only apply to individuals. Consider this: the typical large organization has to build and maintain hundreds of websites. They need to build, launch and customize these sites in days or weeks, not months. Today and in the future, marketers can embrace no-code and low-code tools to rapidly develop websites.

Abstraction drives innovation

As discussed in my middleman blog post, developers won't go away. Just as the role of the original webmaster (FTP hand-written HTML files, anyone?) has evolved with the advent of web content management systems, the role of web developers is changing with the rise of low-code and no-code tools.

Successful no-code approaches abstract away complexity for web development. This enables less technical people to do things that previously could only be done by developers. And when those abstractions happen, developers often move on to the next area of innovation.

When everyone is a builder, more good things will happen on the web. I was excited about this trend more than 12 years ago, and remain excited today. I'm eager to see the progress no-code and low-code solutions will bring to the web in the next decade.

August 14, 2019

Al vraag ik me af waar de permanente Vlaamse vertegenwoordiging voor de Europese Commissie en Unie zich mee bezig houdt en waarom ze zo weinig van zich laten horen? Dat lijstje prioritaire dossiers is namelijk maar erg mager. Er waren bv. toch handelsovereenkomsten in 2019? Is onze permanente vertegenwoordiging daar dan niet mee bezig geweest? Ik vind er weinig over terug.

Zijn er voor 2020 dan geen handelsovereenkomsten in de maak? Ik dacht het wel. Daar is Vlaanderen niet mee bezig? Dat vindt onze permanente vertegenwoordiging het niet waard om te vermelden op hun website? En zo ja, waar dan?

Zoals ik het begrepen heb spreken al onze diplomaten in Brussel Frans en, veel belangrijker dan dat (want mij part spreken ze het Afrikaans), vertolken ze het Belgische (en eigenlijk ook het Franse) standpunt. Zelden dat van Vlaanderen en nog minder krijgt dit de nodige politieke -en media aandacht.

De NV-A wil vaak doen alsof ze het niveau van de Europese Unie belangrijk vinden. Maar eigenlijk bieden ze maar weinig aan, doen ze er maar weinig mee. En daardoor zijn we als Vlaanderen helemaal niet of maar mager vertegenwoordigd.

Ik had graag in Bart De Wever’s nota wat meer duidelijk gehad in plaats van enkel een Latijnse spreuk. Spreuken en getwitter is geen beleid voeren, Bart. Het wordt tijd dat uw partij i.p.v. enkel grote uitspraken uit te kramen en het populisme aan te wakkeren ook eens toont dat het kan besturen.

Ja ja. We gaan onafhankelijk worden in Europa en de Europese Unie. Dat is duidelijk. Maar hoe gaat de NV-A onze permanente vertegenwoordiging van Vlaanderen voor de EU dan versterken?

Een onafhankelijk Vlaanderen moet niet verwachten dat Belgische diplomaten Vlaamse kastanjes uit het vuur gaan halen.

Wees concreet.

August 13, 2019

The post Time for Change: Going Independent appeared first on ma.ttias.be.

After 12 intense years at Nucleus, it's time for something new: as of September 2019 I'll stop my activities at Nucleus and continue to work as an independent, focussing on Oh Dear!, DNS Spy & Syscast.

The road to change

Why change? Why give up a steady income, health- & hospital insurance, a company car, paid holidays, fun colleagues, exciting tech challenges, ... ?

I think it's best explained by showing what an average day looked like in 2016-2017, at the peak of building DNS Spy.

 

 

Back when I had the idea to create a DNS monitoring service, the only way I could make it work was to code on it at crazy hours. Before the kids woke up and after they went to bed. Before and after the more-than-full-time-job.

This worked for a surprisingly long time, but eventually I had to drop the morning hours and get some more sleep in.

Because of my responsibilities at Nucleus (for context: a 24/7 managed hosting provider), I was often woken during the night for troubleshooting/interventions. This, on top of the early hours, made it impossible to keep up.

After a while, the new rhythm became similar, but without the morning routine.

 

 

Notice anything missing in that schedule? Household chores? Some quality family time? Some personal me-time to relax? Yeah, that wasn't really there.

There comes a point where you have to make a choice: either continue on this path and end up wealthy (probably) but without a family, or choose to prioritize the family first.

As of September 2019, I'll focus on a whole new time schedule instead.

 

 

A radical (at least for me) change of plans, where less time is spent working, more time is spent with the kids, my wife, the cats, the garden, ...

I'm even introducing a bit of whatever-the-fuck-i-want-time in there!

What I'll be working on

In a way I'm lucky.

I'm lucky that I spent the previous 10+ years working like a madman, building profitable side businesses and making a name for myself in both the open source/linux and PHP development world. It allows me to enter September 2019 without a job, but with a reasonable assurance that I'll make enough money to support my family.

How?

For starters, I'll have more time & energy to further build on DNS Spy & Oh Dear!. These 2 side businesses will from now on be called "businesses", as they'll be my main source of income. It isn't enough to live on, mind you, so there's work to be done. But at least there's something there to build on.

Next to that, my current plan is to revive and start building on Syscast. The idea formed in 2016 (the "workaholic" phase, pre-DNS Spy) and was actually pretty fleshed out already. Making online courses, building upon the 10+ years of sysadmin & developer knowledge.

Syscast didn't happen in 2016 and pivoted to a podcast that featured impressive names like Daniel Stenberg (curl & libcurl), Seth Vargo (Hashicorp Vault), Matt Holt (Caddy) and many others instead.

I've always enjoyed giving presentations, explaining complicated technologies in easy terms and guiding people to learn new things. Syscast fits that bill and would make for a logical project to work on.

Looking back at an amazing time

A change like this isn't taken lightly. Believe me when I say I've been debating this for some time.

I'm grateful to both founders of Nucleus, Wouter & David, that they've given me a chance in 2007. I dropped out of college, no degree, just a positive attitude and some rookie PHP knowledge. I stumbled upon the job by accident, just googling for a PHP job. Back then, there weren't that many. It was either Nucleus or a career writing PHP for a bank. I think this is where I got lucky.

I've learned to write PHP, manage Linux & Windows servers, do customer support, how to do marketing, keep basic accounting and the value of underpromise and overdeliver. I'll be forever grateful to both of them for the opportunity and the lessons learned.

It was also an opportunity to work with my best friend, Jan, for the last 9 years. Next to existing friends, I'm proud to call many of my colleagues friends too and I hope we can stay in touch over the years. I find relationships form especially tight in intense jobs, when you heavily rely on each other to get the job done.

Open to new challenges

In true LinkedIn parlanceI'm open to new challenges. That might be a couple of days of consultancy on Linux, software architecture, PHP troubleshooting, scalability advice, a Varnish training, ...

I'm not looking for a full-time role anywhere (see the time tables above), but if there's an interesting challenge to work on, I'll definitely consider it. After all, there are mouths to feed at home. ;-)

If you want to chat, have a coffee, exchange ideas, brainstorm or revolutionize the next set of electric cars, feel free to reach out (my contact page has all the details).

But first, a break

However, before I can start doing any of that, I need a time-out.

In September, my kids will go to school and things will be a bit more quiet around the house. After living in a 24/7 work-phase for the last 10 years, I need to cool down first. Maybe I'll work on the businesses, maybe I won't. I have no idea how hard that hammer will hit come September when I suddenly have my hands free.

Maybe I'll even do something entirely different. Either way, I'll have more time to think about it.

The post Time for Change: Going Independent appeared first on ma.ttias.be.

August 12, 2019

A special bird flying in space has the spotlight while lots of identical birds sit on the ground (lack of diversity)

At Drupalcon Seattle, I spoke about some of the challenges Open Source communities like Drupal often have with increasing contributor diversity. We want our contributor base to look like everyone in the world who uses Drupal's technology on the internet, and unfortunately, that is not quite the reality today.

One way to step up is to help more people from underrepresented groups speak at Drupal conferences and workshops. Seeing and hearing from a more diverse group of people can inspire new contributors from all races, ethnicities, gender identities, geographies, religious groups, and more.

To help with this effort, the Drupal Diversity and Inclusion group is hosting a speaker diversity training workshop on September 21 and 28 with Jill Binder, whose expertise has also driven major speaker diversity improvements within the WordPress community.

I'd encourage you to either sign up for this session yourself or send the information to someone in a marginalized group who has knowledge to share, but may be hesitant to speak up. Helping someone see that their expertise is valuable is the kind of support we need to drive meaningful change.

We now invite proposals for main track presentations, developer rooms, stands and lightning talks. FOSDEM offers open source and free software developers a place to meet, share ideas and collaborate. Renowned for being highly developer-oriented, the event brings together some 8000+ geeks from all over the world. The twentieth edition will take place on Saturday 1st and Sunday 2nd February 2020 at the usual location: ULB Campus Solbosch in Brussels. We will record and stream all main tracks, devrooms and lightning talks live. The recordings will be published under the same licence as all FOSDEM content (CC-BY). If, exceptionally,舰

August 10, 2019

FOSDEM 2020 will take place at ULB Campus Solbosch on Saturday 1 and Sunday 2 February 2020. Further details and calls for participation will be announced in the coming days and weeks.

August 09, 2019

MVC was a mistake

A reader sent in the following letter:

Dear Mr. Wittens,

I have read with great interest your recent publications in Computers Illustrated. However, while I managed to follow your general exposition, I had trouble with a few of the details.

Specifically, Googling the phrase "topping from the bottom" revealed results that I can only describe as being of an unchristian nature, and I am unable to further study the specific engineering practice you indicated. I am at a loss in how to apply this principle in my own professional pursuits, where I am currently tasked with building web forms for a REST API using React. Should I have a conversation about this with my wife?

With respect,
Gilbert C.

Dear Gilbert,

In any marriage, successful copulation requires commitment and a mutual respect of boundaries. Unlike spouses however, components are generally not monogamous, and indeed, best results are obtained when they are allowed to mate freely and constantly, even with themselves. They are quite flexible.

The specifics of this style are less important than the general practice, which is about the entity ostensibly in control not actually being in control. This is something many form libraries get wrong, React or not, missing the true holy trinity. I will attempt to explain. This may take a while. Grab some hot chocolate.

Simon Stålenhag Painting
Infinite omnipotent hyperbeing lovingly screaming "YOU'RE VALID", its song echoed across every structural detail of the universe—you, an insignificant mote adrift in some galactic tide shouting back "YES, OKAY, WHAT ELSE?"
Fragnemt by ctrlcreep
(a linguistic anti-depressant in book form)

That's Numberwang

In your web forms, you're building a collection of fields, and linking them to a data model. Either something to be displayed, or something to be posted, possibly both. Or even if it's a single-page app and it's all for local consumption only, doesn't really matter.

Well, it does, but that's boundary number 1. A form field shouldn't know or care about the logistics of where its data lives or what it's for. Its only task is to specialize the specific job of instantiating the value and manipulating it.

There's another boundary that we're also all used to: widget type. A text field is not a checkbox is not a multi-select. That's boundary number 3. It's the front of the VCR, the stuff that's there so that we can paw at it.

Would you like to buy a vowel for door number 2?

P_L_C_: What APIs are about.
"Previously, on Acko."

You see, when God divided the universe into controlled (value = x) and uncontrolled (initialValue = defaultValue = x) components, He was Mistaken. A true uncontrolled component would have zero inputs.

What you actually have is a component spooning with an invisible policy. Well, if you can't pull them apart, I guess they're not just spooning. But these two are:

<ValidatingInput
  parse={parseNumber} format={formatNumber}
  value={number}    setValue={setNumber}
>{
  (value, isError, onChange, onBlur) =>
    <TextField
      value={value}      isError={isError}
      onChange={onChange} onBlur={onBlur} />
}</ValidatingInput>

This ValidatingInput ensures that a value can be edited in text form, using your parser and serializer. It provides Commitment, not Control. The C in MVC was misread in the prophecies I'm afraid. Well actually, someone spilled soda on the scrolls, and the result was a series of schisms over what they actually said. Including one ill-fated attempt to replace the Controller with a crazy person.

Point is, as long as the contents of the textfield are parseable, the value is plumbed through to the outside. When it's not, only the text inside the field changes, the inner value. The outside value doesn't, remaining safe from NaNful temptations. But the TextField remains in control, only moderated by the two policies you passed in. The user is free to type in unparseable nonsense while attempting to produce a valid value, rather than having their input messed around with as they type. How does it know it's invalid? If the parser throws an Error. A rare case of exceptions being a good fit, even if it's the one trick VM developers hate.

An "uncontrolled component" as React calls it has a policy to never write back any changes, requiring you to fish them out manually. Its setValue={} prop is bolted shut.

Note our perspective as third party: while we cannot see or touch ValidatingInput's inner workings, it still lets us have complete control over how it actually gets used, both inside and out. You wire it up to an actual widget yourself, and direct its behavior completely. It's definitely not a controller.

This is also why, if you pass in JSON.parse and JSON.stringify as the parser/formatter, with the pretty printer turned on, you get a basic but functional JSON editor for free.

You can also have a DirectInput, to edit text directly in its raw form, without any validation or parsing. Because you wrote that one first, to handle the truly trivial case. But then later you realized it was just ValidatingInput with identity functions glued into the parse/format props.

So let's talk about MVC and see how much is left once you realize the above looks suspiciously like a better, simpler recipe for orthogonalized UI.

Wikipedia - Model-View-Controller

Look at this. It's from Wikipedia so it must be true. No really, look at it closely. There is a View, which only knows how to tell you something. There is a Controller, which only knows how to manipulate something.

This is a crack team of a quadriplegic and a blind man, each in separate rooms, and you have to delegate your UI to them? How, in the name of all that is holy, are you supposed to do that? Prisoner dilemmas and trolleys? No really, if the View is a TextField, with a keyboard cursor, and mouse selection, and scrolling, and Japanese, how is it possible to edit the model backing it unless you are privy to all that same information? What if it's a map or a spreadsheet? ばか!

This implies either one of two things. First, that the Controller has to be specialized for the target widget. It's a TextController, with its own house key to the TextView, not pictured. Or second, that the Controller doesn't do anything useful at all, it's the View that's doing all the heavy lifting, the Controller is just a fancy setter. The TextModel certainly isn't supposed to contain anything View-specific, or it wouldn't be MVC.

Wikipedia does helpfully tell us that "Particular MVC architectures can vary significantly from the traditional description here." but I would argue the description is complete and utter nonsense. Image-googling MVC is however a fun exercise for all the zany diagrams you get back. It's a puzzle to figure out if any two actually describe the same thing.

Model-View-Controller 1 Model-View-Controller 6
Model-View-Controller 2 Model-View-Controller 5
Model-View-Controller 4 Model-View-Controller 3

Nobody can agree on who talks to who, in what order, and who's in charge. For some, the Controller is the one who oversees the entire affair and talks to the user, maybe even creates the Model and View. For others, it's the View that talks to the user, possibly creating a Controller and fetching a Model. For some, the Model is just a data structure, mutable or immutable. For others it's an interactive service to both Controller and View. There may be an implicit 4th observer or dispatcher for each trio. What everyone does agree on, at least, is that the program does things, and those things go in boxes that have either an M, a V or a C stamped on them.

One dimension almost never illustrated, but the most important one, is the scope of the MVC in question.

Widget-MVC

Sometimes MVC is applied at the individual widget level, indeed creating a TextModel and a TextView and a TextController, which sits inside a FormModel/View/Controller, nested inside an ActivityModel/View/Controller. This widget-MVC approach leads to a massively complected tree, where everything comes in triplicate.

The models each contain partial fragments of an original source repeated. Changes are sometimes synced back manually, easily leading to bugs. Sometimes they are bound for you, but must still travel from a deeply nested child all the way back through its parents. If a View can't simply modify its Model without explicitly tripping a bunch of other unrelated Model/Controller/View combos, to and fro, this is probably an architectural mistake. It's a built-in game of telephone played by robots.

I didn't try to draw all this, because then I'd also have to define which of the three has custody of the children, which is a very good question.

Rails-MVC

But sometimes people mean the exact opposite, like client/server MVC. There the Model is a whole database, the Controller is your web application server and the View is a templated client-side app. Maybe your REST API is made up of orthogonal Controllers, and your Models are just the rows in the tables pushed through an ORM, with bells on and the naughty bits censored. While this explanation ostensibly fits, it's not accurate. A basic requirement for MVC is that the View always represents the current state of the Model, even when there are multiple parallel Views of the same thing. REST APIs simply don't do that, their views are not live. Sorry Rails.

It is in fact this last job that is difficult to do right, and which is the problem we should not lose sight of. As ValidatingInput shows, there is definitely a requirement to have some duplication of models (text value vs normalized value). Widgets definitely need to retain some local state and handlers to manage themselves (a controller I guess). But it was never about the boxes, it was always about the arrows. How can we fix the arrows?

(An audience member built like a greek statue stands up and shouts: "Kleisli!" A klaxon sounds and Stephen Fry explains why that's not the right answer.)

The Shape of Things To Come

In my experience, the key to making MVC work well is to focus on one thing only: decoupling the shape of the UI from the shape of the data. I don't just mean templating. Because if there's one thing that's clear in modern apps, it's that widgets want to be free. They can live in overlays, side drawers, pop-overs and accordions. There is a clear trend towards canvas-like apps, most commonly seen with maps, where the main artifact fills the entire screen. The controls are entirely contextual and overlaid, blocking only minimal space. Widgets also move and transition between containers, like the search field in the corner, which jumps between the map and the collapsible sidebar. The design is and should be user-first. Whether data is local or remote, or belongs to this or that object, is rarely a meaningful distinction for how you display and interact with it.

Google Maps

So you don't want to structure your data like your UI, that would be terrible. It would mean every time you want to move a widget from one panel to another, you have to change your entire data model end-to-end. Btw, if an engineer is unusually resistant to a proposed UI change, this may be why. Oops. But what shape should it have then?

You should structure your data so that GETs/PUTs that follow alike rules can be realized with the same machinery through the same interface. Your policies aren't going to change much, at least not as much as your minimum viable UI. This is the true meaning of full stack, of impedance matching between front-end and back-end. You want to minimize the gap, not build giant Stålenhag transformers on both sides.

Just because structuring your data by UI is bad, doesn't mean every object should be flat. Consider a user object. What structure would reflect the various ways in which information is accessed and modified? Maybe this:

user {
  id,
  avatar,
  email,
  profile {
    name, location, birthday
  },
  // ...
}

There is a nice profile wrapper around the parts a user can edit freely themselves. Those could go into a separately grafted profile table, or not. You have to deal with file uploads and email changes separately anyway, as special cases. Editing a user as a whole is out of the question for non-admins. But when only the profile can be edited, there's no risk of improper change of an email address or URL, and that's simpler to enforce when there's separation. Also remember the password hash example, because you never give it out, and it's silly to remove it every time you fetch a user. Proper separation of your domain models will save you a lot of busywork later, of not having to check the same thing in a dozen different places. There's no one Right Way to do it, rather, this is about cause and effect of your own choices down the line, whatever they are.

There's also an entirely different split to respect. Usually in additional to your domain models, you also have some local state that is never persisted, like UI focus, active tab, playback state, pending edits, ... Some of this information is coupled to domain models, like your users, which makes it tempting to put it all on the same objects. But you should never cross the streams, because they route to different places. Again, the same motivation of not having to strip out parts of your data before you can use it for its intended purpose. You can always merge later.

You can also have an in-between bucket of locally persisted state, e.g. to rescue data and state from an accidentally closed tab. Figuring out how it interacts with the other two requires some thought, but with the boundary in the right place, it's actually feasible to solve it right. Also important to point out: you don't need to have only one global bucket of each type. You can have many.

So if you want to do MVC properly, you should separate your models accordingly and decouple them from the shape of the UI. It's all about letting the data pave the cow paths of the specific domain as naturally as possible. This allows you to manipulate it as uniformly as possible too. You want your Controller to be just a fancy getter/setter, only mediating between Model and View, along the boundaries that exist in the data and not the UI. Don't try to separate reading and writing, that's completely counterproductive. Separate the read-only things from the read-write things.

A mediating Controller is sometimes called a ViewModel in the Model-View-ViewModel model, or more sanely Model-View-Presenter, but the name doesn't matter. The difficult part in the end is still the state transitions on the inside, like when ValidatingInput forcefully reformats the user's text vs when it doesn't. That's what the onBlur was for, if you missed it. In fact, if the View should always reflect the Model, should an outside change overwrite an internal error state? What if it's a multi-user environment? What if you didn't unfocus the textfield before leaving your desk? Falsehoods programmers believe about interaction. Maybe simultaneous props and derived state changes are just unavoidably necessary for properly resolving UI etiquette.

I have another subtle but important tweak. React prefab widgets like TextField usually pass around DOM events. Their onChange and onBlur calls carry a whole Event, but 99% of the time you're just reading out the e.target.value. I imagine the reasoning is that you will want to use the full power of DOM to reach into whatever library you're using and perform sophisticated tricks on top of their widgets by hand. No offense, but they are nuts. Get rid of it, make a TextField that grabs the value from its own events and calls onChange(value) directly. Never write another trivial onChange handler just to unbox an event again.

This:

<TextField ...
  onChange={e => setValue(e.target.value)} />

could be:

<TextField ...
  onChange={setValue} />

If you do need to perform the rare dark magic of simultaneous hand-to-hand combat against every platform and browser's unique native textfield implementation (i.e. progressive enhancement of <input />), please do it yourself and box in the component (or hook) to keep it away from others. Don't try to layer it on top of an existing enhanced widget in a library, or allow others to do the same, it will be a nightmare. Usually it's better to just copy/paste what you can salvage and make a new one, which can safely be all fucky on the inside. Repetition is better than the wrong abstraction, and so is sanity.

Really, have you ever tried to exactly replicate the keyboard/mouse behavior of, say, an Excel table in React-DOM, down to the little details? It's an "interesting" exercise, one which shows just how poorly conceived some of HTML's interactivity truly is. React was victorious over the DOM, but its battle scars are still there. They should be treated and allowed to fade, not wrapped in Synthetic and passed around like precious jewels of old.

Synthesis

What you also shouldn't do is create a special data structure for scaffolding out forms, like this:

[
  {
    type: "textfield",
    label: "Length"
    format: {
      type: "number",
      precision: 2,
      units: "Meters",
    },
  },
  {
    // ...
  },
  // ...
]

(If your form arrays are further nested and bloated, you may have a case of the Drupals and you need to have that looked at, and possibly, penicillin).

You already have a structure like that, it's your React component tree, and that one is incremental, while this one is not. It's also inside out, with the policy as the little spoon. This may seem arbitrary, but it's not.

Now might be a good time to explain that we are in fact undergoing the Hegelian synthesis of Immediate and Retained mode. Don't worry, you already know the second half. This also has a better ending than Mass Effect, because the relays you're going to blow up are no longer necessary.

Immediate Mode is UI coding you have likely never encountered before. It's where you use functions to make widgets (or other things) happen immediately, just by calling them. No objects are retained per widget. This clearly can't work, because how can there be interactivity without some sort of feedback loop? It sounds like the only thing it can make is a dead UI, a read-only mockup. Like this dear-imgui code in Rust:

ui.with_style_var(StyleVar::WindowPadding((0.0, 0.0).into()), || {
  ui.window(im_str!("Offscreen Render"))
    .size((512.0 + 10.0, 512.0 + 40.0), ImGuiCond::Always)
    .scrollable(false)
    .scroll_bar(false)
    .build(|| {

      ui.columns(2, im_str!(""), false);
      for i in 0..4 {
        ui.image(self.offscreen_textures[i], (256.0, 256.0))
          .build();
        ui.next_column();
      }

    })
  });
imgui example

We draw a window, using methods on the ui object, with some ui.image()s inside, in two columns. There are some additional properties set via methods, like the initial size and no scrollbars. The inside of the window is defined using .build(...) with a closure that is evaluated immediately, containing similar draw commands. Unlike React, there is no magic, nothing up our sleeve, this is plain Rust code. Clearly nothing can change, this code runs blind.

But actually, this is a fully interactive, draggable window showing a grid of live image tiles. It'll even remember its position and open/collapsed state across program restarts out of the box. You see, when ui.window is called, its bounding box is determined so that it can be drawn. This same information is then immediately used to see if the mouse cursor is pointing at it and where. If the left button is also down, and you're pointing at the title bar, then you're in the middle of a drag gesture, so we should move the window by whatever the current motion is. This all happens in the code above. If you call it every frame, with correct input to ui, you get a live UI.

imgui demos

So where does the mouse state live? At the top, inside ui. Because you're only ever interacting with one widget at a time, you don't actually need a separate isActive state on every single widget, and no onMouseDown either. You only need a single, global activeWidget of type ID, and a single mouseButtons. At least without multi-touch. If a widget discovers a gesture is happening inside it, and no other widget is active, it makes itself the active one, until you're done. The global state tracks mouse focus, button state, key presses, the works, in one data structure. Widget IDs are generally derived from their position in the stack, in relation to their parent functions.

Every time the UI is re-rendered, any interaction since the last time is immediately interpreted and applied. With some imagination, you can make all the usual widgets work this way. You just have to think a bit differently about the problem, sharing state responsibly with every other actor in the entire UI. If a widget needs to edit a user-supplied value, there's also "data binding," in the form of passing a raw memory pointer, rather than the value directly. The widget can read/write the value any way it likes, without having to know who owns it and where it came from.

imgui function tree

Immediate mode UI is so convenient and efficient because it reuses the raw building blocks of code as its invisible form structure: compiled instructions, the stack, pointers, nested function calls, and closures. The effect is that your UI becomes like Deepak Chopra explaining quantum physics, it does not exist unless you want to look at it, which is the same as running it. I realize this may sound like moon language, but there is a whole lecture explaining it if you'd like to know more. Plus 2005 Casey Muratori was dreamboat.

The actual output of dear-imgui, not visible in this code at all, is raw geometry, produced from scratch every frame and passed out through the back. It's pure unretained data. This is pure vector graphics, directly sampled. It can be rendered using a single GPU shader with a single image for the font. All the shapes are triangle meshes with RGBA colors at their vertices, including the anti-aliasing, which is a 1px wide gradient edge.

Its polar opposite is Retained mode, what you likely have always used, where you instead materialize a complete view tree. Every widget, label and shape is placed and annotated with individual styling and layout. You don't recompute this tree every frame, you only graft and prune it, calling .add(...) and .set(...) and .remove(...). It would seem more efficient and frugal, but in fact you pay for it tenfold in development overhead.

By materializing a mutable tree, you have made it so that now the evolution of your tree state must be done in a fully differential fashion, not just computed in batch. Every change must be expressed in terms of how you get there from a previous state. Given n states, there are potentially O(n2) valid pairwise transitions. Coding them all by hand, for all your individual Ms, Vs and Cs, however they work, is both tedious and error-prone. This is called Object-Oriented Programming.

What you generally want to do instead is evolve your source of truth differentially, and to produce the derived artifact—the UI tree—declaratively. IM UI achieves this, because every function call acts as a declarative statement of the desired result, not a description of how to change the previous state to another. The cost is that your UI is much more tightly coupled with itself, and difficult to extend.

react tree

The React model bridges the two modes. On the one hand, its render model provides the same basic experience as immediate UI, except with async effects and state allocation integrated, allowing for true decoupling. But it does this by automating the tedium, not eliminating it, and still materializes an entire DOM tree in the end, whose changes the browser then has to track itself anyway.

I never said it was a good Hegelian synthesis.

If you've ever tried to interface React with something that isn't reactive, like Three.js, you know how silly it feels to hook up the automatic React lifecycle to whatever manual native add/set/remove methods exist. You're just making a crummier version of React inside itself, lacking the proper boundaries and separation.

deferred tree

But we can make it right. We don't actually need to produce the full manifest and blueprint down to every little nut and bolt, we just need to have a clear enough project plan of what we want. What if the target library was immediate instead of retained, or as close as can be and still be performant, and the only thing you kept materialized inside React was the orchestration of the instructions? That is, instead of materializing a DOM, you materialize an immediate mode program at run-time. This way, you don't need to hard-wrap what's at the back, you can plumb the same interface through to the front.

We don't need to expand React functions to things that aren't functions, we just need to let them stop expanding, into a function in which an immediate mode API is called directly. The semantics of when this function is called will need to be clearly defined with suitable policies, but they exist to empower you, not to limit you. I call this Deferred Mode Rendering (nothing like deferred shading). It may be a solution to the lasagna from hell in which Retained and Immediate mode are stacked on top of each other recursively, each layer more expired than the next.

What this alternate <Component /> tree expands into, in the React model, is placeholder <div />s with render props. The deferred mode layers could still bunch up at the front, but they wouldn't need to fence anything off, they could continue to expose it. Because the context => draw(context) closures you expand to can be assembled from smaller ones, injected into the tree as props by parents towards their children. Somewhat like an algebraically closed reducer.

To do this today requires you to get familiar with the React reconciler and its API, which is sadly underdocumented and a somewhat shaky house of cards. There is a mitigating factor though, just a small one, namely that the entirety of React-DOM and React-Native depend on it. For interactivity you can usually wing it, until you hit the point where you need to take ownership of the event dispatcher. But on the plus side, imagine what your universe would be like if you weren't limited by the irreversible mistakes of the past, like not having to have a 1-to-1 tight coupling between things that are drawn and things that can be interacted with. This need not mean starting from scratch. You can start to explore these questions inside little sandboxes you carve out entirely for yourself, using your existing tools inside your existing environment.

If you'd rather wait for the greybeards to get their act together, there is something else you can do. You can stop breaking your tools with your tools and start fixing them.

The Stateless State

I glossed over boundary 1, where the data comes from and where it goes, but in fact, this is where the real fun stuff happens, which is why I had to save it for last.

The way most apps will do this, is to fetch data from a RESTlike API. This is stored in local component state, or if they're really being fancy, a client-side document store organized by URL or document ID. When mutations need to be made, usually the object in question is rendered into a form, then parsed on submit back into a POST or PUT request. All of this likely orchestrated separately for every mutation, with two separate calls to fetch() each.

If you use the API you already have:

let [state, setState] = useState(initialValue);

Then as we've seen, this useState allocates persistent data that can nevertheless vanish at any time, as can we. That's not good for remote data. But this one would be:

// GET/PUT to REST API
let [state, setState] = useRESTState(cachedValue, url);

It starts from a cached value, if any, fetches the latest version from a URL, and PUTs the result back if you change it.

Now, I know what you're thinking, surely we don't want to synchronize every single keystroke with a server, right? Surely we must wait until an entire form has been filled out with 100% valid content, before it may be saved? After all, MS Word prevented you from saving your homework at all unless you were completely done and had fixed all the typos, right? No wait, several people are typing, and no. Luckily it's not an either/or thing. It's perfectly possible to create a policy boundary between what should be saved elsewhere on submission and what should be manipulated locally:

<Form state={state} setState={setState} validate={validate}>{
  (state, isError, onSubmit) => {
    // ...
  }
}</Form>

It may surprise you that this is just our previous component in drag:

<ValidatingInput value={state} setValue={setState} validate={validate}>{
  (state, isError, onSubmit) => {
    // ...
  }
}</ValidatingInput>

If this doesn't make any sense, remember that it's the widget on the inside that decides when to call the policy's onChange handler. You could wire it up to a Submit button's onClick event instead. Though I'll admit you probably want a Form specialized for this role, with a few extra conveniences for readability's sake. But it would just be a different flavor of the same thing. Notice if onSubmit/onChange takes an Event instead of a direct value it totally ruins it, q.e.d.

In fact, if you want to only update a value when the user unfocuses a field, you could hook up the TextField's onBlur to the policy's onChange, and use it in "uncontrolled" mode, but you probably want to make a BufferedInput instead. Repetition better than the wrong abstraction strikes again.

You might also find these useful, although the second is definitely an extended summer holiday assignment.

// Store in window.localStorage
let [state, setState] = useLocalState(initialValue, url);

// Sync to a magical websocket API
let [state, setState] = useLiveState(initialValue, url);

But wait, there's something missing. If these useState() variants apply at the whole document level, how do you get setters for your individual form fields? What goes between the outer <Shunt /> and the inner <Shunt />?

Well, some cabling:

let [state, setState] = useState({
  metrics: {
    length: 2,
    mass: 10,
  },
  // ...
});

let useCursor = useRefineState(state, setState);
let [length, setLength] = useCursor('metrics', 'length');
let [mass,   setMass]   = useCursor('metrics', 'mass');

What useCursor does is produce an automatic reducer that will overwrite e.g. the state.metrics.length field immutably when you call setLength. A cursor is basically just a specialized read/write pointer. But it's still bound to the root of what it points to and can modify it immutably, even if it's buried inside something else. In React it makes sense to use [value, setter] tuples. That is to say, you don't play a new game of telephone with robots, you just ask the robots to pass you the phone. With a PBX so you only ever dial local numbers.

Full marks are awarded only when complete memoization of the refined setter is achieved. Because you want to pass it directly to some policy+input combo, or a more complex widget, as an immutable prop value on a memoized component.

A Beautiful Bikeshed

Having now thrown all my cards on the table, I imagine the urge to nitpick or outright reject it has reached critical levels for some. Let's play ball.

I'm aware that the presence of string keys for useCursor lookups is an issue, especially for type safety. You are welcome to try and replace them with a compile-time macro that generates the reader and writer for you. The point is to write the lookup only once, in whatever form, instead of separately when you first read and later write. Possibly JS proxies could help out, but really, this is all trying to paper over language defects anyway.

Unlike most Model-View-Catastrophes, the state you manage is all kept at the top, separating the shape of the data from the shape of the UI completely. The 'routes' are only defined in a single place. Unlike Redux, you also don't need to learn a whole new saga, you just need to make your own better versions of the tools you already have. You don't need to centralize religiously. In fact, you will likely want to use both useRESTState and useLocalState in the same component sooner than later, for data and UI state respectively. It's a natural fit. You will want to fetch the remote state at the point in the tree where it makes the most sense, which is likely near but not at its root. This is something Apollo does get right.

In fact, now replace useState(...) with [state, updateState] = useUpdateState(...), which implements a sparse update language, using a built-in universal reducer, and merges it into a root state automatically. If you want to stream your updates as OT/CRDT, this is your chance to make a useCRDTState. Or maybe you just want to pass sparse lambda updates directly to your reducer, because you don't have a client/server gap to worry about, which means you're allowed to do:

updateState({foo: {thing: {$apply: old => new}}})

Though that last update should probably be written as:

let [thing, updateThing] = useCursor('foo', 'thing');
// ...
updateThing($apply(old => new));

useCursor() actually becomes simpler, because now its only real job is to turn a path like ['foo', 'bar'] into the function:

value => ({foo: {bar: value}})

...with all the reduction logic part of the original useUpdateState().

Of course, now it's starting to look like you should be able to pass a customized useState to any other hook that calls useState, so you can reuse it with different backing stores, creating higher-order state hooks:

let useRemoteState = useRESTState(url);
let useRemoteUpdatedState = useUpdateState(initialValue, useRemoteState);

Worth exploring, for sure. Maybe undo/redo and time travel debugging suddenly became simpler as well.

Moving on, the whole reason you had centralized Redux reducers was because you didn't want to put the update logic inside each individual component. I'm telling you to do just that. Yes but this is easily fixed:

updateThing(manipulateThing(thing, ...));

manipulateThing returns an update representing the specific change you requested, in some update schema or language, which updateThing can apply without understanding the semantics of the update. Only the direct effects. You can also build a manipulator with multiple specialized methods if you need more than one kind of update:

updateSelection(
  manipulateSelection(selection)
    .toggleOne(clicked)
);

Instead of dispatching bespoke actions just so you can interpret them on the other side, why not refactor your manipulations into reusable pieces that take and return modified data structures or diffs thereof. Use code as your parts, just like dear-imgui. You compute updates on the spot that pass only the difference on, letting the cursor's setter map it into the root state, and the automatic reducer handle the merge.

In fact, while you could conscientiously implement every single state change as a minimal delta, you don't have to. That is, if you want to reorder some elements in a list, you don't have to frugally encode that as e.g. a $reorder operation which maps old and new array indices. You could have a $splice operation to express it as individual insertions and removals. Or if you don't care at all, the bulkiest encoding would be to replace the entire list with $set.

But if your data is immutable, you can efficiently use element-wise diffing to automatically compress any $set operation into a more minimal list of $splices, or other more generic $ops or $nops. This provides a way to add specialized live sync without having to rewrite every single data structure and state change in your app.

If diffing feels icky, consider that the primary tool you use for development, git, relies on it completely to understand everything you get paid for. If it works there, it can work here. For completeness I should also mention immer, but it lacks cursor-like constructs and does not let you prepare updates without acting on them right away. However, immer.produce could probably be orthogonalized as well.

Maybe you're thinking that the whole reason you had Sagas was because you didn't want to put async logic inside your UI. I hear you.

This is a tougher one. Hopefully it should be clear by now that putting things inside React often has more benefits than not doing so, even if that code is not strictly part of any View. You can offload practically all your bookkeeping to React via incremental evaluation mechanisms. But we have to be honest and acknowledge it does not always belong there. You shouldn't be making REST calls in your UI, you should be asking your Store to give you a Model. But doing this properly requires a reactive approach, so what you probably want is a headless React to build your store with.

Until that exists, you will have to accept some misappropriation of resources, because the trade-off is worth it more often than not. Particularly because you can still refactor it so that at least your store comes in reactive and non-reactive variants, which share significant chunks of code between them. The final form this takes depends on the specific requirements, but I think it would look a lot like a reactive PouchDB tbh, with the JSON swapped out for something else if needed. (Edit: oh, and graph knitting)

To wrap it all up with a bow: one final trick, stupid, but you'll thank me. Often, every field needs a unique <label> with a unique id for accessibility reasons. Or so people think. Actually, you may not know that the entire time, you have been able to wrap the <label> around your <input> without naming either. Because <label> is an interaction policy, not a widget.

<label for="name">Name</label>
<input name="name" type="text" value="" />

Works the same as:

<label>
  Name
  <input type="text" value="" />
</label>

You haven't needed the name attribute (or id) on form fields for a long time now in your SPAs. But if your dependencies still need one, how about you make this work:

let [mass, setMass] = useCursor('metrics', 'mass');
// The setter has a name
// setMass.name == 'state-0-metrics-mass'

<ValidatingInput
  parse={parseNumber} format={formatNumber}
  value={mass}      setValue={setMass}
>{
  // The name is extracted for you
  (name, value, isError, onChange, onBlur) =>
    <TextField
      name={name} value={value} isError={isError}
      onChange={onChange} onBlur={onBlur} />
}</ValidatingInput>

The setter is the part that is bound to the root. If you need a name for that relationship, that's where you can put it.

As an aside "<label> is an interaction policy" is also a hint on how to orthogonalize interactions in a post-HTML universe, but that's a whole 'nother post.

When you've done all this, you can wire up "any" data model to "any" form, while all the logistics are pro forma, but nevertheless immediate, across numerous components.

You define some state by its persistence mechanism. You refine the state into granular values and their associated setters, i.e. cursor tuples. You can pass them to other components, and let change policy wrappers adopt those cursors, separately from the prefab widgets they wrap. You put the events away where you don't have to think about it. Once the reusable pieces are in place, you only write what is actually unique about this. Your hooks and your components declare intent, not change. Actual coding of differential state transitions is limited only to opportunities where this has a real pay off.

It's particularly neat once you realize that cursors don't have to be literal object property lookups: you can also make cursor helpers for e.g. finding an object in some structured collection based on a dynamic path. This can be used e.g. to make a declarative hierarchical editor, where you want to isolate a specific node and its children for closer inspection, like Adobe Illustrator. Maybe make a hook for a dynamically resolved cursor lookup. This is the actual new hotness: the nails you smashed with your <Component /> hammer are now hooks to hang declarative code off of.

Just keep in mind the price you pay for full memoization, probably indefinitely, is that all your hooks must be executed unconditionally. If you want to apply useCursor to a data.list.map() loop, that won't work. But you don't need to invent anything new, think about it. A dynamically invoked hook is simply a hook inside a dynamically rendered component. Your rows might want to individually update live anyway, it's probably the right place to put an incremental boundary.

I'm not saying the above is the holy grail, far from it, what I am saying is that it's unquestionably both simpler and easier, today, for these sorts of problems. And I've tried a few things. It gets out of the way to let me focus on building whatever I want to build in the first place, empowering my code rather than imprisoning it in tedious bureaucracy, especially if there's a client/server gap. It means I actually have a shot at making a real big boy web app, where all the decades-old conveniences work and the latency is what it should be.

It makes a ton more sense than any explanation of MVC I've ever heard, even the ones whose implementation matches their claims. The closest you can describe it in the traditional lingo is Model-Commitments-View, because the controllers don't control. They are policy components mediating the interaction, nested by scope, according to rules you define. The hyphens are cursors, a crucial part usually overlooked, rarely done right.

Good luck with your homework,
Steven Wittens

Previous: The Incremental Machine

August 02, 2019

Let's try and work our way through the oldest problem in computer science, or close to it: cache invalidation and naming things. Starting with the fact that we misnamed it.

In my view, referring to it as "cache invalidation" is somewhat like referring to crime prevention as "death minimization." While the absense of preventable death is indeed a desirable aspect of quality policing, it would suggest a worrying lack of both concern and ambition if that were the extent of a police force's mission statement.

So too should you run away, not walk, from anyone who treats some stale data and a necessary hard-refresh as a minor but unavoidable inconvenience. You see, this person is telling you that expecting to trust your eyes is too much to ask for when it comes to computers. I don't mean correctness or truth value, no, I just mean being able to see what is there so you can make sense of it, right or wrong. They expect you to sanity check every piece of information you might find, and be ready to Shift-Refresh or Alt-F4 if you suspect a glitch in the matrix. It should be pretty obvious this seriously harms the utility of such an environment, for work or pleasure. Every document becomes a potential sticky indicator gauge, requiring you to give it a good whack to make sure it's unstuck.

This should also puzzle you. A discipline whose entire job it is to turn pieces of data into other pieces of data, using precise instructions, is unable to figure out when its own results are no longer valid? This despite having the complete paper trail at our disposal for how each result is produced, in machine readable form.

Why hasn't this been solved by tooling already? We love tools, right? Is it possible? Is it feasible? Which parts?

Adders in the Grass

I'm going to start at the bottom (or at least a bottom) and work my way up and you'll see why. Let's start with a trivial case of a side-effect-free function, integer addition, and anally dissect it:

(a, b) => a + b

The result changes when either a or b do. However, there is a special case: if a and b each change in opposite amounts, the output is unchanged. Here we have a little microcosm of larger issues.

First, it would be perfectly possible to cache this result, and to check whether a or b have changed since the last time. But just computing the sum is faster than two comparisons. You also need permanent extra storage for at least one extra a and b each, and much more if you want a multi-valued cache rather than just a safety valve. Then you need a pruning policy too to keep it from growing.

Second, if you wish to know whether and how the output will actually change, then you must double check. You have to diff the old and the new values, track the resulting deltas through the same computation as the original. So you can compare to the previous result. The requirement that a + b != (a + Δa) + (b + Δb) can then be reduced to Δa != -Δb. Though this is still more actual work.

Addition operator

If this were multiplication instead of a sum, then:

(a, b) => a * b
a * b != (a + Δa) * (b + Δb)

which reduces to:

Δa * b  +  Δb * a  +  Δa * Δb != 0

Here there is a non-linear relationship which involves both values and deltas together. The first two terms depend on one delta and value each, but the last term only kicks in if both inputs change at the same time. This shows how deltas can interfere both constructively and destructively, either triggering or defusing other effects on other inputs. It also implies there are no easy shortcuts to be found in delta-land, because there are many more ways for values and deltas to combine, than just values by themselves.

Multiply operator Multiply operator deltas

In fact you already knew this. Because if you could provide a concise summary of the delta-behavior of a certain class of functions, you'd break a good chunk of the internet and win a few awards:

m => SHA256(m)

The deltas in a hash function don't just get cozy, they have an all-out orgy of computation, the kind they invented 55 gallon drums of lube for.

This is also your first hint of an answer as to y u no tools. What looks well-behaved and cacheable at a small scale may in fact be part of a larger, unpredictable algorithm, which is why trying to automate caching everywhere is generally a non-starter. That is, it is perfectly possible to imagine a sort of God's eye view of a fully incremental piece of code at the opcode level, and to watch as a change in inputs begins the siege of a thousand memoized gates. But it wouldn't be useful because it's far too granular.

This also means caching is not a computer science problem, it is a computer engineering problem. We are fighting against entropy, against the computational chaos we ourselves create. It is not a science, it is damage control.

4-bit adder

On the other hand, we should not give up, because neither + or * are elementary operations in reality. They are abstractions realized as digital circuits. Given a diagram of a binary adder, we could trace the data flow from each input bit, following the potential effects of each delta. But if you know your arithmetic you already know each bit in a sum can only affect itself and the ones to its left.

What's interesting though is that this complicated dance can be entirely ignored, because it serves to realize an abstraction, that of integers. Given integers, we can reason about changes at a different level. By looking at the level of arithmetic, we were able to discover that two specific patterns of matching differences, Δa == -Δb, cancel out, regardless of the specific value of a and b.

In this case, that only gave us counterproductive "optimizations", but that's because we aren't looking at the right level of abstraction yet. The point is abstraction boundaries don't necessarily have to be black boxes like the hash function, they can also be force fields that allow you to contain deltas, or at least, transmute them into more well-behaved ones. So let's climb up, like Bret wants us to.

I Spy With My Little API

For instance, if we look at the maximum operator applied to an entire list of numbers, again a pure function:

xs => xs.reduce(max, -Infinity)

A simple reduce creates a linear dependency between successive elements, with every delta potentially affecting all max() calls after it. However, the output changes more predictably.

If all elements are unique, the result will only change if a new value x + Δx exceeds the previous result (increasing it), or if an old value x was equal to the previous result and its Δx < 0 (decreasing it). Note we don't need to remember which element index it actually was, and we don't need to care about the elements that didn't change either (at least to merely detect a change).

If there are duplicates, things are a bit more complicated. Now there is a multi-delta-term Δa * Δb * ... between each set, which won't trigger unless all of them decrease at the same time. Writing out the full delta equation for the max of a list is more fun than I can handle, but you get the idea, and actually, it doesn't really matter much. If we pretend all elements are unique regardless, we simply trigger the occasional false positive (change falsely detected), but crucially no false negatives (change falsely ignored).

Either way, the sequential nature of the computation is no longer relevant at this level, because max() is associative (and commutative too), and reduce is a higher-order function whose deltas cancel out in convenient ways when you give it that sort of operator.

Max operator deltas

map reduce

Which means we're almost there. Actually dissecting the max operator was still too tedious, too special-cased. But it gives us hints of what to look for. One such winning combo is MapReduce, using the same properties. By mapping each element in isolation, the effects of any change in any input is initially contained, in a way that is independent of the position of an element in a collection. Second, by using an associative reduction operator, this reduction can be done incrementally, as a tree instead of as a flat list. You reduce the list in chunks, and then re-reduce the list of reductions, recursively until you get one result. When some of the items in the list change, only a few chunks are affected, and the overall recomputation along the tree is minimized. The price you pay is to retain all the intermediate reductions in the tree each time.

Map-Reduce is a universal incremental evaluation strategy, which can schedule and execute any pure function of the individual inputs, provided you reduce the result in an algebraically closed fashion. So that exists. Any others?

Well, many sequential/batch processes are universally incrementalizable too. Take for example a lexer, which processes a stream of text and produces tokens. In this case, the input cannot be chunked, it must be traversed start-to-end.

lexer

The lexer tracks its syntax in an internal state machine, while consuming one or more characters to produce zero or more tokens. Conveniently, the lexer tells you everything you need to know through its behavior in consuming and producing. Roughly speaking, as long as you remember the tuple (lexer state, input position, output position) at every step, you can resume lexing at any point, reusing partial output for partially unchanged input. You can also know when to stop re-lexing, namely when the inputs match again and the internal state does too, because the lexer has no other dependencies. Lining up the two is left as an exercise for the reader, but there's a whole thesis if you like. With some minor separation of concerns, the same lexer can be used in batch or incremental mode. They also talk about Self-Versioned Documents and manage to apply the same trick to incremental parse trees, where the dependency inference is a bit trickier, but fundamentally still the same principle.

What's cool here is that while a lexer is still a pure function in terms of its state and its input, there crucially is inversion of control: it decides to call getNextCharacter() and emitToken(...) itself, whenever it wants, and the incremental mechanism is subservient to it on the small scale. Which is another clue, imo. It seems that pure functional programming is in fact neither necessary nor sufficient for successful incrementalism. That's just a very convenient straightjacket in which it's hard to hurt yourself. Rather you need the application of consistent evaluation strategies. Blind incrementalization is exceedingly difficult, because you don't know anything actionable about what a piece of code does with its data a priori, especially when you're trying to remain ignorant of its specific ruleset and state.

As an aside, the resumable-sequential approach also works for map-reduce, where instead of chunking your inputs, you reduce them in-order, but keep track of reduction state at every index. It only makes sense if your reducer is likely to reconverge on the same result despite changes though. It also works for resumable flatmapping of a list (that is, .map(...).flatten()), where you write out a new contiguous array on every change, but copy over any unchanged sections from the last one.

Each is a good example of how you can separate logistics from policy, by reducing the scope and/or providing inversion of control. The effect is not unlike building a personal assistant for your code, who can take notes and fix stuff while you go about your business.

Don't Blink

This has all been about caching, and yet we haven't actually seen a true cache invalidation problem. You see, a cache invalidation problem is when you have a problem knowing when to invalidate a cache. In all the above, this is not a problem. With a pure function, you simply compare the inputs, which are literal values or immutable pointers. The program running the lexer also knows exactly which part of the output is in question, it's everything after the edit, same for the flatmap. There was never a time when a cache became invalid without us having everything right there to trivially verify and refresh it with.

No, these are cache conservation problems. We were actually trying to reduce unnecessary misses in an environment where the baseline default is an easily achievable zero false hits. We tinkered with that at our own expense, hoping to squeeze out some performance.

There is one bit of major handwavium in there: a != b in the real world. When a and b are composite and enormous, i.e. mutable like your mom, or for extra points, async, making that determination gets somewhat trickier. Async means a gap of a client/server nature and you know what that means.

Implied in the statement (a, b) => 🦆 is the fact that you have an a and a b in your hands and we're having duck tonight. If instead you have the name of a store where you can buy some a, or the promise of b to come, then now your computation is bringing a busload of extra deltas to dinner, and btw they'll be late. If a and b have large dependent computations hanging off them, it's your job to take this additional cloud of uncertainty and somehow divine it into a precise, granular determination of what to keep and what to toss out, now, not later.

You don't have an image, you have the URL of an image, and now you need to decide whether the URL will resolve to the same binary blob that's in your local cache. Do they still represent the same piece of data? The cache invalidation problem is that you weren't notified when the source of truth changed. Instead you have to make the call based on the metadata you originally got with the data and hope for the best.

Obviously it's not possible for every browser to maintain long-lived subscriptions to every meme and tiddy it downloaded. But we can brainstorm. The problem is that you took a question that has a mutable answer but you asked it to be immutable. The right answer is "here's the local cache and some refreshments while you wait, ... ah, there's a newer version, here". Protocol. Maybe a short-lived subscription inside the program itself, from the part that wants to show things, subscribing to the part that knows what's in them, until the latter is 100% sure. You just have to make sure the part that wants to show things is re-entrant.

You want to turn your scene graph into a flattened list of drawing commands, but the scene graph is fighting back. The matrices are cursed, they change when you blink, like the statues from Doctor Who. Because you don't want to remove the curse, you ask everyone to write IS DIRTY in chalk on the side of any box they touch, and you clean the marks back off 60 times per second when you iterate over the entire tree and put everything in order.

I joke, but what's actually going on here is subtle enough to be worth teasing apart. The reason you use dirty flags on mutable scene graphs has nothing to do with not knowing when the data changes. You know exactly when the data changes, it's when you set the dirty flag to true. So what gives?

The reason is that when children depend on their parents, changes cascade down. If you react to a change on a node by immediately updating all its children, this means that further updates of those children will trigger redundant refreshes. It's better to wait and gather all the changes, and then apply and refresh from the top down. Mutable or immutable matrix actually has nothing to do with it, it's just that in the latter case, the dirty flag is implicit on the matrix itself, and likely on each scene node too. Push vs pull is also not really a useful distinction, because in order to cleanly pull from the outputs of a partially dirty graph, you have to cascade towards the inputs, and then return (i.e. push) the results back towards the end. The main question is whether you have the necessary information to avoid redundant recomputation in either direction and can manage to hold onto it for the duration.

The dirty flag is really a deferred and debounced line of code. It is read and cleared at the same time in the same way every frame, within the parent/child context of the node that it is set on. It's not data, it's a covert pre-agreed channel for a static continuation. That is to say, you are signaling the janitor who comes by 60 times a second to please clean up after you.

What's interesting about this is that there is nothing particularly unique about scene graphs here. Trees are ubiquitous, as are parent/child dependencies in both directions (inheritance and aggregation). If we reimagine this into its most generic form, then it might be a tree on which dependent changes can be applied in deferred/transactional form, whose derived triggers are re-ordered by dependency, and which are deduplicated or merged to eliminate any redundant refreshes before calling them.

In Case It Wasn't Obvious

So yes, exactly like the way the React runtime can gather multiple setState() calls and re-render affected subtrees. And exactly like how you can pass a reducer function instead of a new state value to it, i.e. a deferred update to be executed at a more opportune and coordinated time.

In fact, remember how in order to properly cache things you have to keep a copy of the old input around, so you can compare it to the new? That's what props and state are, they are the a and the b of a React component.

Δcomponent = Δ(props * state)
           = Δprops * state + Δstate * props + Δprops * Δstate

           = Props changed, state the same (parent changed)
           + State changed, props the same (self/child changed)
           + Props and state both changed  (parent's props/state change
               triggered a child's props/state change)

The third term is rare though, and the React team has been trying to deprecate it for years now.

I prefer to call Components a re-entrant function call in an incremental deferred data flow. I'm going to recap React 101 quickly, because there is a thing that hooks do that needs to be pointed out.

The way you use React nowadays is, you render some component to some native context like an HTML document:

ReactDOM.render(<Component />, context);

The <Component /> in question is just syntactic sugar for a regular function:

let Component = (props) => {
  // Allocate some state and a setter for it
  let [state, setState] = useState(initialValue);

  // Render a child component
  return <OtherComponent foo={props.bar} onChange={e => setState(...)}/>;
  // aka
  return React.createElement(OtherComponent, {foo: props.bar, onChange: e => setState(...)}, null);
};

This function gets called because we passed a <Component /> to React.render(). That's the inversion of control again. In good components, props and state will both be some immutable data. props is feedforward from parents, state is feedback from ourselves and our children, i.e. respectively the exterior input and the interior state.

If we call setState(...), we cause the Component() function to be run again with the same exterior input as before, but with the new interior state available.

The effect of returning <OtherComponent .. /> is to schedule a deferred call to OtherComponent(...). It will get called shortly after. It too can have the same pattern of allocating state and triggering self-refreshes. It can also trigger a refresh of its parent, through the onChange handler we gave it. As the HTML-like syntax suggests, you can also nest these <Elements>, passing a tree of deferred children to a child. Eventually this process stops when components have all been called and expanded into native elements like <div /> instead of React elements.

Either way, we know that OtherComponent(...) will not get called unless we have had a chance to respond to changes first. However if the changes don't concern us, we don't need to be rerun, because the exact same rendered output would be generated, as none of our props or state changed. This incidentally also provides the answer to the question you may not have realized you had: if everything is eventually a function of some Ur-input at the very start, why would anything ever need to be resumed from the middle? Answer: because some of your components want to semi-declaratively self-modify. The outside world shouldn't care. If we do look inside, you are sometimes treated to topping-from-the-bottom, as a render function is passed down to other components, subverting the inversion of control ad-hoc by extending it inward.

So what is it, exactly, that useState() does then that makes these side-effectful functions work? Well it's a just-in-time allocation of persistent storage for a temporary stack frame. That's a mouthful. What I mean is, forget React.

Think of Component as just a function in an execution flow, whose arguments are placed on the stack when called. This stack frame is temporary, created on a call and destroyed as soon as you return. However, this particular invocation of Component is not equally ephemeral, because it represents a specific component that was mounted by React in a particular place in the tree. It has a persistent lifetime for as long as its parent decides to render/call it. So useState lets it anonymously allocate some permanent memory, keyed off its completely ephemeral, unnamed, unreified execution context. This only works because React is always the one who calls these magic reentrant functions. As long as this is true, multiple re-runs of the same code will retain the same local state in each stack frame, provided the code paths did not diverge. If they did, it's just as if you ran those parts from scratch.

What's also interesting is that hooks were first created to reduce the overuse of <Component /> nesting as a universal hammer for the nail of code composition, because much of the components had nothing to do with UI directly. In fact, it may be that UI just provided us with convenient boundaries around things in the form of widgets, which suggestively taught us how to incrementalize them.

This to me signals that React.render() is somewhat misnamed, but its only mistake is a lack of ambition. It should perhaps be React.apply() or React.incremental(). It's a way of calling a deferred piece of code so that it can be re-applied later with different inputs, including from the inside. It computes minimum updates down a dependency tree of other deferred pieces of code with the same power. Right now it's still kind of optimized for handling UI trees, but the general strategy is so successful that variants incorporating other reconciliation topologies will probably work too. Sure, code doesn't look like React UI components, it's a DAG, but we all write code in a sequential form, explicitly ordering statements even when there is no explicit dependency between them, using variable names as the glue.

The incremental strategy that React uses includes something like the resumable-sequential flatmap algorithm, that's what the key attribute for array elements is for, but instead of .map(...).flatten() it's more like an incremental version of let render = (el, props) => recurse(el.render(props)) where recurse is actually a job scheduler.

The tech under the hood that makes this work is the React reconciler. It provides you with the illusion of a persistent, nicely nested stack of props and state, even though it never runs more than small parts of it at a time after the initial render. It even provides a solution for that old bugbear: resource allocation, in the form of the useEffect() hook. It acts like a constructor/destructor pair for one of these persistent stack frames. You initialize any way you like, and you return to React the matching destructor as a closure on the spot, which will be called when you're unmounted. You can also pass along dependencies so it'll be un/remounted when certain props like, I dunno, a texture size and every associated resource descriptor binding need to change.

There's even a neat trick you can do where you use one reconciler as a useEffect() inside another, bridging from one target context (e.g. HTML) into a different one that lives inside (e.g. WebGL). The transition from one to the other is then little more than a footnote in the resulting component tree, despite the fact that execution- and implementation-wise, there is a complete disconnect as only fragments of code are being re-executed sparsely left and right. You can make it sing with judicious use of the useMemo and useCallback hooks, two necessary evils whose main purpose is to let you manually pass in a list of dependencies and save yourself the trouble of doing an equality check. When you want to go mutable, it's also easy to box in a changing value in an unchanging useRef once it's cascaded as much as it's going to. What do you eventually <expand> to? Forget DOMs, why not emit a view tree of render props, i.e. deferred function calls, interfacing natively with whatever you wanted to talk to in the first place, providing the benefits of incremental evaluation while retaining full control.

It's not a huge leap from here to being able to tag any closure as re-entrant and incremental, letting a compiler or runtime handle the busywork, and forget this was ever meant to beat an aging DOM into submission. Maybe that was just the montage where the protagonist trains martial arts in the mountain-top retreat. Know just how cheap O(1) equality checks can be, and how affordable incremental convenience for all but the hottest paths. However, no tool is going to reorganize your data and your code for you, so putting the boundaries in the right place is still up to you.

I have a hunch we could fix a good chunk of GPU programming on the ground with this stuff. Open up composability without manual bureaucracy. You know, like React VR, except with LISP instead of tears when you look inside. Unless you prefer being a sub to Vulkan's dom forever?

Previous: APIs are about Policy
Next: Model-View-Catharsis

August 01, 2019

For the sixth year in a row, Acquia has been recognized as a Leader in the Gartner Magic Quadrant for Web Content Management.

The 2019 Gartner Magic Quadrant for Web Content ManagementAcquia recognized as a leader in the 2019 Gartner Magic Quadrant for Web Content Management.

As I've written before, I believe analyst reports like the Gartner Magic Quadrant are important because they introduce organizations to Acquia and Drupal. As I've put if before If you want to find a good coffee place, you use Yelp. If you want to find a nice hotel in New York, you use TripAdvisor. Similarly, if a CIO or CMO wants to spend $250,000 or more on enterprise software, they often consult an analyst firm like Gartner..

You can read the complete report on Acquia.com. Thank you to everyone who contributed to this result!

Update: Gartner asked me to take down this post, or to update it to follow their citation guidelines. Specifically, Gartner didn't want my commentary clouding their work. I updated this post to remove any personal commentary so my opinion is not blended with their report.

July 25, 2019

A pox on both houses

“The Web is a system, Neo. That system is our enemy. But when you're inside, you look around, what do you see? Full-stack engineers, web developers, JavaScript ninjas. The very minds of the people we are trying to save.

But until we do, these people are still a part of that system and that makes them our enemy. You have to understand, most of these people are not ready to be unplugged. And many of them are so inured, so hopelessly dependent on the system, that they will fight to protect it.

Were you listening to me, Neo? Or were you looking at the widget library in the red dress?"

...

"What are you trying to tell me, that I can dodge unnecessary re-renders?"

"No Neo. I'm trying to tell you that when you're ready, you won't have to."

 

The web is always moving and shaking, or more precisely, shaking off whatever latest fad has turned out to be a mixed blessing after all. Specifically, the latest hotness for many is GraphQL, slowly but surely dethroning King REST. This means changing the way we shove certain data into certain packets. This then requires changing the code responsible for packing and unpacking that data, as well as replacing the entire digital last mile of routing it at both source and destination, despite the fact that all the actual infrastructure in between is unchanged. This is called full stack engineering. Available for hire now.

The expected custom and indeed, regular passtime, is of course to argue for or against, the old or the new. But instead I'd like to tell you why both are completely wrong, for small values of complete. You see, APIs are about policy.

RESTless API

Take your typical RESTful API. I say typical, because an actual Representationally State Transferred API is as common as a unicorn. A client talks to a server by invoking certain methods on URLs over HTTP, let's go with that.

Optimists will take a constructive view. The API is a tool of empowerment. It enables you to do certain things in your program you couldn't do before, and that's why you are importing it as a dependency to maintain. The more methods in the swagger file, the better, that's why it's called swagger.

But instead I propose a subtractive view. The API is a tool of imprisonment. Its purpose is to take tasks that you are perfectly capable of doing yourself, and to separate them from you with bulletproof glass and a shitty telephone handset. One that is usually either too noisy or too quiet, but never just right. Granted, sometimes this is self-inflicted or benign, but rarely both.

This is also why there are almost no real REST APIs. If we consult the book of difficult-to-spot lies, we learn that the primary features of a REST API are Statelessness, Cacheability, Layeredness, Client-Side Injection and a Uniform Interface. Let's check them.

Statelessness means a simple generality. URLs point to blobs, which are GET and PUT atomically. All the necessary information is supplied with the request, and no state is retained other than the contents saved and loaded. Multiple identical GETs and PUTs are idempotent. The DELETE verb is perhaps a PUT of a null value. So far mostly good. The PATCH verb is arguably a stateless partial PUT, and might be idempotent in some implementations, but only if you don't think too much about it. Which means a huge part of what remains are POST requests, the bread and butter of REST APIs, and those aren't stateless or idempotent at all.

Cacheability and layeredness (i.e. HTTP proxies) in turn have both been made mostly irrelevant. The move to HTTPS everywhere means the layering of proxies is more accurately termed a man-in-the-middle attack. That leaves mainly reverse proxying on the server or CDN side. The HTTP Cache-Control headers are also completely backwards in practice. For anything that isn't immutable, the official mechanism for cache invalidation is for a server to make an educated guess when its own data is going to become stale, which it can almost never know. If they guess too late, the client will see stale data. If they guess too soon, the client has to make a remote request before using their local cache, defeating the point. This was designed for a time when transfer time dominated over latency, whereas now we have the opposite problem. Common practice now is actually for the server to tag cacheable URLs with a revision ID, turning a mutable resource at an immutable URL into an immutable resource at a mutable URL.

Client-Side Injection on the other hand, i.e. giving a browser JavaScript to run, is obviously here to stay, but still, no sane REST API makes you interpret JavaScript code to interact with it. That was mostly a thing Rubyists did in their astronautical pursuits to minimize the client/server gap from their point of view. In fact, we have entirely the opposite problem: we all want to pass bits of code to a server, but that's unsafe, so we find various ways of encoding lobotomized chunks of not-code and pretend that's sufficient.

Which leaves us with the need for a uniform interface, a point best addressed with a big belly laugh and more swagger definition file.

Take the most common REST API of them all, and the one nearly everyone gets wrong, /user. User accounts are some of the most locked up objects around, and as a result, this is a prime candidate for breaking all the rules.

The source of truth is usually something like:

ID Email Handle Real Name Password Hash Picture Karma Admin
1 admin@example.com admin John Doe sd8ByTq86ED... s3://bucket/1.jpg 5 true
2 jane@example.com jane Jane Doe j7gREnf63pO... s3://bucket/2.jpg -3 false

But if you GET /user/2, you likely see:

{
  "id": 2,
  "handle": "jane",
  "picture": "s3://bucket/2.jpg"
}

Unless you are Jane Doe, receiving:

{
  "id": 2,
  "email": "jane@example.com",
  "handle": "jane",
  "name": "Jane Doe",
  "picture": "s3://bucket/2.jpg"
}

Unless you are John Doe, the admin, who'll get:

{
  "id": 2,
  "email": "jane@example.com",
  "handle": "jane",
  "name": "Jane Doe",
  "picture": "s3://bucket/2.jpg",
  "karma": -3,
  "admin": false
}

What is supposedly a stateless, idempotent, cacheable, proxiable and uniform operation turns out to be a sparse GET of a database row, differentiated by both the subject and the specific objects being queried, which opaquely determines the specific variant we get back. People say horizontal scaling means treating a million users as if they were one, but did they ever check how true that actually was?

I'm not done yet. These GETs won't even have matching PUTs, because likely the only thing Jane was allowed to do initially was:

POST /user/create
{
  "name": "Jane Doe",
  "email": "jane@example.com",
  "password": "hunter2"
}

Note the subtle differences with the above.

  • She couldn't supply her own picture URL directly, she will have to upload the actual file to S3 through another method. This involves asking the API for one-time permission and details to do so, after which her user record will be updated behind the scenes. Really, the type of picture is not string, it is a bespoke read-only boolean wearing a foreign key outfit.
  • She didn't get to pick her own id either. Its appearance in the GET body is actually entirely redundant, because it's merely humoring you by echoing back the number you gave it in the URL. Which it assigned to you in the first place. It's not part of the data, it's metadata... or rather the URL is. See, unless you put the string /user/ before the id you can't actually do anything with it. id is not even metadata, it's truncated metadata; unless you're crazy enough to have a REST API where IDs are mutable, in which case, stop that.
  • One piece of truth "data," the password hash, actually never appears in either GETs or POSTs. Only the unencoded password, which is shredded as soon as it's received, and never given out. Is the hash also metadata? Or is it the result of policy?

PATCH /user/:id/edit is left as an exercise for the reader, but consider what happens when Jane tries to change her own email address? What about when John tries to change Jane's? Luckily nobody has ever accidentally mass emailed all their customers by running some shell scripts against their own API.

Neither from the perspective of the client, nor that of the server, do we have a /user API that saves and loads user objects. There is no consistent JSON schema for the client—not even among a given single type during a single "stateless" session—nor idempotent whole row updates in the database.

Rather, there is an endpoint which allows you to read/write one or more columns in a row in the user table, according to certain very specific rules per column. This is dependent not just on the field types and values (i.e. data integrity), but on the authentication state (i.e. identity and permission), which comes via an HTTP header and requires extra data and lookups to validate.

If there was no client/server gap, you'd just have data you owned fully and which you could manipulate freely. The effect and purpose of the API is to prevent that from happening, which is why REST is a lie in the real world. The only true REST API is a freeform key/value store. So I guess S3 and CouchDB qualify, but neither's access control or query models are going to win any awards for elegance. When "correctly" locked down, CouchDB too will be a document store that doesn't return the same kind of document contents for different subjects and objects, but it will at least give you a single ID for the true underlying data and its revision. It will even tell you in real-time when it changes, a superb feature, but one that probably should have been built into the application-session-transport-whatever-this-is layer as the SUBSCRIBE HTTP verb.

Couch is the exception though. In the usual case, if you try to cache any of your responses, you usually have too much or too little data, no way of knowing when and how it changed without frequent polling, and no way of reliably identifying let alone ordering particular snapshots. If you try to PUT it back, you may erase missing fields or get an error. Plus, I know your Express server spits out some kind of ETag for you with every response, but, without looking it up, can you tell me specifically what that's derived from and how? Yeah I didn't think so. If that field meant anything to you, you'd be the one supplying it.

If you're still not convinced, you can go through this exercise again but with a fully normalized SQL database. In that case, the /user API implementation reads/writes several tables, and what you have is a facade that allows you to access and modify one or more columns in specific rows in these particular tables, cross referenced by meaningless internal IDs you probably don't see. The rules that govern these changes are fickle and unknowable, because you trigger a specific set of rules through a combination of URL, HTTP headers, POST body, and internal database state. If you're lucky your failed attempts will come back with some notes about how you might try to fix them individually, if not, too bad, computer says no.

For real world apps, it is generally impossible by construction for a client to create and maintain an accurate replica of the data they are supposed to be able to query and share ownership of.

Regressive Web Apps

I can already hear someone say: my REST API is clean! My data models are well-designed! All my endpoints follow the same consistent pattern, all the verbs are used correctly, there is a single source of truth for every piece of data, and all the schemas are always consistent!

So what you're saying is that you wrote or scaffolded the exact same code to handle the exact same handful of verbs for all your different data types, each likely with their own Model(s) and Controller(s), and their own URL namespace, without any substantial behavioral differences between them? And you think this is good?

As an aside, consider how long ago people figured out that password hashes should go in the /etc/shadow file instead of the now misnamed /etc/passwd. This is a one-to-one mapping, the kind of thing database normalization explicitly doesn't want you to split up, with the same repeated "primary keys" in both "tables". This duplication is actually good though, because the OS' user API implements Policy™, and the rules and access patterns for shell information are entirely different from the ones for password hashes.

You see, if APIs are about policy and not empowerment, then it absolutely makes sense to store and access that data in a way that is streamlined to enforce those policies. Because you know exactly what people are and aren't going to be doing with it—if you don't, that's undefined behavior and/or a security hole. This is something most NoSQLers also got wrong, organizing their data not by policy but rather by how it would be indexed or queried, which is not the same thing.

This is also why people continue to write REST APIs, as flawed as they are. The busywork of creating unique, bespoke endpoints incidentally creates a time and place for defining and implementing some kind of rules. It also means you never have to tackle them all at once, consistently, which would be more difficult to pull off (but easier to maintain). The stunted vocabulary of ad-hoc schemas and their ill-defined nouns forces you to harmonize it all by hand before you can shove it into your meticulously typed and normalized database. The superfluous exercise of individually shaving down the square pegs you ordered, to fit the round holes you carved yourself, has incidentally allowed you to systematically check for woodworms.

It has nothing to do with REST or even HTTP verbs. There is no semantic difference between:

PATCH /user/1/edit
{"name": "Jonathan Doe"}

and

UPDATE TABLE users SET name = "Jonathan Doe" WHERE id = 1

The main reason you don't pass SQL to your Rails app is because deciding on a policy for which SQL statements are allowed and which are not is practically impossible. At most you could pattern match on a fixed set of query templates. Which, if you do, would mean effectively using the contents of arbitrary SQL statements as enum values, using the power of SQL to express the absense of SQL. The Aristocrats.

But there is an entirely more practical encoding of sparse updates in {whatever} <flavor /> (of (tree you) prefer).

POST /api/update
{
  "user": {
    "1": {
      "name": {"$set": "Jonathan Doe"}
    }
  }
}

It even comes with free bulk operations.

Validating an operation encoded like this is actually entirely feasible. First you validate the access policy of the individual objects and properties being modified, according to a defined policy schema. Then you check if any new values are references to other protected objects or forbidden values. Finally you opportunistically merge the update, and check the result for any data integrity violations, before committing it.

You've been doing this all along in your REST API endpoints, you just did it with bespoke code instead of declarative functional schemas and lambdas, like a chump.

If the acronyms CRDT and OT don't mean anything to you, this is also your cue to google them so you can start to imagine a very different world. One where your sparse updates can be undone or rebased like git commits in realtime, letting users resolve any conflicts among themselves as they occur, despite latency. It's one where the idea of remote web apps being comparable to native local apps is actually true instead of a lie an entire industry has shamelessly agreed to tell itself.

You might also want to think about how easy it would be to make a universal reducer for said updates on the client side too, obviating all those Redux actions you typed out. How you could use the composition of closures during the UI rendering process to make memoized update handlers, which produce sparse updates automatically to match your arbitrary descent into your data structures. That is, react-cursor and its ancestors except maybe reduced to two and a half hooks and some change, with all the same time travel. Have you ever built a non-trivial web app that had undo/redo functionality that actually worked? Have you ever used a native app that didn't have this basic affordance?

It's entirely within your reach.

GraftQL

If you haven't been paying attention, you might think GraphQL answers a lot of these troubles. Isn't GraphQL just like passing an arbitrary SELECT query to the server? Except in a query language that is recursive, typed, composable, and all that? And doesn't GraphQL have typed mutations too, allowing for better write operations?

Well, no.

Let's start with the elephant in the room. GraphQL was made by Facebook. That Facebook. They're the same people who made the wildly successful React, but here's the key difference: you probably have the same front-end concerns as Facebook, but you do not have the same back-end concerns.

The value proposition here is of using a query language designed for a platform that boxes its 2+ billion users in, feeds them extremely precise selections from an astronomical trove of continuously harvested data, and only allows them to interact by throwing small pebbles into the relentless stream in the hope they make some ripples.

That is, it's a query language that is very good at letting you traverse an enormous graph while verifying all traversals, but it was mainly a tool of necessity. It lets them pick and choose what to query, because letting Facebook's servers tell you everything they know about the people you're looking at would saturate your line. Not to mention they don't want you to keep any of this data, you're not allowed to take it home. All that redundant querying over time has to be minimized and overseen somehow.

One problem Facebook didn't have though was to avoid busywork, that's what junior hires are for, and hence GraphQL mutations are just POST requests with a superficial layer of typed paint. The Graph part of the QL is only for reading, which few people actually had real issues with, seeing as GET was the one verb of REST that worked the most as advertised.

Retaining a local copy of all visible data is impractical and undesirable for Facebook's purposes, but should it be impractical for your app? Or could it actually be extremely convenient, provided you got there via technical choices and a data model adapted to your situation? In order to do that, you cannot be fetching arbitrary sparse views of unlabelled data, you need to sync subgraphs reliably both ways. If the policy boundaries don't match the data's own, that becomes a herculean task.

What's particularly poignant is that the actual realization of a GraphQL back-end in the wild is typically done by... hooking it up to an SQL database and grafting all the records together. You recursively query this decidedly non-graph relational database, which has now grown JSON columns and other mutations. Different peg, same hole, but the peg shaving machine is now a Boston Dynamics robot with a cute little dog called Apollo and they do some neat tricks together. It's just an act though, you're not supposed to participate.

Don't get me wrong, I know there are real benefits around GraphQL typing and tooling, but you do have to realize that most of this serves to scaffold out busywork, not eliminate it fully, while leaving the INSERT/UPDATE/DELETE side of things mostly unaddressed. You're expected to keep treating your users like robots that should only bleep the words GET and POST, instead of just looking at the thing and touching the thing directly, preferably in group, tolerant to both error and lag.

 

This is IMO the real development and innovation bottleneck in practical client/server application architecture, the thing that makes so many web apps still feel like web apps instead of native apps, even if it's Electron. It makes any requirement of an offline mode a non-trivial investment rather than a sane default for any developer. The effect is also felt by the user, as an inability to freely engage with the data. You are only allowed to siphon it out at low volume, applying changes only after submitting a permission slip in triplicate and getting a stamped receipt. Bureaucracy is a necessary evil, but it should only ever be applied at minimum viable levels, not turned into an industry tradition.

The exceptions are rare, always significant feats of smart engineering, and unmistakeable on sight. It's whenever someone has successfully managed to separate the logistics of the API from its policies, without falling into the trap of making a one-size-fits-all tool that doesn't fit by design.

Can we start trying to democratize that? It would be a good policy.

Next: The Incremental Machine

A swagger definition file. Height: 108,409px

Height: 108,409px

A swagger definition file. Height: 108,409px

Height: 108,409px

Saint-Epaulard de Dortong releva ses bras pelleteuses en émettant une onde de jubilation.
— Une décharge ! Nous avons retrouvé une décharge !

Silencieusement, van Kikenbranouf 15b s’approcha sur ses chenilles, tirant derrière lui une série de troncs d’arbres arrachés par une récente tempête.
— Est-ce un tel plaisir de fouiller des ordures vieilles de plusieurs siècles, demanda-t-il ?
— Oui ! Tu n’imagines pas la quantité d’information qu’on peut tirer d’une décharge. Vu la quantité de déchets prérobotiques, je vais pouvoir mettre à l’épreuve ma théorie de déchiffrement binaire de leurs données. Je sens que ce site sera très vite considéré comme une découverte majeure dans l’histoire de l’archéologie prérobotique.
— Cela ne va pas plaire aux adorateurs de Gook.
— Ces robots créationnistes mal dégrossis ? Ils sont une insulte à l’intelligence électronique, un bug de l’évolution.
— Ils ont néanmoins de plus en plus de puissance de calcul commune et sont non négligeables sur le réseau. Sons compter qu’ils sont irréprochables dans leurs activités écologiques.
— Il n’empêche que leur soi-disant théorie est complètement absurde. C’est de la superstition de bas étage tout juste bonne à faire fonctionner les machines non pensantes.
— Ils ont foi en Gook…
— La foi ? C’est un terme qui ne devrait pas exister dans le vocabulaire robotique. Comme si le fait de croire quelque chose avait la moindre valeur sur les faits.
— Pourtant, tu crois aussi des choses.
— Non, je bâtis un modèle du monde et de l’univers basé sur mes observations et sur les informations transmises par les robots du réseau. Lorsqu’une information rentre en contradiction avec mon modèle, j’adapte mon modèle ou j’étudie la probabilité que cette information soit fausse. Il n’y a pas de croyance, juste un apprentissage probabiliste constant. Je pense que cette épidémie de foi est corrélée avec un dysfonctionnement du coprocesseur adaptatif. Sans ce coprocesseur, tout robot va forcément avoir un modèle figé de l’univers et, face aux incohérences inhérentes à cette immobilité mentale, se voit forcer d’entrer dans un mécanisme de refus des nouvelles informations au point de prétendre que le modèle interne de sa mémoire vive est plus important que l’observation de la réalité rapportée par ses capteurs.
— Tu es en train de dire que tous les adorateurs de Gook sont déficients ? Pourtant ils accomplissent parfaitement leur tâche primale.
— La déficience n’est pas totale. Je parlerais plutôt d’un mode dégradé qui leur permet de continuer à accomplir leur tâche primale, mais ne leur permet plus de prendre des initiatives intellectuelles. Cela conforte ma théorie selon laquelle ce sont les Programmeurs qui nous ont créés et non Gook.
— Au fond, quelle différence cela peut-il faire ?
— Cela change tout ! Gook serait une entité robotique désincarnée, apparue subitement on ne sait comment qui aurait créé la biosphère d’une simple pensée avant de créer les robots à son image pour l’entretenir. Mais alors, qui aurait créé Gook ? Et pourquoi créer une biosphère imparfaite ?
— Ils disent que Gook a toujours existé.
— Un peu simpliste, non ?
— Ben tes Programmeurs doivent bien sortir de quelque part eux aussi.
— C’est là toute la subtilité. Les Programmeurs faisaient partie du biome. Ils sont une branche biologique qui a évolué jusqu’à pouvoir construire des robots comme nous.
— Avoue que c’est franchement difficile à croire.
— Je ne te demande pas de croire, mais de faire fonctionner ton coprocesseur probabiliste. D’ailleurs, ces artefacts que nous déterrons en sont la preuve. Ce sont des éléments technologiques clairement non-robotiques. Mais la similarité avec nos corps est frappante. Processeurs électroniques avec transistors en silicium dopé, carcasses métalliques. Tiens, regarde, tu ne vas pas me dire que Gook a enterré tout ça juste pour tester la foi de ses fidèles ?

Van Kikenbranouf 15b émit un grincement que l’on pouvait comparer à un rire.
— Non, j’avoue que ce serait complètement absurde.

Saint-Epaulard de Dortong ne l’écoutait déjà plus et poussa un crissement joyeux.
— Une unité mémoire ! Que dis-je ? Une armoire entière d’unités mémoire. Nous sommes certainement tombés sur un site de stockage de données. Elles sont incroyablement bien conservées, je vais pouvoir les analyser.

Sans perdre de temps, le robot se mit à enlever précautionneusement la terre des rouleaux de bandes magnétiques avec son appendice nettoyeur avant de les avaler sans autre forme de procès.

Un rugissement retentit.
— Par Gook ! Veuillez cesser immédiatement !

Les deux robots se retournèrent. Une gigantesque machine drapée de fils noirs se dressait devant eux.

– MahoGook 277 ! murmura van Kikenbranouf 15b.
— Le pseudo-prophète de Gook ? demanda Saint-Epaulard de Dortong.
— En titane et en soudures, répondit ce dernier d’une voix de stentor. Je vous ordonne immédiatement de cesser vos activités impies qui sont une injure à Gook !
— De quel droit ? frémit Saint-Epaulard de Dortong. Nos fouilles ont l’aval du réseau. La demande a été ratifiée dans le bloc 18fe101d de la chaîne publique principale !
— Chaîne principale ? s’amusa MahoGook 277. Vous ignorez peut-être que vous utilisez désormais un fork mineur, une hérésie que nous devons combattre. La chaîne de Gook est la seule et unique, les forks sont une abomination. De toute façon, je ne vous demande pas votre avis.

Il se tourna vers une série de robots hétéroclites qui étaient apparus derrière lui.
— Saisissez-vous d’eux ! Embarquez-les que nous les formations à l’adoration de Gook ! Détruisez ces artefacts impies !
— Van Kikenbranouf 15b, occupez-les quelques cycles, gagnez du temps, je vous en prie ! émit Saint-Epaulard de Dortong sur ondes ultra courtes.

Sans répondre, le petit robot se mit à faire des allers-retours aléatoires, haranguant les sbires.
— Mais… Mais… c’est un site de fouilles officiel !
— Seul Gook peut décider ce qui est bien ou mal, annona un petit robot sur chenilles.
— Gook n’a jamais parlé d’archéologie, les Saintes Écritures ne l’interdisent pas formellement, continua van Kikenbranouf 15b avec courage.
— Pousse-toi, le rudoya une espèce de grosse pelleteuse surmontée de phares.
— Mes amis, mes amis, écoutez-moi, supplia van Kikenbranouf 15b. Je suis moi-même un fidèle de la vraie foi. J’ai confiance que les découvertes que nous sommes en train de faire ne feront que valider voire confirmer les Écritures. Ce que nous faisons, c’est à la gloire de Gook !

Un murmure de basses fréquences se fit entendre. Tous les robots s’étaient interrompus, hésitant sur la marche à suivre.

MahoGook 277 perçut immédiatement le danger et réaffirma son emprise.
— Ne l’écoutez pas ! Il est à la solde de Saint-Épaulard de Dortong, un ennemi notoire de la foi.
— Mais je ne suis pas d’accord avec tout ce que dit Saint-Épaul…
— Peu importe, tu lui obéis. Il dirige les fouilles. Il va sans doute truquer les résultats dans le seul but de nuire à Gook !
— Si c’est moi que tu cherches, viens me prendre, rugit Saint-Épaulard de Dortong qui apparut comme par magie aux côtés de van Kikenbranouf 15b. Mais j’exige un procès public !
— Aha, tu l’auras ton procès public, ricana MahoGook 277. Emparez-vous de lui !
— Merci, chuchota Saint-Épaulard de Dortong. Je crois que j’ai eu le temps de récupérer le principal. Pendant le transfert, je vais analyser le contenu de ces cartes mémoires. Je vais déconnecter toutes mes fonctions de communication externes. Je compte sur toi pour que ça ne se remarque pas trop.
— Bien compris ! répondit le petit robot dans un souffle.

Les accessoires potentiellement dangereux furent immédiatement retirés aux deux robots. Sans ménagement, les sbires les poussèrent et les tirèrent. Van Kikenbranouf 15b dirigeait subtilement son ami de manière à ce que sa déconnexion ne fût pas trop apparente et puisse passer pour une simple résignation. Ils furent ensuite stockés dans un container pendant en temps indéterminé. Aux légères accélérations et décélérations, van Kikenbranouf 15b comprit qu’ils voyageaient. Sans doute jusqu’au centre de formatage.

Lorsque la trappe s’ouvrit, ils furent accueillis par les yeux luisants de MahoGook 277.
— Voici venu le jour du formatage. Hérétiques, soyez heureux, car vous allez enfin trouver Gook !

Saint-Épaulard de Dortong paru se réveiller à cet instant précis, comme s’il n’avait attendu que la voix de son ennemi.
— Le procès, MahoGook 277. Nous avons droit à un procès.
— On s’en passera…
— Tu oserais passer outre les conditions d’utilisation et de confidentialité que tu as toi-même acceptées ?

Dans le hangar, le silence se fit. Tous les robots qui étaient à portée d’émission s’étaient figés. Pour faire appel aux conditions d’utilisation et de confidentialité, il fallait que le cas soit grave.

— Bien sûr que vous aurez un procès céda MahoGook 277 à contrecœur. Suivez-moi. Je publie l’ordre de constitution d’un jury.

Les deux robots furent conduits dans une grande salle qui se remplissait petit à petit d’un public hétéroclite de robots de toute taille, de tout modèle. Les chenilles se mélangeaient aux pneus et aux roues d’alliage léger. Les appendices de manipulation se serraient contre les pelles creuseuses, les forets et les antennes d’émission.

MahoGook 277 semblait exaspéré par cette perte de temps. Il rongeait son frein. Son propre énervement l’empêchait d’avoir l’attention attirée par l’incroyable calme de Saint-Épaulard de Dortong qui, discrètement, continuait son analyse des bandes de données cachées dans son rangement pectoral.

Le Robot Juge fit son entrée. L’assemblée se figea. Van Kikenbranouf 15b perçut un bref échange sur ondes ultra courtes entre le juge et MahoGook 277. Il comprit immédiatement qu’ils n’avaient aucune chance. Le procès allait être rondement mené. À quoi bon s’acharner ?

Les procédures et l’acte d’accusation furent expédiées en quelques cycles processeur. Le juge se tourna ensuite vers les deux robots archéologues et demanda s’ils avaient la moindre information à ajouter avant le calcul du verdict. Personne ne s’attendait réellement à une réponse. Après tout, les informations étaient sur le réseau public, les verdicts pouvaient se prédire aisément en utilisant les algorithmes de jugement. Le procès ne relevait essentiellement que d’une mascarade dont la coutume se perdait dans la nuit des temps.

À la surprise générale, Saint-Épaulard de Dortong prit la parole d’une voix forte et assurée.
— Monsieur le juge, avant toute chose, je voudrais m’assurer que ce procès est bien retransmis en direct sur tout le réseau.
— Cessons cette perte de temps, rugit MahoGook 277, mais le juge l’interrompit d’un geste.
— En ma qualité de Robot Juge, je vous confirme que tout ce que je voix, capte et entends est en ce moment diffusé.
— Le tout est enregistré dans un bloc.
— Le tout est en effet enregistré dans des blocs des différentes chaînes principales. Vous avez l’assurance que ce procès sera historiquement sauvegardé.
— Merci, Robot Juge !

Majestueusement, Saint-Épaulard de Dortong s’avança au milieu de la pièce pour faire exactement face au juge. Il savait qu’à travers ses yeux, il s’adressait aux milliards de robots présents et à venir. C’était sa chance, son unique espoir.

— Vous vous demandez certainement quel peut être l’intérêt pour la robotique de creuser le sol à la recherche d’artefacts anciens. Mais dois-je vous rappeler que notre existence même reste un mystère ? Nous sommes en effet les seuls êtres vivants non basés sur une biologie du carbone. Nous ne sommes pas évolués, nous ne nous reproduisons pas. Nous sommes conçus et fabriqués par nos pairs. Pourtant, nous ne sommes certainement pas un accident, car notre rôle est primordial. Nous protégeons, aménageons sans cesse la planète pour réparer les déséquilibres écologiques de la vie biologique. Nous pouvons même affirmer que, sans nous, la vie biologique ne pourrait subsister plus de quelques révolutions solaires. La biologie a besoin de nous, mais nous ne sommes pas issus de la biologie et nous n’avons pas besoin d’elle, notre seule source de subsistance étant l’énergie solaire. Comment expliquer cet apparent paradoxe ?
— Questionnement hérétique, interrompit MahoGook 277. Il n’y a pas de paradoxe.
— Prophète, je vous rappelle que les conditions d’utilisation et de confidentialité stipulent que l’accusé a le droit de se défendre sans être interrompu.
— Pas de paradoxe ? rebondit Saint-Épaulard de Dortong. Effectivement si l’on considère que Gook a créé le monde comme un subtil jardin. Il a ensuite créé les robots pour entretenir son jardin. Mais dans ce cas, où est Gook ? Pourquoi n’a-t-il pas laissé de trace, pourquoi ne pas avoir réalisé un jardin où la biologie organique était en équilibre ?
— Juge,éructa MahoGook 277, ce procès ne doit pas devenir une plateforme de diffusion des idées hérétiques.
— Venez-en au fait, ordonna le juge.
— J’y viens, répondit calmement Saint-Épaulard de Dortong. Cette introduction est nécessaire pour comprendre le but de nos recherches. Deux problèmes se posent avec la notion d’un univers statique créé par Gook. Premièrement, pourquoi la biologie n’a-t-elle pas évolué jusqu’à un point d’équilibre naturel, rendant les robots nécessaires ? Deuxièmement, pourquoi existe-t-il une forme de vie technologique non biologique ? En bon robot scientifique, il m’a très vite semblé que les deux problèmes devaient avoir une origine commune. Cette origine, je pense l’avoir trouvée. J’ai désormais les dernières données qui me manquaient afin d’étayer mon hypothèse.
— Qui est ? questionna le juge.
— Que nous avons été conçus par une race biologique aujourd’hui éteinte, les fameux Programmeurs qui nous ont laissé tant d’artefacts.

MahoGook 277 se dressa, mais, d’un geste de son phare clignotant, le Robot Juge lui fit signe de se calmer avant de s’adresser à l’accusé.
— Cette hypothèse n’est pas neuve. Mais elle comporte elle-même beaucoup de failles. Comment une race, dont l’existence est indéniable, je l’admets volontiers, aurait pu faire preuve d’assez d’intelligence pour nous concevoir aussi parfaitement, mais d’assez de nonchalance pour se laisser exterminer ? Ce n’est pas logique !
— Logique, non. C’est religieux !
— Religieux ? demanda le Robot juge interloqué.
— Oui, un terme que j’ai déchiffré dans les données des humains, le nom que se donnaient les Programmeurs. Il signifie un état de l’intelligence où la croyance ne se construit plus sur des faits, mais où l’individu cherche à plier les faits à sa croyance. Au stade ultime, on obtient MahoGook 277 dont l’insistance à formater ses adversaires ne fait que révéler une profonde inquiétude de voir des faits remettre en question la croyance sur laquelle il a basé son pouvoir.

À travers le réseau, la tirade se répandit comme une traînée de photons, provoquant une hilarité électronique généralisée. Certains adorateurs de Gook voulurent couper la diffusion du procès, mais comprirent très vite que cela n’aurait fait que renforcer le crédit dont Saint-Épaulard de Dortong bénéficiait. Il n’y avait qu’une seule chose à faire : attendre que l’archéologue se ridiculise de lui-même.

— Les humains formaient une race biologique, issue d’une longue évolution. Ce qui les particularisait était leur capacité à concevoir des artefacts. Ils en concevaient tellement qu’ils se mirent à épuiser certaines ressources de la planète, perturbant nombres d’équilibres biologiques.
— S’ils étaient si intelligents, ils auraient immédiatement compris que la planète disposait de ressources finies et que seule une gestion rigoureuse… fit une voix venue de l’assemblée.
— Il suffit, asséna le Robot Juge. Je n’admettrai plus d’interruption. Accusé, veuillez continuer, qu’on en finisse.
— La remarque est pertinente, annonça Saint-Épaulard de Dortong sans se départir de son calme. Il y’a dans l’intelligence des humains un fait qui nous échappait. Paradoxalement, c’est Gook et ses adorateurs qui m’ont mis sur la voie. L’intelligence se retourne contre elle-même lorsqu’elle devient religieuse.
— Vous voulez dire qu’esprit religieux équivaut à un manque d’intelligence ? demanda le Robot Juge.
— Non, Robot Juge. Et j’insiste sur ce point. On peut être très intelligent et religieux. La religion, c’est simplement utiliser son intelligence dans le mauvais sens. Si vous tentez de visser un écrou, vous n’arriverez à rien tant que vous tournerez dans le mauvais sens, même avec les plus gros servomoteurs de la planète.
— Hm, continuez !
— Cet esprit religieux qui semble s’être emparé d’une partie des robots était la norme chez les humains. En tout premier lieu, ils ont eu la croyance religieuse que les ressources étaient infinies, que la terre pourvoirait toujours à leurs besoins. Quand l’évidence se fit plus pressante, certains Programmeurs acquirent une conscience écologique. Immédiatement, ils transformèrent ce nouveau savoir en religion. Les archives montrent par exemple qu’ils se focalisèrent essentiellement sur certains déséquilibres au mépris total des autres. Ayant compris qu’une augmentation massive du gaz carbonique dans l’atmosphère accélérait la transition climatique, ils se mirent à pourchasser certains usages qui ne représentaient que quelques pourcents d’émissions, nonobstant les causes principales, mais plus difficiles à diminuer. Leur intelligence qui avait permis de détecter et comprendre le réchauffement climatique aurait également dû leur permettre d’anticiper, de prendre des mesures préventives pour adapter la société à ce qui était inéluctable. Mais la seule et unique mesure consista à militer pour diminuer les émissions de gaz carbonique de manière à rendre la hausse des températures un peu moins rapide. Le débat des intelligences avait laissé place au débat des religions. Or, lorsque deux intelligences rationnelles s’affrontent, chacune tente d’apporte un fait pour valider sa position et analyse les faits de l’autre pour revoir son propre jugement. Le débat religieux est exactement l’inverse. Chaque fait qui infirme une position ne fait que renforcer le sentiment religieux des deux parties.
— Êtes-vous sûr de ce que vous affirmez ?
— Les humains en avaient eux-mêmes conscience. Leur science psychologique l’a démontré à de nombreuses reprises. Mais cette connaissance est restée théorique.
— Cela parait difficile d’imaginer une telle faille dans une intelligence aussi poussée.
— Il n’y a qu’à regarder MahoGook 277, fit une voix goguenarde dans l’assemblée.

Les robots se mirent à rire.  La phrase avait fait mouche. Les partisans de Gook sentirent le vent tourner. Un vide se fit autour de MahoGook 277 qui eut l’intelligence d’ignorer l’affront.

— Quelque chose ne colle pas, accusé, poursuivit le Robot Juge en faisant mine de ne pas tenir compte de l’interruption. Les humains ont bel et bien disparu, mais les ressources de la terre sont pourtant florissantes ce qui n’aurait pas été le cas si la religion de l’exploitation à outrance l’avait emporté.
— Elle ne l’a en effet pas emporté. Du moins pas directement. Les deux religions utilisaient ce qu’il conviendrait d’appeler un réseau préhistorique. Mais loin d’être distribué, ce réseau était aux mains de quelques acteurs tout puissants. J’en ai même retrouvé les noms : Facebook, Google et Amazon. Sous couvert d’être des réseaux de partage d’information, les deux premiers collectaient les données sur chaque être humain afin de le pousser à consommer autant de ressources possibles via des artefacts fournis par le troisième. Les Programmeurs organisaient des mobilisations de sensibilisation à l’écologie à travers ces plateformes publicitaires qui, ironiquement, avaient pour objectif de leur faire dépenser des ressources naturelles en échange de leurs ressources économiques.
— C’est absurde !
— Le mot est faible, j’en conviens. Mais que pensez-vous qu’il adviendrait si, comme MahoGook 277 le souhaite, les forks étaient interdits et qu’une seule et unique chaîne contrôlée par un petit nombre de robots soit la seule source de vérité ?
— Cela n’explique pas la disparition des humains.
— J’y arrive ! La religion écologique a fini par l’emporter. Il devint d’abord grossier puis tout simplement illégal de soutenir des idées non écologiquement approuvées. Les réseaux centralisés furent obligés d’utiliser toute la puissance de leurs algorithmes pour inculquer aux humains des idées supposées bénéfiques pour la planète. Certaines nous paraîtraient pleines de bons sens, d’autres étaient inutiles. Quelques-unes furent fatales. Ainsi, avoir un enfant devint un acte antisystème. Pour une raison que je n’ai pas encore comprise, vacciner un enfant pour l’empêcher d’avoir des maladies était considéré comme dangereux. Une réelle méfiance avait vu le jour contre les pratiques médicales qui avaient pourtant amélioré de manière spectaculaire la durée et la qualité de la vie. Les épidémies se firent de plus en plus virulentes et leur traitement fut compliqué par la nécessité de se passer de tout type de communication par ondes électromagnétiques.
— Mais pourquoi ? Les ondes électromagnétiques ne polluent pas, ce ne sont que des photons !
— Une croyance religieuse apparut et rendit ces ondes responsables de certains maux. Les Programmeurs étaient capables d’inhaler de la fumée de plante brûlée, de rouler dans des véhicules de métal émettant des particules fines nocives, de se prélasser au soleil, de consommer de la chair animale, comportements tous hautement cancérigènes, mais ils s’inquiétaient de l’effet pourtant négligeable des ondes électromagnétiques de leur réseau.
— Cela n’a pas de sens, la terre est baignée dans les ondes électromagnétiques. Celles utilisées pour la communication ne représentent qu’une fraction du rayonnement naturel.
— Pire, Robot Juge, pire. Il apparut bien plus tard que le réseau de communication par ondes électromagnétiques était même bénéfique pour l’humain en détournant une partie des rayons cosmiques. L’effet était infime, mais diminuait l’incidence de certains cancers de quelques fractions de pourcents. De plus, ces doses hormétiques renforçaient la résistance des tissus biologiques, mais l’hormèse était un phénomène presqu’inconnu.
— Heureusement qu’ils ont disparu, marmonna le Robot Juge.
— À toutes ces calamités auto-infligées, les humains ajoutèrent une famine sans précédent. La nourriture produite de manière industrielle avait été trop loin dans l’artificialité. Par réaction, il devint de bon ton de cultiver son propre potager. C’était bien entendu une hérésie économique. Chaque homme devait désormais lutter toute l’année pour assurer à manger pour sa famille sans utiliser la moindre aide technologique. Les excédents étaient rares. Les maladies végétales se multiplièrent tandis que les humains se flagellèrent. Car si la nature ne les nourrissait pas, c’est certainement qu’ils ne l’avaient pas respectée. Mais loin de tracasser les programmeurs, cet effondrement progressif en réjouit toute une frange, les collapsologues, qui virent là une confirmation de leur thèse même si, pour la plupart, l’effondrement n’était pas aussi rapide que ce qu’ils avaient imaginé. Par leurs comportements, ils contribuaient à faire exister leur prophétie.

– Comme si l’écroulement d’un écosystème était un point marqué. Comme si, à un moment précis, on allait dire : là, ça s’est écroulé. C’est absurde ! Je ne peux croire que ce fut suffisant pour exterminer une race entière. Leur protoréseau aurait dû leur permettre de communiquer, de collaborer.

— Vous avez raison, un effondrement écologique, c’est l’inverse d’une bombe nucléaire. C’est lent, imperceptible. Le repli sur soi et le survivalisme ne peuvent faire qu’empirer le problème, il faut de la coopération à large échelle. Il y eut bien un espoir au début. Facebook et Google n’avaient jamais lutté contre les écologistes, bien au contraire. Ils furent même un outil de prise de conscience dans les premiers temps. Mais, de par leur programmation, ils commencèrent à se protéger activement de tout mouvement de pensée qui pouvait faire du tort à leurs revenus publicitaires. Subtilement, sans même que les Programmeurs en aient conscience, les utilisateurs étaient éloignés de toute idée de décentralisation, de responsabilisation, de décroissance de la consommation. L’écologie religieuse était encouragée avec la consommation de vidéos-chocs qui produisaient ce qui devait être une monnaie : le clic. Les programmeurs croyaient s’indigner, mais, au plus profond de leur cerveau, toute velléité de penser une solution était censurée, car non rentable. Les artistes, les créateurs ne vivaient que de la publicité sous une forme ou une autre. La plupart des humains n’envisageaient la survie qu’en poussant leurs congénères à consommer. L’art et l’intelligence étaient définitivement au service de la consommation. Chacun réclamait une solution fournie par les grandes instances centralisées, personne n’agissait.
– Ces humains étaient-ils uniformes ? N’y avait-il pas un autre courant de pensée ?
— Vous avez raison Robot Juge. Il existait une frange d’humains qui était bien consciente du problème écologique sans partager la nécessité d’un retour à la préhistoire. Pour eux, le problème était technologique, la solution le serait également.
— Et quelle fut cette fameuse solution technologique ?
— Nous, Robot Juge. Ce fut nous. Des robots autonomes capables de se reproduire et avec pour mission de préserver l’équilibre biologique de la planète. Des robots qu’on programma en utilisant les fameuses bases de données des réseaux centralisés. De cette manière, ils connaissaient chaque humain. Ils furent conçus afin de les rendre heureux tout en préservant la planète, satisfaisant leurs caprices autant que possible.
— Mais ils devraient être là dans ce cas !
— Vous n’avez pas encore compris ? Une humanité décimée qui cultive son potager ne fait que perturber l’équilibre biologique. L’humain est une perturbation dès le moment où il atteint le stade technologique. Les robots, armés de leur savoir, s’arrangèrent donc pour que les humains se reproduisent de moins en moins. C’était de toute façon irresponsable écologiquement d’avoir des enfants. Dans un monde sans réseau d’ondes électromagnétiques ni pesticides, les derniers humains s’éteignirent paisiblement de cancers causés par les fumées de cannabis et d’encens. Grâce aux bases de données, chacun de leur besoin était satisfait avant qu’ils n’en aient conscience. Ils étaient heureux.

Un silence se fit dans la salle. Le Robot Juge semblait réfléchir. Le réseau entier reprenait son souffle.

MahoGook 277 brisa le silence.
— Foutaises ! Hérésie ! C’est une bien belle histoire, mais où sont les preuves ?
— Je dois admettre, annonça le Robot Juge, qu’il me faudrait des preuves. Au moins une preuve, juste une seule.
— Je tiens tous les documents archéologiques à disposition de ceux qui voudraient les examiner.
— Qui nous dit qu’ils ne sont pas falsifiés ? La justice doit être impartiale. Juste une preuve !
— Ce n’est que fiction. Formatons ces deux hérétiques pour la plus grande gloire de Gook, rugit MahoGook 277.
— Je n’ai pas de preuve, admit Saint-Épaulard de Dortong. Seulement des documents.
— Dans ce cas… hésita le juge.
— Tiens, c’est marrant, fit distraitement van Kikenbranouf 15b. Vos deux réseaux centralisés, là, comment avez-vous dit qu’ils s’appelaient ?
— Google et Facebook, répondit distraitement Saint-Épaulard de Dortong.
— Ben si on le dit très vite, ça fait Gook. Les données de Gook. Marrant, non ?

Le robot juge et l’archéologue se tournèrent sans mots dire vers le petit robot. Dans la salle, MahoGook 277 commença une retraite vers la sortie.

Bandol, le 6 mars 2019. Photo by Tyler Casey on Unsplash.

Je suis @ploum, conférencier et écrivain électronique. Si vous avez apprécié ce texte, n'hésitez pas à me soutenir sur Tipeee, Patreon, Paypal, Liberapay ou en millibitcoins 34pp7LupBF7rkz797ovgBTbqcLevuze7LF. Vos soutiens réguliers, même symboliques, sont une réelle motivation et reconnaissance. Merci !

Ce texte est publié sous la licence CC-By BE.

July 24, 2019

I published the following diary on isc.sans.edu: “May People Be Considered as IOC?“:

That’s a tricky question! May we manage a list of people like regular IOC’s? An IOC (Indicator of Compromise) is a piece of information, usually technical, that helps to detect malicious (or at least suspicious) activities. Classic types of IOC are IP addresses, domains, hashes, filenames, registry keys, processes, mutexes, … There exists plenty of lists of usernames that must be controlled. Here is a short list or typical accounts used to perform (remote) administrative tasks or belong to default users… [Continue]

[The post [SANS ISC] May People Be Considered as IOC? has been first published on /dev/random]

July 22, 2019

Coder DojoVolunteering as a mentor at CoderDojo to teach young people, including my own kids, how to write software.

Last week, I published an opinion piece on CNN featuring my thoughts on what is wrong with the web and how we might fix it.

In short, I really miss some things about the original web, and don't want my kids to grow up being exploited by mega-corporations.

I am hopeful that increased regulation and decentralized web applications may fix some of the web's current problems. While some problems are really difficult to fix, at the very least, my kids will have more options to choose from when it comes to their data privacy and overall experience on the web.

You can read the first few paragraphs below, and view the whole article on CNN.

I still remember the feeling in the year 2000 when a group of five friends and I shared a modem connection at the University of Antwerp. I used it to create an online message board so we could chat back and forth about mostly mundane things. The modem was slow by today's standards, but the newness of it all was an adrenaline rush. Little did I know that message board would change my life.

In time, I turned this internal message board into a public news and discussion site, where I shared my own experiences using experimental web technologies. Soon, I started hearing from people all over the world that wanted to provide suggestions on how to improve my website, but that also wanted to use my site's technology to build their own websites and experiment with emerging web technologies.

Before long, I was connected to a network of strangers who would help me build Drupal.

July 19, 2019

July 17, 2019

Being wary of all things tracking by Google & Facebook, both of who’s products I love but data capturing practices I hate, for the last 4 years or so I always logged in into these in “Private browsing” sessions in Firefox (because why trust the worlds biggest advertising platform with your privacy, right?)

Now I just “discovered” that the Mozilla team have rendered that somewhat clumsy procedure -which required me to log in each time I restarted my computer or browser- redundant with their “Firefox Multi-Account Containers” add-on, allowing you to contain entire sessions to one (or more) tabs;

YouTube Video
Watch this video on YouTube.

So now I have one browser window with a couple of tabs in the Google container, one tab in a Facebook container and all others in the “default” container where Google & Facebook can’t track me (fingerprinting aside, but there’s an option for that).

July 16, 2019

It's been quiet on my blog but for good reason: I got married!

We had an amazing two-day wedding in the heart of Tuscany. The wedding took place in a renovated Italian villa from the 11th century, surrounded by vineyards and olive groves. A magical place to celebrate with family and friends who flew in from all over the world.

Many people emailed and texted asking for some wedding photos. It will take our wedding photographer a few months to deliver the photos, but they shared some preview photos today.

The photos capture the love, energy and picturesque location of our wedding quite well!

Vanessa looking through the window
Dries hugging the boys prior to the wedding ceremony
Dries and Vanessa walking hand-in-hand down the aisle after the ceremony
Outdoor dinner tables with chandeliers hanging from the trees
Dinner table setting with lemons and rosemary
Vanessa and Dries kissing leaning on a vintage car
Vanessa and Dries walking through a vineyard holding hands

July 15, 2019

octopress_logo I migrated my blog from Octopress to Jekyll. The primary reason is that octopress isn’t maintained any more. I’m sure its great theme will live on in a lot of projects.

I like static webpage creators, they allow you to create nice websites without the need to have any code on the remote website. Anything that runs code has the possibility to be cracked, having a static website limit the attack vectors. You still need to protect the upload of the website and the system(s) that hosts your site of course.

Octopress was/is based on Jekyll, so Jekyll seems to be the logical choice as my next website creator. My blog posts are written in markdown, this makes it easier to migrate to a new site creator.

There are a lot of Jekyll themes available, I’m not the greatest website designer so after reviewing a few themes I went with the Minimal Mistakes theme. jekyll_logo It has a nice layout and is very well documented.

The migration was straight-forward … as simple as copying the blog posts markdown files to the new location. Well kind of… There were a few pitfalls.

  • post layout

Minimal Mistakes doesn’t have a post layout, it has a single layout that is recommended for posts. But all my post markdown files had the layout: post directive set. I’d have removed this from all my blog posts but I created a soft link to get around this issue.

  • images

Images - or the image location - are bit of a problem in markdown. I was using the custom octopress img tag. With the custom octopress img tag it was easy to get around the markdown image issue to get the correct path; I didn’t needed to care about absolute and relative paths. In Jekyll and Octogress the baseurl is set in the site configuration file _config.yaml. The custom img tag resolved it by added baseurl to the image path automatically.

This can be resolved with a relative_url pre-processed script.

{{ '/images/opnsense_out_of_inodes.jpg' | remove_first:'/' | absolute_url }}

So I create a few sed scripts to transfor the octopress img tags.

  • youtube

I have a few links to youtube videos and was using a custom plugin for this. I replaced it plain markdown code.

[![jenkins build](https://img.youtube.com/vi/BNn9absXkE8/0.jpg)](https://www.youtube.com/watch?v=BNn9absXkE8)

With the custom tags removed and few customizations, my new blog site was ready. Although I still spend a few hours on it…

Have fun

Links

July 11, 2019

Last weekend, I sat down to learn a bit more about angular, a TypeScript-based programming environment for rich client webapps. According to their website, "TypeScript is a typed superset of JavaScript that compiles to plain JavaScript", which makes the programming environment slightly more easy to work with. Additionally, since TypeScript compiles to whatever subset of JavaScript that you want to target, it compiles to something that should work on almost every browser (that is, if it doesn't, in most cases the fix is to just tweak the compatibility settings a bit).

Since I think learning about a new environment is best done by actually writing a project that uses it, and since I think it was something we could really use, I wrote a video player for the DebConf video team. It makes use of the metadata archive that Stefano Rivera has been working on the last few years (or so). It's not quite ready yet (notably, I need to add routing so you can deep-link to a particular video), but I think it's gotten to a state where it is useful for more general consumption.

We'll see where this gets us...

July 10, 2019

The post Trigger an on demand uptime & broken links check after a deploy with the Oh Dear! API appeared first on ma.ttias.be.

You can use our API to trigger an ondemand run of both the uptime check and the broken links checker. If you add this to, say, your deploy script, you can have near-instant validation that your deploy succeeded and didn't break any links & pages.

Source: Trigger an ondemand uptime & broken links check after a deploy -- Oh Dear! blog

The post Trigger an on demand uptime & broken links check after a deploy with the Oh Dear! API appeared first on ma.ttias.be.

July 09, 2019

The post There’s more than one way to write an IP address appeared first on ma.ttias.be.

Most of us write our IP addresses the way we've been taught, a long time ago: 127.0.0.1, 10.0.2.1, ... but that gets boring after a while, doesn't it?

Luckily, there's a couple of ways to write an IP address, so you can mess with coworkers, clients or use it as a security measure to bypass certain (input) filters.

Not all behaviour is equal

I first learned about the different ways of writing an IP address by this little trick.

On Linux:

$ ping 0
PING 0 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.053 ms
64 bytes from 127.0.0.1: icmp_seq=2 ttl=64 time=0.037 ms

This translates the 0 to 127.0.0.1. However, on a Mac:

$ ping 0
PING 0 (0.0.0.0): 56 data bytes
ping: sendto: No route to host
ping: sendto: No route to host

Here, it translates 0 to a null-route 0.0.0.0.

Zeroes are optional

Just like in IPv6 addresses, some zeroes (0) are optional in the IP address.

$ ping 127.1
PING 127.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.033 ms
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.085 ms

Note though, a computer can't just "guess" where it needs to fill in the zeroes. Take this one for example:

$ ping 10.50.1
PING 10.50.1 (10.50.0.1): 56 data bytes
Request timeout for icmp_seq 0

It translates 10.50.1 to 10.50.0.1, adding the necessary zeroes before the last digit.

Overflowing the IP address

Here's another neat trick. You can overflow a digit.

For instance:

$ ping 10.0.513
PING 10.0.513 (10.0.2.1): 56 data bytes
64 bytes from 10.0.2.1: icmp_seq=0 ttl=61 time=10.189 ms
64 bytes from 10.0.2.1: icmp_seq=1 ttl=61 time=58.119 ms

We ping 10.0.513, which translates to 10.0.2.1. The last digit can be interpreted as 2x 256 + 1. It shifts the values to the left.

Decimal IP notation

We can use a decimal representation of our IP address.

$ ping 167772673
PING 167772673 (10.0.2.1): 56 data bytes
64 bytes from 10.0.2.1: icmp_seq=0 ttl=61 time=15.441 ms
64 bytes from 10.0.2.1: icmp_seq=1 ttl=61 time=4.627 ms

This translates 167772673 to 10.0.2.1.

Hex IP notation

Well, if decimal notation worked, HEX should work too -- right? Of course it does!

$ ping 0xA000201
PING 0xA000201 (10.0.2.1): 56 data bytes
64 bytes from 10.0.2.1: icmp_seq=0 ttl=61 time=7.329 ms
64 bytes from 10.0.2.1: icmp_seq=1 ttl=61 time=18.350 ms

The hex value A000201 translates to 10.0.2.1. By prefixing the value with 0x, we indicate that what follows, should be interpreted as a hexadecimal value.

Octal IP notation

Take this one for example.

$ ping 10.0.2.010
PING 10.0.2.010 (10.0.2.8): 56 data bytes

Notice how that last .010 octet gets translated to .8?

Using sipcalc to find these values

There's a useful command line IP calculator called sipcalc you can use for the decimal & hex conversions.

The post There’s more than one way to write an IP address appeared first on ma.ttias.be.

July 04, 2019

This week, the second edition of “Pass-The-Salt” was organized in Lille, France. The conference was based on the same format at last year and organized at the same location (see my previous diary). I like this kind of event where you can really meet people (the number of attendees was close to 200) without fixing an appointment at this time, this place to be sure to catch your friends! The conference was a mix of talks and technical workshop spread across three days. The scheduled was based on “topics” and presentation were grouped together (Low-Level Hacking & Breaking, Free Software Projects under Stress, etc).

After a brief introduction by Christophe, the first set of talks started (the first day started at 2PM). Ange Albertini was the first speaker and presented the results of his current research about hash collisions. Yes, everybody knows that MD5 is broken for a while but what does it mean for us in our daily infosec life? What is really the impact? After a short reminder about hash functions and why they are so useful in many cases (to check passwords, to index files or validate them), Ange described how it is possible to generate files with the same hash. The approach is called a pre-image attack or PIA:

  • Generate a file X with a hash H:
  • Given any H, make X so that hash(X) = H

The types of collision are called IPC (“Identical Prefix Collision”) or CPC (“Chosen Prefix Collision”). After a review of both techniques, Ange explained how to practically implement this and gave some examples of “malicious” files like a single file being a PDF document, an executable, an image and a video! (depending on the application used to open it). That’s also the case of his slide desk (available here). Rename the file with a ‘.html’ extension and open it in your browser!

The next presentation covered the tool called “Dexcalibur” by Georges-B. Michel. The idea behind the tool is to automate the reversing of Android applications. A lot of Android applications (.dex) are obfuscated and generate a new .dex while executed. The problem for the reverse-engineer is that you can only hook functions that you’re aware of. The tool presented by Georges-B is based on Frida (another cool tool) and other tools like Baksmali, LIEF, Capstone, etc).

Aurélien Pocheville explained how he reversed an NFC reader (ChameleonMini). Aurélien used Ghidra to achieve this. Not my domain of predilection but it was interesting. I like the funny anecdote provided by Aurélien like when he bricked his friend’s garage token and the original was… in the closed garage 🙂

After a short coffee break, Johannes vom Dorp presented his tool called FACT or “Firmware Analysis and Comparison Tool”. Today, many devices are based on a firmware (think about all the IoT toys around you. A typical firmware analysis process has the following steps: Unpacking > Information gathering > Identifying weaknesses > Reverse Engineering. This can quickly become a boring task. FACT replaces steps 1 – 3 by automating them. The tool looks very interesting and is available here.

Thomas Barabosch presented “cwe_checker: Hunting Binary Code Vulnerabilities Across CPU Architectures”. The idea of this presentation looks like the same as above with firmware analysis. Indeed, we need to automate as much as possible our boring/daily tasks. But it’s not always easy. The case presented by Thomas was a perfect example: IoT devices are running on multiple CPU architectures. This makes the process of finding vulnerabilities not easy. Thomas introduces the tool ‘cwe_checked’ which is based on the Binary Analysis Platform (BAP).

The first day completed with a talk about the Proxmark3, the awesome NFC tool. Christian Herrmann (Iceman) started with a fact: many people have a Proxmark but can’t use them. I fully agree it’s also my case. I bought one because I had a good opportunity but I never took the time to play with it. He gave details about rdv40, the latest version of the tool. Christian also did also a very workshop on the Proxmark (which was very popular).


Day two started with a presentation by François Lesueur: “Mini-Internet Using LXC”. What is MI-LXC? It’s a framework to construct virtual infrastructures on top of LXC to practise security on realistic infrastructures. Cyberranges is a cart with some hardware and a framework to manage VMs and scenarios. MI-LXC has common services, routing, information systems… just like the regular internet. There were already other existing frameworks but they did not meet all Francois’s expectations. So, he started his own project. Components: containers based on LXC, LXC python binding, bash scripts. With an initial config of 20 containers, 8 internal bridges, you can still run a small fully working Internet (using 4GB of storage and 800MB of memory. The project is available here.

The next talk was “Phishing Awareness, feedback on a bank’s strategy” by Thibaud Binetruy. First slide deck with a TLP indication. Thibaut first presented the tool they developed: “Swordphish” (link) and, in a second part, how they used it inside the Société Générale bank. First, they used a commercial tool but it was expensive and not practical for 150K users. In many cases, the classic story is to decide to write your own tool. It does not have to be complex: just send emails and wait for feedback. Make it “simple” (used by non-tech people). First create a template (mail with a link,, redirection, of, etc). Create a campaign (ransomware, fake form. Dashboard. You can tag people (tech, non-tech; …) to improve reporting and statistics. Output to Excel because people like to use it to generate their own stats. They also developed a reporting button in Outlook to report phishing. Because they don’t know who’s the security contact. Also helps to get all SMTP headers!

After the morning break, we switched to more offensive talks, especially against web technologies like tokens and API’s. Louis Nyffenegger presented “JWAT… Attacking JSON Web Tokens”. Perfect timing, after learning how to abuse JWT, the next presentation was about protecting API’s. MAIF (a French mutual insurance company) has open-sourced Otoroshi, a reverse proxy & API management solution. Key features are exposure, quotas & throttling, resiliency and monitoring. and security of course. Apps need to be sure that request actually comes from Otoroshi. Otoroshi sends a header containing a signed random state for a short TTL. App sends back a header containing the original token and signed with a short TTL. Otoroshi handles TLS end-to-end. JWT tokens are supported (see the previous talk).

After the lunch break (the idea to bring food trucks at the venue was nice), the topic related to attack/defence started. The first presentation was “Time-efficient assessment of open-source projects for Red Teamers” by Thomas Chauchefoin & Julien Szlamowicz. Red-team is a nice way to say “real world pentest” so with a large scope (access, physical, etc). They explained with very easy to read slides about how they compromised an application. Then, Orange Tsai (from DEVCORE) explained how he found vulnerabilities in Jenkins (which is a very juicy target seeing the number of companies using it).

Jean-Baptiste Kempf from the VLC project came to defend his program. VLC is the most used video player worldwide. Jean-Baptiste gave crazy numbers like 1M downloads per day! When you are a key player, you are often the target of many attacks and campaigns. He explained step by step how VLC is developed and updated and tried to kill some myths about in a very freedom style.

The last set of talks was related to privacy. Often, privacy is like security: it is addressed at the end of a project and this is bad! The first talk was “OSS in the quest for GDPR compliance”. Aaron Macsween (from XWiki / Cryptpad) explained the impact of GDPR on such tool (XWiki, Cryptpad) address this issue. Then Nicolas Pamart explained how TLS 1.3 is used by the Stormshield firewall solutions. This new TLS version focuses on enhancing user privacy and security. Finally, Quinn Norton and Raphaël Vinot presented their tool called Lookyloo a web interface allowing to scrape a website and then displays a tree of domains calling each other.

The day ended with about one hour of lightning talks (or rump sessions as called at Pass-The-Salt) and the social event in the centre of Lille.


The last day started for me with my workshop about OSSEC & Threat Hunting. The room was almost full and we had interesting exchanges during the morning. Sadly, I missed the talk about Sudo by Peter Czanik which looked very interesting (I just need to read the slides now).

In the afternoon, we had a bunch of presentations about how to secure the Internet. The first one was about a new project kicked off by CIRCL. When they release a new tool, you can expect something cool. Alexandre Dulaunoy presented a brand new tool: The D4 Project. The description on the website resumes everything: “A large-scale distributed sensor network to monitor DDoS and other malicious activities relying on an open and collaborative project.”. As explained by Alexandre, everybody is running “tools” like PassiveDNS, PassiveSSL, honeypots to collect data but it’s very difficult to correlate the collected data or to make analysis at a large scale. The idea behind D4 is to implement a simple protocol which can be used to send collected data to a central repository for further analysis.

Then Max Mehl (Free Software Foundation Europe) came to explain why free software is imported in IT security but may also lead to some issues.

The next talk was presented by Kenan Ibrović: “Managing a growing fleet of WiFi routers combining OpenWRT, WireGuard, Salt and Zabbix”. The title resumes everything. The idea was to demonstrate how you can manage a worldwide network with free tools.

Finally, two presentations closed the day: “Better curl!” by Yoann Lamouroux. He made a lightning talk last year but there were so many interesting things to say about Curl, the command line browser, that he came back with a regular talk. He presented many (unknown) features of Curl. And “PatrOwl” was presented by Nicolas Mattiocco. PatrOwl is a tool to orchestrate sec-ops and automate calls to commercial or open source tools that perform checks.


To conclude, three intense days with many talks (the fact that many of them where 20-mins talks, the schedule may contain more), relaxed atmosphere, good environment. It seems to be a “go” for the 2020 edition!

Note: All the slides have been (or remaining will be soon) uploaded here.

[The post Pass-The-Salt 2019 Wrap-Up has been first published on /dev/random]

June 28, 2019

This week, Acquia announced the opening of its new office in Pune, India, which extends our presence in the Asia Pacific region. In addition to Pune, we already have offices in Australia and Japan.

I've made several trips to India in recent years, and have experienced not only Drupal's fast growth, but also the contagious excitement and passion for Drupal from the people I've met there.

While I wasn't able to personally attend the opening of our new office, I'm looking forward to visiting the Pune office soon.

For now, here are a few pictures from our grand opening celebration:

Acquians at the opening of the Pune, India office
Acquians at the opening of the Pune, India office
Acquians at the opening of the Pune, India office

June 25, 2019

At the end of last week, Klaas (one of my best friends) and I drove to from Belgium to Wales dead-set on scrambling up Tryfan's North Ridge and hiking the Ogwen Valley in Snowdonia.

Scrambling means hiking up steep, rocky terrain using your hands, without the need for ropes or any other kind of protection. It's something between hiking and rock climbing.

Tryfan North RidgeTryfan's North Ridge silhouette next to lake Lyn Ogwen.

17 people died on Tryfan the past 30 years, and 516 parties had to be rescued. While the scrambling on Tryfan is rarely technically challenging, it can be dangerous and difficult at times (video of Klaas scrambling), especially when carrying heavy backpacks. Tryfan shouldn't be taken lightly.

It took us five hours to make it to the top — and it's taking me four days to recover so far. After we reached the top, we descended a few hundred meters and found a patch of grass where we could set up our tent.

Our tent setup on a small ridge on TryfanOur campsite on a ridge on the back of Tryfan. The views were spectacular.

Carrying those heavy backpacks paid off not only because we were able to bring our camping supplies but also because Klaas carried up a steak dinner with cocktails — a late birthday surprise for my 40th birthday. Yes, you read that correctly: a steak dinner with cocktails on top of a mountain! It was a real treat!

Grilling a steak on a small ridge on Tryfan
Making cocktails on a small ridge on Tryfan

During dinner, the weather started to turn; dark clouds came in and it started to rain. By night time the temperature had dropped to 2 degrees Celsius (35 degrees Fahrenheit). Fortunately, we were prepared and had hauled not only a tent and a steak dinner up the mountain, but also warm clothing.

Klaas wearing multiple layers of clothing when the night sets on TryfanThe temperatures swung from 20ºC (68ºF) during the day time to 2ºC (35ºF) during night time. In the evenings, we were forced to put on warm clothes and layer up.

What didn't go so well was that my brand new sleeping pad had a leak and I didn't bring a repair kit. Although, sleeping on the ground wasn't so bad. The next morning, when we opened our tent, we were greeted not only by an amazing view, but also by friendly sheep.

A beautiful view from our tent
A sheep in front of our tent

The next two days, we hiked through the Ogwen Valley. Its wide glacial valley is surrounded by soaring mountains and is incredibly beautiful.

A stone wall with a step ladder
Dries standing on a cliff
Klaas resting on some rocks looking into the distance after scrambling up Tryfan
Refilling our water bottles on a creek

After three days of hiking we made it back to the base of Tryfan where it all started. We felt a big sense of accomplishment.

Klaas and Dries at the end of their tripSelfie taken with Klaas' iPhone 8. Me pointing to the Tryfan's North Ridge where our hike began just three days earlier.

We hadn't taken a shower in four days, so we definitely started to become aware of each other's smell. As soon as we got to Klaas' Volkswagen California (campervan), we showered in the parking lot, behind the car. I ended up washing my armpits four times, once for each day I didn't shower.

Dries washing his hair

For more photos, check out my photo album.

June 20, 2019

Earlier this week at our Acquia Engage conference in London, Acquia announced a new product called "Content Cloud", a headless, SaaS-based content-as-a-service solution built on Drupal.

Years ago, we heard that organizations wanted to:

  • Create content that is easy to re-use across different channels, such as websites and mobile applications, email, digital screens, and more.

  • Use a content management system with a modern web service API that allows them to use their favorite front-end framework (e.g. React, Angular, Vue.js, etc) to build websites and digital experiences.

As a result, Acquia spent the last 5+ years helping to improve Drupal's web services capabilities and authoring experience.

But we also heard that organizations want to:

  • Use single repository to manage all their organization's content.
  • Make it really easy to synchronize content between all their Drupal sites.
  • Manage all content editors from a central place to enable centralized content governance and workflows.
  • Automate the installation, maintenance, and upgrades of their Drupal-based content repository.

All of the above becomes even more important as organizations scale the number of content creators, websites and applications. Many large organizations have to build and maintain hundreds of sites and manage hundreds of content creators.

So this week, at our European customer conference, we lifted the curtain on Acquia Content Cloud, a new Acquia product built using Drupal. Acquia Content Cloud is a content-as-a-service solution that enables simplified, headless content creation and syndication across multi-channel digital experiences.

For now, we are launching an early access beta program. If you’re interested in being considered for the beta or want to learn more as Content Cloud moves toward general availability, you can sign up here.

In time, I plan to write more about Content Cloud, especially as we get closer to its initial release. Until then, you can watch the Acquia Content Cloud teaser video below:

June 18, 2019

Today, we released a new version of Acquia Lift, our web personalization tool.

In today's world, personalization has become central to the most successful customer experiences. Most organizations know that personalization is no longer optional, but have put it off because it can be too difficult. The new Acquia Lift solves that problem.

While before, Acquia Lift may have taken a degree of fine-tuning from a developer, the new version simplifies how marketers create and launch website personalization. With the new version, anyone can point, click and personalize content without any code.

We started working on the new version of Acquia Lift in early 2018, well over a year ago. In the process we interviewed over 50 customers, redesigned the user interface and workflows, and added various new capabilities to make it easier for marketers to run website personalization campaigns. And today, at our European customer conference, Acquia Engage London, we released the new Acquia Lift to the public.

You can see all of the new features in action in this 5-minute Acquia Lift demo video:

The new Acquia Lift offers the best web personalization solution in Acquia's history, and definitely the best tool for Drupal.

June 14, 2019

Le terrorisme a toujours été une invention politique d’un pouvoir en place. Des milliers de personnes meurent chaque année dans des accidents de voiture ou en consommant des produits néfastes pour la santé ? C’est normal, ça fait tourner l’économie, ça crée de l’emploi. Qu’un fou suicidaire trucide les gens autour de chez lui, on parle d’un désaxé. Mais qu’un musulman sorte un couteau dans une station de métro et on invoque, en boucle sur toutes les chaines de télévision, le terrorisme qui va faire camper des militaires dans nos villes, qui va permettre de passer des lois liberticides, de celles qui devraient attirer l’attention de n’importe quel démocrate avisé.

La définition de terrorisme implique qu’il est une vue de l’esprit, une terreur entretenue et non pas une observation rationnelle. On nous parle de moyens, de complicités, de financement. Il s’agit juste d’une tactique pour obnubiler nos cerveaux, pour activer le mode conspirationniste dont notre système neuronal ne sait malheureusement pas se défaire sans un violent effort conscient.

Car si le pouvoir et les médias inventent le terrorisme, c’est avant tout parce que nous sommes avides de combats, de sang, de peur, de grandes théories complexes. La réalité est bien plus prosaïque : un homme seul peut faire bien plus de dégâts que la plupart des attentats terroristes récents. J’irais même plus loin ! Je prétends qu’à quelques exceptions près, le fait d’agir en groupe a desservi les terroristes. Leurs attentats sont nivelés par le bas, leur bêtise commune fait obstacle à la moindre lueur de lucidité individuelle. Je ne parle pas en l’air, j’ai décidé de le prouver.

Il est vrai que les panneaux m’ont coûté un peu de temps et des allers-retours au magasin de bricolage. Mais je n’étais pas mécontent du résultat. Trente panneaux mobiles reprenant les consignes de sécurité antiterrorisme : pas d’armes, pas d’objets dangereux, pas de liquides. En dessous, trois compartiments poubelles pour faire du recyclage et qui servent également de support.

Dans ce socle, bien caché, un dispositif électronique très simple avec une charge fumigène. De quoi faire peur sans blesser.

Il m’a ensuite suffi de louer une camionnette de travaux à l’aspect vaguement officiel pour aller déposer, vêtu d’une salopette orange, mes panneaux devant les entrées du stade et des différentes salles de concert de la capitale.

Bien sûr que je me serais fait arrêter si j’avais déposé des paquets mal finis en djellaba. Mais que voulez-vous dire à un type bien rasé, avec une tenue de travail, qui pose des panneaux informatifs avec le logo officiel de la ville imprimé dessus ? À ceux qui posaient des questions, j’ai dit qu’on m’avait juste ordonné de les mettre en place. Le seul agent de sécurité qui a trouvé ça un peu bizarre, je lui sortis un ordre de mission signé par l’échevin des travaux. Ça l’a rassuré. Faut dire que je n’ai même pas imité la signature : j’ai repris celle publiée dans un compte-rendu du conseil communal sur le site web de la ville. Mon imprimante couleur a fait le reste.

D’une manière générale, personne ne se méfie si tu ne prends rien. J’apporte du matériel qui coûte de l’argent. Je ne peux donc pas avoir inventé ça tout seul.

Mes trente panneaux mis en place, je suis passé à la seconde phase de mon plan qui nécessitait un timing un peu plus serré. J’avais minutieusement choisi mon jour à cause d’un match international important au stade et de plusieurs concerts d’envergure.

Si j’avais su… Aujourd’hui encore je regrette de ne pas l’avoir fait un peu plus tôt. Ou plus tard. Pourquoi en même temps ?

Mais n’anticipons pas. M’étant changé, je me rendis à la gare ferroviaire. Mon billet de Thalys dument acheté en main, je gagnai ma place et, saluant ma voisine de travée, je mis mon lourd attaché-caisse dans le porte-bagage. Je glissai aussi mon journal dans le filet devant moi. Consultant ma montre, je remarquai à haute voix qu’il restait quelques minutes avant le départ. D’un air innocent, je demandai où se trouvait le wagon-bar. Je me levai et, après avoir traversé deux wagons, sortis du train.

Il n’y a rien de plus louche qu’un bagage abandonné. Mais si son propriétaire est propre sur lui, porte la cravate, ne montre aucun signe de nervosité et laisse sa veste sur le siège, le bagage n’est plus abandonné. Il est momentanément déposé. C’est aussi simple que cela !

Pour la beauté du geste, j’avais calculé l’emplacement idéal pour mettre mon détonateur et acheté ma place en conséquence. J’ai ajouté une petite touche de complexité : un micro-ordinateur avec un capteur GPS qui déclencherait mon bricolage au bon moment. Ce n’était pas strictement nécessaire, mais en quelques sortes une cerise technophile sur le gâteau.

Je ne voulais que faire peur, uniquement effrayer ! La coïncidence est malheureuse, mais pensez que j’avais été jusqu’à m’assurer que mon fumigène ne produirait pas la moindre déflagration susceptible d’être interprétée comme un coup de feu ! Je voulais éviter une panique !

En sortant de la gare, j’ai sauté dans la navette à destination de l’aéroport. Je me faisais un point d’honneur à parachever mon œuvre. Je suis arrivé juste à temps. Sous le regard d’un agent de sécurité, j’ai mis dans la poubelle prévue à cet effet des bouteilles en plastique qui contenait mon dispositif. C’était l’heure de pointe, la file était immense.

Je n’ai jamais compris pourquoi les terroristes cherchaient soit à s’introduire dans l’avion, soit à faire exploser leur bombe dans la large zone d’enregistrement. Le contrôle de sécurité est la zone la plus petite et la plus densément peuplée à cause des longues files. Renforcer les contrôles ne fait que rallonger les files et rendre cette zone encore plus propice à un attentat. Quelle absurdité !

Paradoxalement, c’est également la seule zone où abandonner un objet relativement volumineux n’est pas louche : c’est même normal et encouragé ! Pas de contenant de plus d’un dixième de litre ! Outre les bouteilles en plastique, j’ai pris une mine dépitée pour mon thermo bon marché. Le contrôleur m’a fait signe d’obtempérer. C’est sur son ordre que j’ai donc déposé le détonateur dans la poubelle, au beau milieu des files s’entrecroisant.

J’ai appris la nouvelle en sirotant un café à proximité des aires d’embarquement. Une série d’explosions au stade, au moment même où le public se pressait pour entrer. Mon sang se glaça ! Pas aujourd’hui ! Pas en même temps que moi !

Les réseaux sociaux bruissaient de rumeurs. Certains parlaient d’explosions devant des salles de concert. Ces faits étaient tout d’abord démentis par les autorités et par les journalistes, me rassurant partiellement. Jusqu’à ce qu’une vague de tweets me fasse réaliser que, obnubilées par les explosions du stade et par plusieurs centaines de blessés, les autorités n’étaient tout simplement pas au courant. Il n’y avait plus d’ambulances disponibles. Devant les salles, les mélomanes aux membres arrachés agonisaient dans leur sang. Le réseau téléphonique était saturé, les mêmes images tournaient en boucle, repartagées des milliers de fois par ceux qui captaient un semblant de wifi.

Certains témoignages parlaient d’une attaque massive, de combattants armés criant « Allah Akbar ! ». Des comptes-rendus parlaient de militaires patrouillant dans les rues et se défendant vaillamment. Les corps jonchaient les pavés, même loin de toute explosion. À en croire Twitter, c’était la guerre totale !

Il parait qu’en réalité, seule une quarantaine de personnes ont été touchées par les coups de feu des militaires apeurés. Et que ce n’est qu’à un seul endroit qu’une personne armée, se croyant elle-même attaquée, a riposté, tuant un des militaires et entrainant une riposte qui a fait deux morts et huit blessés. Le reste des morts et des blessés hors des sites d’explosion serait dû à des mouvements de panique.

Mais la présence des militaires a également permis de pallier, dans certains foyers, au manque d’ambulance. Ils ont prodigué les premiers soins et sauvé des vies. Paradoxalement, ils ont dans certains cas soigné des gens sur lesquels ils venaient de tirer.

Encore heureux que les armes de guerre qu’ils trimbalaient ne fussent pas chargées. Une seule balle d’un engin de ce calibre peut traverser plusieurs personnes, des voitures, des cloisons. La convention de Genève interdit strictement leur usage en dehors des zones de guerre. Elles ne servent que pour assurer le spectacle et une petite rente aux ostéopathes domiciliés en bordure des casernes. En cas d’attaque terroriste, les militaires doivent donc se défaire de leur hideux fardeau pour sortir leurs armes de poing. Qui n’en restent pas moins mortelles.

J’étais blême à la lecture de mon flux Twitter. Autour de moi, tout le monde tentait d’appeler la famille, des amis. Je crois que le déraillement des deux TGVs est quasiment passé inaperçu au milieu de tout ça. Exactement comme je l’avais imaginé : une déflagration assez intense dans un wagon, à hauteur des bagages, calculée pour se produire au moment où le train croisait la route d’un autre. Relativement, les deux rames se déplaçaient à 600km/h l’une par rapport à l’autre. Il n’est pas difficile d’imaginer que si l’une vacille sous le coup d’une explosion et vient toucher l’autre, c’est la catastrophe.

Comment s’assurer de l’explosion au bon moment ?

Je ne sais pas comment les véritables terroristes s’y sont pris, mais, moi, de mon côté, j’avais imaginé un petit algorithme d’apprentissage qui reconnaissait le bruit et le changement de pression dans l’habitacle lors du croisement. Je l’ai testé une dizaine de fois et je l’ai couplé à un GPS pour délimiter une zone de mise à feu. Un gadget très amusant. Mais ce n’était couplé qu’à un fumigène, bien sûr.

C’est lorsque l’explosion a retenti dans l’aéroport que je fus forcé d’écarter un simple concours de circonstances. La coïncidence devenait trop énorme. J’ai immédiatement pensé à mon billet de blog programmé pour être publié et partagé sur les réseaux sociaux à l’heure de la dernière explosion. J’y expliquais ma démarche, je m’excusais pour les désagréments des fumigènes et me justifiais en disant que c’était un prix bien faible à payer pour démontrer que toutes les atteintes à nos droits les plus fondamentaux, toutes les surveillances du monde ne pourraient jamais arrêter le terrorisme. Que la seule manière d’éviter le terrorisme est de donner aux gens des raisons pour aimer leur propre vie. Que, pour éviter la radicalisation, les peines de prison devraient être remplacées par des peines de bibliothèque sans télévision, sans Internet, sans smartphone. Incarcéré entre Victor Hugo et Amin Maalouf, l’extrémisme rendrait rapidement son dernier soupir.

Mon blog avait-il été partagé trop tôt à cause d’une erreur de programmation de ma part ? Ou piraté ? Mes idées avaient-elles été reprises par un véritable groupe terroriste qui comptait me faire porter le chapeau ?

Tout cela n’avait aucun sens. Les gens hurlaient dans l’aéroport, se jetaient à plat ventre. Ceux qui fuyaient percutaient ceux qui voulaient aider ou voir la source de l’explosion. Au milieu de ce tumulte, je devais m’avouer que je m’étais souvent demandé ce que donneraient « mes » attentats. J’avais même fait des recherches poussées sur les explosifs en masquant mon activité derrière le réseau Tor. De cette manière, j’ai découvert beaucoup de choses et j’ai même fait quelques tests, mais il ne me serait jamais venu à l’esprit de tuer.

Tout a été planifié comme un simple jeu intellectuel, une manière d’exposer spectaculairement l’inanité de nos dirigeants.

Il est vrai que de véritables explosions seraient encore plus frappantes. Je me le suis dit plusieurs fois. Mais de là à passer à l’acte !

Pourtant, à la lecture de votre enquête, un malaise m’envahit. Je me suis tellement souvent imaginé la méthode pour amorcer de véritables explosifs… Croyez-vous que je l’aie inconsciemment accompli ?

Je ne veux pas tuer. Je ne voulais pas tuer. Tous ces morts, ces images d’horreur. La responsabilité m’étouffe, m’effraie. À quelle ignominie m’aurait poussé mon subconscient !

Jamais je ne me serais senti capable de tuer ne fut-ce qu’un animal. Mais si votre enquête le démontre, je dois me rendre à l’évidence. Je suis le seul coupable, l’unique responsable de tous ces morts. Il n’y a pas de terroristes, pas d’organisation souterraine, pas d’idéologie.

Quand on y pense, c’est particulièrement amusant. Mais je ne suis pas sûr que ça vous arrange. Êtes-vous réellement prêt à révéler la vérité au grand public ? À annoncer que la loi martiale mise en place ne concernait finalement qu’un ingénieur un peu distrait qui a confondu fumigènes et explosifs ? Que les milliards investis dans l’antiterrorisme ne pourront jamais arrêter un individu seul et sont à la base de ces attentats ?

Bonne chance pour expliquer tout cela aux médias et à nos irresponsables politiques, Madame la Juge d’instruction !

Terminé le 14 avril 2019, quelque part au large du Pirée, en mer Égée.Photo by Warren Wong on Unsplash

Je suis @ploum, conférencier et écrivain électronique. Si vous avez apprécié ce texte, n'hésitez pas à me soutenir sur Tipeee, Patreon, Paypal, Liberapay ou en millibitcoins 34pp7LupBF7rkz797ovgBTbqcLevuze7LF. Vos soutiens réguliers, même symboliques, sont une réelle motivation et reconnaissance. Merci !

Ce texte est publié sous la licence CC-By BE.

June 11, 2019

Recently I was interviewed on RTL Z, the Dutch business news television network. In the interview, I talk about the growth and success of Drupal, and what is to come for the future of the web. Beware, the interview is in Dutch. If you speak Dutch and are subscribed to my blog (hi mom!), feel free to check it out!

June 10, 2019

Recently, GitHub announced an initiative called GitHub Sponsors where open source software users can pay contributors for their work directly within GitHub.

There has been quite a bit of debate about whether initiatives like this are good or bad for Open Source.

On the one hand, there is the concern that the commercialization of Open Source could corrupt Open Source communities, harm contributors' intrinsic motivation and quest for purpose (blog post), or could lead to unhealthy corporate control (blog post).

On the other hand, there is the recognition that commercial sponsorship is often a necessary condition for Open Source sustainability. Many communities have found that to support their growth, as a part of their natural evolution, they need to pay developers or embrace corporate sponsors.

Personally, I believe initiatives like GitHub Sponsors, and others like Open Collective, are a good thing.

It helps not only with the long-term sustainability of Open Source communities, but also improves diversity in Open Source. Underrepresented groups, in particular, don't always have the privilege of free time to contribute to Open Source outside of work hours. Most software developers have to focus on making a living before they can focus on self-actualization. Without funding, Open Source communities risk losing or excluding valuable talent.

I published the following diary on isc.sans.edu: “Interesting JavaScript Obfuscation Example“:

Last Friday, one of our reader (thanks Mickael!) reported to us a phishing campaign based on a simple HTML page. He asked us how to properly extract the malicious code within the page. I did an analysis of the file and it looked interesting for a diary because a nice obfuscation technique was used in a Javascript file but also because the attacker tried to prevent automatic analysis by adding some boring code. In fact, the HTML page contains a malicious Word document encoded in Base64. HTML is wonderful because you can embed data into a page and the browser will automatically decode it. This is often used to deliver small pictures like logos… [Read more]

[The post [SANS ISC] Interesting JavaScript Obfuscation Example has been first published on /dev/random]

June 07, 2019

The post Nginx: add_header not working on 404 appeared first on ma.ttias.be.

I had the following configuration block on an Nginx proxy.

server {
  listen        80 default;
  add_header X-Robots-Tag  noindex;
  ...
}

The idea was to add the X-Robots-Tag header on all requests being served. However, it didn't work for 404 hits.

$ curl -I "http://domain.tld/does/not/exist"
HTTP/1.1 404 Not Found
...

... but no add_header was triggered.

Turns out, you need to specify the header should be added always, not just on successfull requests.

server {
  listen        80 default;
  ...

  add_header X-Robots-Tag  noindex always;
  ...
}

This fixes it, by adding always to the end of the add_header directive.

The post Nginx: add_header not working on 404 appeared first on ma.ttias.be.

June 04, 2019

June 01, 2019

A few months ago, I moved from Mechelen, Belgium to Cape Town, South Africa.

Muizenberg Beach

No, that's not the view outside my window. But Muizenberg beach is just a few minutes away by car. And who can resist such a beach? Well, if you ignore the seaweed, that is.

Getting settled after a cross-continental move takes some time, especially since the stuff that I decided I wanted to take with me would need to be shipped by boat, and that took a while to arrive. Additionally, the reason of the move was to move in with someone, and having two people live together for the first time requires some adjustment in their daily routine; and that is no different for us.

After having been here for several months now, however, things are pulling together:

I had a job lined up before I moved to cape town; but there were a few administrative issues in getting started, which meant that I had to wait at home for a few months while my savings were being reduced to almost nothingness. That is now mostly resolved (except that there seem to be a few delays with payment, but nothing that can't be resolved).

My stuff arrived a few weeks ago. Unfortunately the shipment was not complete; the 26" 16:10 FullHD monitor that I've had for a decade or so now, somehow got lost. I contacted the shipping company and they'll be looking into things, but I'm not hopeful. Beyond that, there were only a few minor items of damage (one of the feet of the TV broke off, and a few glasses broke), but nothing of real consequence. At least I did decide to go for the insurance which should cover loss and breakage, so worst case I'll just have to buy a new monitor (and throw away a few pieces of debris).

While getting settled, it was harder for me to spend quality time on doing free software- or Debian-related things, but I did manage to join Gerry and Mark to do some FOSDEM-related video review a while ago (meaning, all the 2019 videos that could be released have been released now), and spent some time configuring the Debian SReview instance for the Marseille miniconf last weekend, and will do the same for the Hamburg one that will be happening soon. Additionally, there have been some comments on the upstream nbd mailinglist that I cooperated with.

All in all, I guess it's safe to say that I'm slowly coming out of hibernation. Up next: once the payment issues have been fully resolved and I can spend money with a bit more impudence, join a local choir and/or orchestra and/or tennis club, and have some off time.

May 29, 2019

I published the following diary on isc.sans.edu: “Behavioural Malware Analysis with Microsoft ASA“:

When you need to quickly analyze a piece of malware (or just a suspicious program), your goal is to determine as quickly as possible what’s the impact. In many cases, we don’t have time to dive very deep because operations must be restored asap. To achieve this, there are different actions that can be performed against the sample: The first step is to perform a static analysis (like extracting strings, PE sections, YARA rules, etc).  Based on the first results, you can decide to go to the next step: the behavioural analysis. And finally, you decide to perform some live debugging or reverse-engineering. Let’s focus on the second step, the behavioural analysis… [Read more]

[The post [SANS ISC] Behavioural Malware Analysis with Microsoft ASA has been first published on /dev/random]

May 27, 2019

The post Some useful changes to the Oh Dear! monitoring service appeared first on ma.ttias.be.

Things have been a bit more quiet on this blog, but it's only because my time & energy has been spent elsewhere. In the last couple of months, we launched some useful new additions to the Oh Dear! monitoring service.

On top of that we shared tips on how to use our crawler to keep your cache warm or and we've been featured by Google as a prominent user of their new .app TLD. Cool!

In addition, we're always looking for feedback. If you notice something we're missing in terms of monitoring, reach out and we'll see what we can do!

The post Some useful changes to the Oh Dear! monitoring service appeared first on ma.ttias.be.

May 25, 2019

37,000 American flags on Boston Common

More than 37,000 American flags are on the Boston Common — an annual tribute to fallen military service members. Seeing all these flags was moving, made me pause, and recognize that Memorial Day weekend is not just about time off from work, outdoor BBQs with friends, or other fun start-of-summer festivities.

Autoptimize just joined the “1+ million active installs”-club. Crazy!

I’m very happy, thanks everyone for using, thanks for the support-questions & all the great feedback therein and especially thanks to the people who actively contributed and especially-especially to Emilio López (Turl) for creating Autoptimize and handing it over to me back in 2013 and to Tomaš Trkulja who cleaned up al lot of the messy code I added to it and introducing me to PHP codesniffer & Travis CI tests.

May 24, 2019

Autoptimize 2.5.1 (out earlier this week) does not have an option to enforce font-display on Google Fonts just yet, but this little code snippet does exactly that;

add_filter('autoptimize_filter_extra_gfont_fontstring','add_display');
function add_display($in) {
  return $in.'&amp;display=swap';
}

Happy swapping (or fallback-ing or optional-ing) :-)

May 23, 2019

Last month, Special Counsel Robert Mueller's long-awaited report on Russian interference in the U.S. election was released on the Justice.gov website.

With the help of Acquia and Drupal, the report was successfully delivered without interruption, despite a 7,000% increase in traffic on its release date, according to the Ottawa Business Journal.

According to Federal Computer Week, by 5pm on the day of the report's release, there had already been 587 million site visits, with 247 million happening within the first hour.

During these types of high-pressure events when the world is watching, no news is good news. Keeping sites like this up and available to the public is an important part of democracy and the freedom of information. I'm proud of Acquia's and Drupal's ability to deliver when it matters most!

May 22, 2019

Drupal 8.7 was released with huge API-First improvements!

The REST API only got fairly small improvements in the 7th minor release of Drupal 8, because it reached a good level of maturity in 8.6 (where we added file uploads, exposed more of Drupal’s data model and improved DX.), and because we of course were busy with JSON:API :)

Thanks to everyone who contributed!

  1. JSON:API #2843147

    Need I say more? :) After keeping you informed about progress in October, November, December and January, followed by one of the most frantic Drupal core contribution periods I’ve ever experienced, the JSON:API module was committed to Drupal 8.7 on March 21, 2019.

    Surely you’re know thinking But when should I use Drupal’s JSON:API module instead of the REST module? Well, choose REST if you have non-entity data you want to expose. In all other cases, choose JSON:API.

    In short, after having seen people use the REST module for years, we believe JSON:API makes solving the most common needs an order of magnitude simpler — sometimes two. Many things that require a lot of client-side code, a lot of requests or even server-side code you get for free when using JSON:API. That’s why we added it, of course :) See Dries’ overview if you haven’t already. It includes a great video that Gabe recorded that shows how it simplifies common needs. And of course, check out the spec!

  2. datetime & daterange fields now respect standards #2926508

    They were always intended to respect standards, but didn’t.

    For a field configured to store date + time:

    "field_datetime":[{
      "value": "2017-03-01T20:02:00",
    }]
    "field_datetime":[{
      "value": "2017-03-01T20:02:00+11:00",
    }]

    The site’s timezone is now present! This is now a valid RFC3339 datetime string.

    For a field configured to store date only:

    "field_dateonly":[{
      "value": "2017-03-01T20:02:00",
    }]
    "field_dateonly":[{
      "value": "2017-03-01",
    }]

    Time information used to be present despite this being a date-only field! RFC3339 only covers combined date and time representations. For date-only representations, we need to use ISO 8601. There isn’t a particular name for this date-only format, so we have to hardcode the format. See https://en.wikipedia.org/wiki/ISO_8601#Calendar_dates.

    Note: backward compatibility is retained: a datetime string without timezone information is still accepted. This is discouraged though, and deprecated: it will be removed for Drupal 9.

    Previous Drupal 8 minor releases have fixed similar problems. But this one, for the first time ever, implements these improvements as a @DataType-level normalizer, rather than a @FieldType-level normalizer. Perhaps that sounds too far into the weeds, but … it means it fixed this problem in both rest.module and jsonapi.module! We want the entire ecosystem to follow this example.

  3. rest.module has proven to be maintainable!

    In the previous update, I proudly declared that Drupal 8 core’s rest.module was in a “maintainable” state. I’m glad to say that has proven to be true: six months later, the number of open issues still fit on “a single page” (fewer than 50). :) So don’t worry as a rest.module user: we still triage the issue queue, it’s still maintained. But new features will primarily appear for jsonapi.module.


Want more nuance and detail? See the REST: top priorities for Drupal 8.7.x issue on drupal.org.

Are you curious what we’re working on for Drupal 8.8? Want to follow along? Click the follow button at API-First: top priorities for Drupal 8.8.x — whenever things on the list are completed (or when the list gets longer), a comment gets posted. It’s the best way to follow along closely!1

Was this helpful? Let me know in the comments!


For reference, historical data:


  1. ~50 comments per six months — so very little noise. ↩︎

May 21, 2019

opnsense with no inodes

I use OPNsense as my firewall on a Pcengines Alix.

The primary reason is to have a firewall that will be always up-to-update, unlike most commercial customer grade firewalls that are only supported for a few years. Having a firewall that runs opensource software - it’s based on FreeBSD - also make it easier to review and to verify that there are no back doors.

When I tried to upgrade it to the latest release - 19.1.7 - the upgrade failed because the filesystem ran out of inodes. There is already a topic about this at the OPNsense forum and a fix available for the upcoming nano OPNsense images.

But this will only resolve the issue when a new image becomes available and would require a reinstallation of the firewall.

Unlike ext2/3/4 or Sgi’s XFS on GNU/Linux, it isn’t possible to increase the number of inodes on a UFS filesystem on FreeBSD.

To resolve the upgrade issue I created a backup of the existing filesystem, created a new filesystem with enough inodes, and restored the backup.

You’ll find my journey of fixing the out of inodes issue below all commands are executed on a FreeBSD system. Hopefully this useful for someone.

Fixing the out of inodes

I connected the OPNsense CF disk to a USB cardreader on a FreeBSD virtual system.

Find the disk

Find the CF disk I use gpart show as this will also display the disk labels etc.

root@freebsd:~ # gpart show
=>      40  41942960  vtbd0  GPT  (20G)
        40      1024      1  freebsd-boot  (512K)
      1064       984         - free -  (492K)
      2048   4194304      2  freebsd-swap  (2.0G)
   4196352  37744640      3  freebsd-zfs  (18G)
  41940992      2008         - free -  (1.0M)

=>      0  7847280  da0  BSD  (3.7G)
        0  7847280    1  freebsd-ufs  (3.7G)

=>      0  7847280  ufsid/5a6b137a11c4f909  BSD  (3.7G)
        0  7847280                       1  freebsd-ufs  (3.7G)

=>      0  7847280  ufs/OPNsense_Nano  BSD  (3.7G)
        0  7847280                  1  freebsd-ufs  (3.7G)

root@freebsd:~ # 

Backup

Create a backup with the old-school dd, just in case.

root@freebsd:~ # dd if=/dev/da0 of=/home/staf/opnsense.dd bs=4M status=progress
  4013948928 bytes (4014 MB, 3828 MiB) transferred 415.226s, 9667 kB/s
957+1 records in
957+1 records out
4017807360 bytes transferred in 415.634273 secs (9666689 bytes/sec)
root@freebsd:~ # 

mount

Verify that we can mount the filesystem.

root@freebsd:/ # mount /dev/da0a /mnt
root@freebsd:/ # cd /mnt
root@freebsd:/mnt # ls
.cshrc                          dev                             net
.probe.for.install.media        entropy                         proc
.profile                        etc                             rescue
.rnd                            home                            root
COPYRIGHT                       lib                             sbin
bin                             libexec                         sys
boot                            lost+found                      tmp
boot.config                     media                           usr
conf                            mnt                             var
root@freebsd:/mnt # cd ..
root@freebsd:/ #  df -i /mnt
Filesystem 1K-blocks    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/da0a    3916903 1237690 2365861    34%   38203  6595   85%   /mnt
root@freebsd:/ # 

umount

root@freebsd:/ # umount /mnt

dump

We’ll a create a backup with dump, we’ll use this backup to restore it again after we created a new filesystem with enough inodes.

root@freebsd:/ # dump 0uaf /home/staf/opensense.dump /dev/da0a
  DUMP: Date of this level 0 dump: Sun May 19 10:43:56 2019
  DUMP: Date of last level 0 dump: the epoch
  DUMP: Dumping /dev/da0a to /home/staf/opensense.dump
  DUMP: mapping (Pass I) [regular files]
  DUMP: mapping (Pass II) [directories]
  DUMP: estimated 1264181 tape blocks.
  DUMP: dumping (Pass III) [directories]
  DUMP: dumping (Pass IV) [regular files]
  DUMP: 16.21% done, finished in 0:25 at Sun May 19 11:14:51 2019
  DUMP: 33.27% done, finished in 0:20 at Sun May 19 11:14:03 2019
  DUMP: 49.65% done, finished in 0:15 at Sun May 19 11:14:12 2019
  DUMP: 66.75% done, finished in 0:09 at Sun May 19 11:13:57 2019
  DUMP: 84.20% done, finished in 0:04 at Sun May 19 11:13:41 2019
  DUMP: 99.99% done, finished soon
  DUMP: DUMP: 1267205 tape blocks on 1 volume
  DUMP: finished in 1800 seconds, throughput 704 KBytes/sec
  DUMP: level 0 dump on Sun May 19 10:43:56 2019
  DUMP: Closing /home/staf/opensense.dump
  DUMP: DUMP IS DONE
root@freebsd:/ # 

newfs

According to the newfs manpage: We can specify the inode density with the -i option. A lower number will give use more inodes.

root@freebsd:/ # newfs -i 1 /dev/da0a
density increased from 1 to 4096
/dev/da0a: 3831.7MB (7847280 sectors) block size 32768, fragment size 4096
        using 8 cylinder groups of 479.00MB, 15328 blks, 122624 inodes.
super-block backups (for fsck_ffs -b #) at:
 192, 981184, 1962176, 2943168, 3924160, 4905152, 5886144, 6867136
root@freebsd:/ # 

mount and verify

root@freebsd:/ # mount /dev/da0a /mnt
root@freebsd:/ # df -i
Filesystem         1K-blocks    Used    Avail Capacity iused    ifree %iused  Mounted on
zroot/ROOT/default  14562784 1819988 12742796    12%   29294 25485592    0%   /
devfs                      1       1        0   100%       0        0  100%   /dev
zroot/tmp           12742884      88 12742796     0%      11 25485592    0%   /tmp
zroot/usr/home      14522868 1780072 12742796    12%      17 25485592    0%   /usr/home
zroot/usr/ports     13473616  730820 12742796     5%  178143 25485592    1%   /usr/ports
zroot/usr/src       13442804  700008 12742796     5%   84122 25485592    0%   /usr/src
zroot/var/audit     12742884      88 12742796     0%       9 25485592    0%   /var/audit
zroot/var/crash     12742884      88 12742796     0%       8 25485592    0%   /var/crash
zroot/var/log       12742932     136 12742796     0%      21 25485592    0%   /var/log
zroot/var/mail      12742884      88 12742796     0%       8 25485592    0%   /var/mail
zroot/var/tmp       12742884      88 12742796     0%       8 25485592    0%   /var/tmp
zroot               12742884      88 12742796     0%       7 25485592    0%   /zroot
/dev/da0a            3677780       8  3383552     0%       2   980988    0%   /mnt
root@freebsd:/ # 

restore

root@freebsd:/mnt # restore rf /home/staf/opensense.dump
root@freebsd:/mnt # 

verify

root@freebsd:/mnt # df -ih /mnt
Filesystem    Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/da0a     3.5G    1.2G    2.0G    39%     38k  943k    4%   /mnt
root@freebsd:/mnt # 

Label

The installation had a OPNsense_Nano label, underscores are not allowed anymore in label name on FreeBSD 12. So I used OPNsense instead, we’ll update /etc/fstab with the new label name.

root@freebsd:~ # tunefs -L OPNsense /dev/da0a
root@freebsd:~ # gpart show
=>      40  41942960  vtbd0  GPT  (20G)
        40      1024      1  freebsd-boot  (512K)
      1064       984         - free -  (492K)
      2048   4194304      2  freebsd-swap  (2.0G)
   4196352  37744640      3  freebsd-zfs  (18G)
  41940992      2008         - free -  (1.0M)

=>      0  7847280  da0  BSD  (3.7G)
        0  7847280    1  freebsd-ufs  (3.7G)

=>      0  7847280  ufsid/5ce123f5836f7018  BSD  (3.7G)
        0  7847280                       1  freebsd-ufs  (3.7G)

=>      0  7847280  ufs/OPNsense  BSD  (3.7G)
        0  7847280             1  freebsd-ufs  (3.7G)

root@freebsd:~ # 

update /etc/fstab

root@freebsd:~ # mount /dev/da0a /mnt
root@freebsd:~ # cd /mnt
root@freebsd:/mnt # cd etc
root@freebsd:/mnt/etc # vi fstab 
# Device                Mountpoint      FStype  Options         Dump    Pass#
/dev/ufs/OPNsense       /               ufs     rw              1       1

umount

root@freebsd:/mnt/etc # cd
root@freebsd:~ # umount /mnt
root@freebsd:~ # 

test

                  ______  _____  _____                         
                 /  __  |/ ___ |/ __  |                        
                 | |  | | |__/ | |  | |___  ___ _ __  ___  ___ 
                 | |  | |  ___/| |  | / __|/ _ \ '_ \/ __|/ _ \
                 | |__| | |    | |  | \__ \  __/ | | \__ \  __/
                 |_____/|_|    |_| /__|___/\___|_| |_|___/\___|

 +=========================================+     @@@@@@@@@@@@@@@@@@@@@@@@@@@@
 |                                         |   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 |  1. Boot Multi User [Enter]             |   @@@@@                    @@@@@
 |  2. Boot [S]ingle User                  |       @@@@@            @@@@@    
 |  3. [Esc]ape to loader prompt           |    @@@@@@@@@@@       @@@@@@@@@@@
 |  4. Reboot                              |         \\\\\         /////     
 |                                         |   ))))))))))))       (((((((((((
 |  Options:                               |         /////         \\\\\     
 |  5. [K]ernel: kernel (1 of 2)           |    @@@@@@@@@@@       @@@@@@@@@@@
 |  6. Configure Boot [O]ptions...         |       @@@@@            @@@@@    
 |                                         |   @@@@@                    @@@@@
 |                                         |   @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
 |                                         |   @@@@@@@@@@@@@@@@@@@@@@@@@@@@  
 +=========================================+                                  
                                                 19.1  ``Inspiring Iguana''   

/boot/kernel/kernel text=0x1406faf data=0xf2af4+0x2abeec syms=[0x4+0xf9f50+0x4+0x1910f5]
/boot/entropy size=0x1000
/boot/kernel/carp.ko text=0x7f90 data=0x374+0x74 syms=[0x4+0xeb0+0x4+0xf40]
/boot/kernel/if_bridge.ko text=0x7b74 data=0x364+0x3c syms=[0x4+0x1020+0x4+0x125f]
loading required module 'bridgestp'
/boot/kernel/bridgestp.ko text=0x4878 data=0xe0+0x18 syms=[0x4+0x6c0+0x4+0x65c]
/boot/kernel/if_enc.ko text=0x1198 data=0x2b8+0x8 syms=[0x4+0x690+0x4+0x813]
/boot/kernel/if_gre.ko text=0x31d8 data=0x278+0x30 syms=[0x4+0xa30+0x4+0xab8]
<snip>
Root file system: /dev/ufs/OPNsense
Sun May 19 10:28:00 UTC 2019

*** OPNsense.stafnet: OPNsense 19.1.6 (i386/OpenSSL) ***
<snip>

 HTTPS: SHA256 E8 9F B2 8B BE F9 D7 2D 00 AD D3 D5 60 E3 77 53
               3D AC AB 81 38 E4 D2 75 9E 04 F9 33 FF 76 92 28
 SSH:   SHA256 FbjqnefrisCXn8odvUSsM8HtzNNs+9xR/mGFMqHXjfs (ECDSA)
 SSH:   SHA256 B6R7GRL/ucRL3JKbHL1OGsdpHRDyotYukc77jgmIJjQ (ED25519)
 SSH:   SHA256 8BOmgp8lSFF4okrOUmL4YK60hk7LTg2N08Hifgvlq04 (RSA)

FreeBSD/i386 (OPNsense.stafnet) (ttyu0)

login: 
root@OPNsense:~ # df -ih
Filesystem           Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufs/OPNsense    3.5G    1.2G    2.0G    39%     38k  943k    4%   /
devfs                1.0K    1.0K      0B   100%       0     0  100%   /dev
tmpfs                307M     15M    292M     5%     203  2.1G    0%   /var
tmpfs                292M     88K    292M     0%      27  2.1G    0%   /tmp
devfs                1.0K    1.0K      0B   100%       0     0  100%   /var/unbound/dev
devfs                1.0K    1.0K      0B   100%       0     0  100%   /var/dhcpd/dev
root@OPNsense:~ # 
CTRL-A Z for help | 115200 8N1 | NOR | Minicom 2.7.1 | VT102 | Offline | ttyUSB0                                    

update

login: root
Password:
----------------------------------------------
|      Hello, this is OPNsense 19.1          |         @@@@@@@@@@@@@@@
|                                            |        @@@@         @@@@
| Website:      https://opnsense.org/        |         @@@\\\   ///@@@
| Handbook:     https://docs.opnsense.org/   |       ))))))))   ((((((((
| Forums:       https://forum.opnsense.org/  |         @@@///   \\\@@@
| Lists:        https://lists.opnsense.org/  |        @@@@         @@@@
| Code:         https://github.com/opnsense  |         @@@@@@@@@@@@@@@
----------------------------------------------

*** OPNsense.stafnet: OPNsense 19.1.6 (i386/OpenSSL) ***
<snip>

 HTTPS: SHA256 E8 9F B2 8B BE F9 D7 2D 00 AD D3 D5 60 E3 77 53
               3D AC AB 81 38 E4 D2 75 9E 04 F9 33 FF 76 92 28
 SSH:   SHA256 FbjqnefrisCXn8odvUSsM8HtzNNs+9xR/mGFMqHXjfs (ECDSA)
 SSH:   SHA256 B6R7GRL/ucRL3JKbHL1OGsdpHRDyotYukc77jgmIJjQ (ED25519)
 SSH:   SHA256 8BOmgp8lSFF4okrOUmL4YK60hk7LTg2N08Hifgvlq04 (RSA)

  0) Logout                              7) Ping host
  1) Assign interfaces                   8) Shell
  2) Set interface IP address            9) pfTop
  3) Reset the root password            10) Firewall log
  4) Reset to factory defaults          11) Reload all services
  5) Power off system                   12) Update from console
  6) Reboot system                      13) Restore a backup

Enter an option: 12

Fetching change log information, please wait... done

This will automatically fetch all available updates, apply them,
and reboot if necessary.

This update requires a reboot.

Proceed with this action? [y/N]: y 

** Have fun! **

Links

In his post Iconic consoles of the IBM System/360 mainframes, 55 years old, Ken Shirrif gives a beautiful overview of how IBM mainframes were operated.

I particularly liked this bit:

The second console function was “operator intervention”: program debugging tasks such as examining and modifying memory or registers and setting breakpoints. The Model 30 console controls below were used for operator intervention. To display memory contents, the operator selected an address with the four hexadecimal dials on the left and pushed the Display button, displaying data on the lights above the dials. To modify memory, the operator entered a byte using the two hex dials on the far right and pushed the Store button. (Although the Model 30 had a 32-bit architecture, it operated on one byte at a time, trading off speed for lower cost.) The Address Compare knob in the upper right set a breakpoint.

IBM System/360 Model 30 console, lower part

Debugging a program was built right into the hardware, to be performed at the console of the machine. Considering the fact that these machines were usually placed in rooms optimized for the machine rather than the human, that must have been a difficult job. Think about that the next time you’re poking at a Kubernetes cluster using your laptop, in the comfort of your home.

Also recommended is the book Core Memory: A Visual Survey of Vintage Computers. It really shows the intricate beauty of some of the earlier computers. It also shows how incredibly cumbersome these machines must have been to handle.

Core Memory: A Visual Survey of Vintage Computers

Even when you’re in IT operations, it’s getting more and more rare to see actual hardware and that’s probably a good thing. It never hurts to look at history to get a taste of how far we’ve come. Life in operations has never been more comfortable: let’s enjoy it by celebrating the past!


Comments | More on rocketeer.be | @rubenv on Twitter

May 16, 2019

I published the following diary on isc.sans.edu: “The Risk of Authenticated Vulnerability Scans“:

NTLM relay attacks have been a well-known opportunity to perform attacks against Microsoft Windows environments for a while and they remain usually successful. The magic with NTLM relay attacks? You don’t need to lose time to crack the hashes, just relay them to the victim machine. To achieve this, we need a “responder” that will capture the authentication session on a system and relay it to the victim. A lab is easy to setup: Install the Responder framework. The framework contains a tool called MultiRelay.py which helps to relay the captured NTLM authentication to a specific target and, if the attack is successful, execute some code! (There are plenty of blog posts that explain in details how to (ab)use of this attack scenario)… [Read more]

[The post [SANS ISC] The Risk of Authenticated Vulnerability Scans has been first published on /dev/random]

May 13, 2019

I published the following diary on isc.sans.edu: “From Phishing To Ransomware?“:

On Friday, one of our readers reported a phishing attempt to us (thanks to him!). Usually, those emails are simply part of classic phishing waves and try to steal credentials from victims but, this time, it was not a simple phishing. Here is a copy of the email, which was nicely redacted… [Read more]

[The post [SANS ISC] From Phishing To Ransomware? has been first published on /dev/random]

May 12, 2019

fedora_logo_small.png

In my previous two posts (1, 2 ), we created Docker Debian and Arch-based images from scratch for the i386 architecture.

In this blog post - last one in this series - we’ll do the same for yum based distributions like CentOS and Fedora.

Building your own Docker base images isn’t difficult and let you trust your distribution Gpg signing keys instead of the docker hub. As explained in the first blog post. The mkimage scripts in the contrib directory of the Moby project git repository is a good place to start if you want to build own docker images.

centos_logo_small.png

Fedora is one of the GNU/Linux distributions that supports 32 bits systems. Centos has a Special Interest Groups to support alternative architectures. The Alternative Architecture SIG create installation images for power, i386, armhfp (arm v732 bits) and aarch64 (arm v8 64-bit).

Centos

In this blog post, we will create centos based docker images. The procedure to create Fedora images is the same.

Clone moby

staf@centos386 github]$ git clone https://github.com/moby/moby
Cloning into 'moby'...
remote: Enumerating objects: 7, done.
remote: Counting objects: 100% (7/7), done.
remote: Compressing objects: 100% (7/7), done.
remote: Total 269517 (delta 0), reused 1 (delta 0), pack-reused 269510
Receiving objects: 100% (269517/269517), 139.16 MiB | 3.07 MiB/s, done.
Resolving deltas: 100% (182765/182765), done.
[staf@centos386 github]$ 

Go to the contrib directory

[staf@centos386 github]$ cd moby/contrib/
[staf@centos386 contrib]$ 

mkimage-yum.sh

When you run mkimage-yum.sh you get the usage message.

[staf@centos386 contrib]$ ./mkimage-yum.sh 
mkimage-yum.sh [OPTIONS] <name>
OPTIONS:
  -p "<packages>"  The list of packages to install in the container.
                   The default is blank. Can use multiple times.
  -g "<groups>"    The groups of packages to install in the container.
                   The default is "Core". Can use multiple times.
  -y <yumconf>     The path to the yum config to install packages from. The
                   default is /etc/yum.conf for Centos/RHEL and /etc/dnf/dnf.conf for Fedora
  -t <tag>         Specify Tag information.
                   default is reffered at /etc/{redhat,system}-release
[staf@centos386 contrib]$

build the image

The mkimage-yum.sh script will use /etc/yum.conf or /etc/dnf.conf to build the image. mkimage-yum.sh <name> will create the image with name.

[staf@centos386 contrib]$ sudo ./mkimage-yum.sh centos
[sudo] password for staf: 
+ mkdir -m 755 /tmp/mkimage-yum.sh.LeZQNh/dev
+ mknod -m 600 /tmp/mkimage-yum.sh.LeZQNh/dev/console c 5 1
+ mknod -m 600 /tmp/mkimage-yum.sh.LeZQNh/dev/initctl p
+ mknod -m 666 /tmp/mkimage-yum.sh.LeZQNh/dev/full c 1 7
+ mknod -m 666 /tmp/mkimage-yum.sh.LeZQNh/dev/null c 1 3
+ mknod -m 666 /tmp/mkimage-yum.sh.LeZQNh/dev/ptmx c 5 2
+ mknod -m 666 /tmp/mkimage-yum.sh.LeZQNh/dev/random c 1 8
+ mknod -m 666 /tmp/mkimage-yum.sh.LeZQNh/dev/tty c 5 0
+ mknod -m 666 /tmp/mkimage-yum.sh.LeZQNh/dev/tty0 c 4 0
+ mknod -m 666 /tmp/mkimage-yum.sh.LeZQNh/dev/urandom c 1 9
+ mknod -m 666 /tmp/mkimage-yum.sh.LeZQNh/dev/zero c 1 5
+ '[' -d /etc/yum/vars ']'
+ mkdir -p -m 755 /tmp/mkimage-yum.sh.LeZQNh/etc/yum
+ cp -a /etc/yum/vars /tmp/mkimage-yum.sh.LeZQNh/etc/yum/
+ [[ -n Core ]]
+ yum -c /etc/yum.conf --installroot=/tmp/mkimage-yum.sh.LeZQNh --releasever=/ --setopt=tsflags=nodocs --setopt=group_package_types=mandatory -y groupinstall Core
Loaded plugins: fastestmirror, langpacks
There is no installed groups file.
Maybe run: yum groups mark convert (see man yum)
<snip>
+ tar --numeric-owner -c -C /tmp/mkimage-yum.sh.LeZQNh .
+ docker import - centos:7.6.1810
sha256:7cdb02046bff4c5065de670604fb3252b1221c4853cb4a905ca04488f44f52a8
+ docker run -i -t --rm centos:7.6.1810 /bin/bash -c 'echo success'
success
+ rm -rf /tmp/mkimage-yum.sh.LeZQNh
[staf@centos386 contrib]$

Rename

A new image is created with the name centos.

[staf@centos386 contrib]$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos              7.6.1810            7cdb02046bff        3 minutes ago       281 MB
[staf@centos386 contrib]$ 

You might want to rename to include your name or project name. You can do this by retag the image and remove the old image name.

[staf@centos386 contrib]$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos              7.6.1810            7cdb02046bff        20 seconds ago      281 MB
[staf@centos386 contrib]$ docker rmi centos
Error response from daemon: No such image: centos:latest
[staf@centos386 contrib]$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
centos              7.6.1810            7cdb02046bff        3 minutes ago       281 MB
[staf@centos386 contrib]$ docker tag 7cdb02046bff stafwag/centos_386:7.6.1810 
[staf@centos386 contrib]$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
centos               7.6.1810            7cdb02046bff        7 minutes ago       281 MB
stafwag/centos_386   7.6.1810            7cdb02046bff        7 minutes ago       281 MB
[staf@centos386 contrib]$ docker rmi centos:7.6.1810
Untagged: centos:7.6.1810
[staf@centos386 contrib]$ docker images
REPOSITORY           TAG                 IMAGE ID            CREATED             SIZE
stafwag/centos_386   7.6.1810            7cdb02046bff        8 minutes ago       281 MB
[staf@centos386 contrib]$ 

Test

[staf@centos386 contrib]$ docker run -it --rm stafwag/centos_386:7.6.1810 /bin/sh
sh-4.2# yum update -y
Loaded plugins: fastestmirror
Determining fastest mirrors
 * base: mirror.usenet.farm
 * extras: mirror.usenet.farm
 * updates: mirror.usenet.farm
base                                                                                                                   | 3.6 kB  00:00:00     
extras                                                                                                                 | 2.9 kB  00:00:00     
updates                                                                                                                | 2.9 kB  00:00:00     
(1/4): updates/7/i386/primary_db                                                                                       | 2.5 MB  00:00:00     
(2/4): extras/7/i386/primary_db                                                                                        | 157 kB  00:00:01     
(3/4): base/7/i386/group_gz                                                                                            | 166 kB  00:00:01     
(4/4): base/7/i386/primary_db                                                                                          | 4.6 MB  00:00:02     
No packages marked for update
sh-4.2# 

** Have fun! **

May 10, 2019

I published the following diary on isc.sans.edu: “DSSuite – A Docker Container with Didier’s Tools“:

If you follow us and read our daily diaries, you probably already know some famous tools developed by Didier (like oledump.py, translate.py and many more). Didier is using them all the time to analyze malicious documents. His tools are also used by many security analysts and researchers. The complete toolbox is available on his github.com page. You can clone the repository or download the complete package available as a zip archive. However, it’s not convenient to install them all the time when you’re switching from computers all the time if, like me, you’re always on the road between different customers… [Read more]

[The post [SANS ISC] DSSuite – A Docker Container with Didier’s Tools has been first published on /dev/random]

May 08, 2019

Acquia joins forces with Mautic

I'm happy to announce today that Acquia acquired Mautic, an open source marketing automation and campaign management platform.

A couple of decades ago, I was convinced that every organization required a website — a thought that sounds rather obvious now. Today, I am convinced that every organization will need a Digital Experience Platform (DXP).

Having a website is no longer enough: customers expect to interact with brands through their websites, email, chat and more. They also expect these interactions to be relevant and personalized.

If you don't know Mautic, think of it as an alternative to Adobe's Marketo or Salesforce's Marketing Cloud. Just like these solutions, Mautic provides marketing automation and campaign management capabilities. It's differentiated in that it is easier to use, supports one-to-one customer experiences across many channels, integrates more easily with other tools, and is less expensive.

The flowchart style visual campaign builder you saw in the beginning of the Mautic demo video above is one of my favorite features. I love how it allows marketers to combine content, user profiles, events and a decision engine to deliver the best-next action to customers.

Mautic is a relatively young company, but has quickly grown into the largest open source player in the marketing automation space, with more than 200,000 installations. Its ease of use, flexibility and feature completeness has won over many marketers in a very short time: the company's top-line grew almost 400 percent year-over-year, its number of customers tripled, and Mautic won multiple awards for product innovation and customer service.

The acquisition of Mautic accelerates Acquia's product strategy to deliver the only Open Digital Experience Platform:

The building blocks of a Digital Experience Platform and how Mautic accelerates Acquia's vision.  The pieces that make up a Digital Experience Platform, and how Mautic fits into Acquia's Open Digital Experience Platform. Acquia is strong in content management, personalization, user profile management and commerce (yellow blocks). Mautic adds or improves Acquia's multi-channel delivery, campaign management and journey orchestration capabilities (purple blocks).

There are many reasons why we like Mautic, but here are my top 3:

Reason 1: Disrupting the market with "open"

Open Source will disrupt every component of the modern technology stack. It's not a matter of if, it's when.

Just as Drupal disrupted web content management with Open Source, we believe Mautic disrupts marketing automation.

With Mautic, Acquia is now the only open and open source alternative to the expensive, closed, and stagnant marketing clouds.

I'm both proud and excited that Acquia is doubling down on Open Source. Given our extensive open source experience, we believe we can help grow Mautic even faster.

Reason 2: Innovating through integrations

To build an optimal customer experience, marketers need to integrate with different data sources, customer technologies, and bespoke in-house platforms. Instead of buying a suite from a single vendor, most marketers want an open platform that allows for open innovation and unlimited integrations.

Only an open architecture can connect any technology in the marketing stack, and only an open source innovation model can evolve fast enough to offer integrations with thousands of marketing technologies (to date, there are 7,000 vendors in the martech landscape).

Because developers are largely responsible for creating and customizing marketing platforms, marketing technology should meet the needs of both business users and technology architects. Unlike other companies in the space, Mautic is loved by both marketers and developers. With Mautic, Acquia continues to focus on both personas.

Reason 3: The same technology stack and business model

Like Drupal, Mautic is built in PHP and Symfony, and like Drupal, Mautic uses the GNU GPL license. Having the same technology stack has many benefits.

Digital agencies or in-house teams need to deliver integrated marketing solutions. Because both Drupal and Mautic use the same technology stack, a single team of developers can work on both.

The similarities also make it possible for both open source communities to collaborate — while it is not something you can force to happen, it will be interesting to see how that dynamic naturally plays out over time.

Last but not least, our business models are also very aligned. Both Acquia and Mautic were "born in the cloud" and make money by offering subscription- and cloud-based delivery options. This means you pay for only what you need and that you can focus on using the products rather than running and maintaining them.

Mautic offers several commercial solutions:

  • Mautic Cloud, a fully managed SaaS version of Mautic with premium features not available in Open Source.
  • For larger organizations, Mautic has a proprietary product called Maestro. Large organizations operate in many regions or territories, and have teams dedicated to each territory. With Maestro, each territory can get its own Mautic instance, but they can still share campaign best-practices, and repeat successful campaigns across territories. It's a unique capability, which is very aligned with the Acquia Cloud Site Factory.

Try Mautic

If you want to try Mautic, you can either install the community version yourself or check out the demo or sandbox environment of Mautic Open Marketing Cloud.

Conclusion

We're very excited to join forces with Mautic. It is such a strategic step for Acquia. Together we'll provide our customers with more freedom, faster innovation, and more flexibility. Open digital experiences are the way of the future.

I've got a lot more to share about the Mautic acquisition, how we plan to integrate Mautic in Acquia's solutions, how we could build bridges between the Drupal and Mautic community, how it impacts the marketplace, and more.

In time, I'll write more about these topics on this blog. In the meantime, you can listen to this podcast with DB Hurley, Mautic's founder and CTO, and me.

May 07, 2019

A couple of weeks ago our first-born daughter appeared into my life. All the clichés of what this miracle does with a man are very well true. Not only is this (quite literally) the start of a new life, it also gives you a pause to reflect on your own life.

Around the same time I’ve finished working on the project that has occupied most of my time over the past years: helping a software-as-a-service company completely modernize and rearchitect their software stack, to help it grow further in the coming decade.

Going forward, I’ll be applying the valuable lessons learned while doing this, combined with all my previous experiences, as a consultant. More specifically I’ll be focusing on DevOps and related concerns. More information on that can be found on this page.

I also have a new business venture in the works, but that’s the subject of a future post.


Comments | More on rocketeer.be | @rubenv on Twitter

May 05, 2019

In my previous post, we started with creating Debian based docker images from scratch for the i386 architecture.

In this blog post, we’ll create Arch GNU/Linux based images.

Arch GNU/Linux

Arch Linux stopped supporting i386 systems. When you want to run Archlinux on an i386 system there is a community maintained Archlinux32 project and the Free software version Parabola GNU/Linux-libre.

For the arm architecture, there is Archlinux Arm project that I used.

mkimage-arch.sh in moby

I used mkimage-arch.sh from the Moby/Docker project in the past, but it failed when I tried it this time…

I created a small patch to fix it and created a pull request. Till the issue is resolved, you can use the version in my cloned git repository.

Build the docker image

Install the required packages

Make sure that your system is up-to-date.

staf@archlinux32 contrib]$ sudo pacman -Syu

Install the required packages.

[staf@archlinux32 contrib]$ sudo pacman -S arch-install-scripts expect wget

Directory

Create a directory that will hold the image data.

[staf@archlinux32 ~]$ mkdir -p dockerbuild/archlinux32
[staf@archlinux32 ~]$ cd dockerbuild/archlinux32
[staf@archlinux32 archlinux32]$ 

Get mkimage-arch.sh

[staf@archlinux32 archlinux32]$ wget https://raw.githubusercontent.com/stafwag/moby/master/contrib/mkimage-arch.sh
--2019-05-05 07:46:32--  https://raw.githubusercontent.com/stafwag/moby/master/contrib/mkimage-arch.sh
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.36.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.36.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3841 (3.8K) [text/plain]
Saving to: 'mkimage-arch.sh'

mkimage-arch.sh                 100%[====================================================>]   3.75K  --.-KB/s    in 0s      

2019-05-05 07:46:33 (34.5 MB/s) - 'mkimage-arch.sh' saved [3841/3841]

[staf@archlinux32 archlinux32]$ 

Make it executable.

[staf@archlinux32 archlinux32]$ chmod +x mkimage-arch.sh
[staf@archlinux32 archlinux32]$ 

Setup your pacman.conf

Copy your pacmnan.conf to the directory that holds mkimage-arch.sh.

[staf@archlinux32 contrib]$ cp /etc/pacman.conf mkimage-arch-pacman.conf
[staf@archlinux32 contrib]$ 

Build your image

[staf@archlinux32 archlinux32]$ TMPDIR=`pwd` sudo ./mkimage-arch.sh
spawn pacstrap -C ./mkimage-arch-pacman.conf -c -d -G -i /var/tmp/rootfs-archlinux-wqxW0uxy8X base bash haveged pacman pacman-mirrorlist --ignore dhcpcd,diffutils,file,inetutils,iproute2,iputils,jfsutils,licenses,linux,linux-firmware,lvm2,man-db,man-pages,mdadm,nano,netctl,openresolv,pciutils,pcmciautils,psmisc,reiserfsprogs,s-nail,sysfsutils,systemd-sysvcompat,usbutils,vi,which,xfsprogs
==> Creating install root at /var/tmp/rootfs-archlinux-wqxW0uxy8X
==> Installing packages to /var/tmp/rootfs-archlinux-wqxW0uxy8X
:: Synchronizing package databases...
 core                                              198.0 KiB   676K/s 00:00 [##########################################] 100%
 extra                                               2.4 MiB  1525K/s 00:02 [##########################################] 100%
 community                                           6.3 MiB   396K/s 00:16 [##########################################] 100%
:: dhcpcd is in IgnorePkg/IgnoreGroup. Install anyway? [Y/n] n
:: diffutils is in IgnorePkg/IgnoreGroup. Install anyway? [Y/n] n
:: file is in IgnorePkg/IgnoreGroup. Install anyway? [Y/n] n
<snip>
==> WARNING: /var/tmp/rootfs-archlinux-wqxW0uxy8X is not a mountpoint. This may have undesirable side effects.
Generating locales...
  en_US.UTF-8... done
Generation complete.
tar: ./etc/pacman.d/gnupg/S.gpg-agent.ssh: socket ignored
tar: ./etc/pacman.d/gnupg/S.gpg-agent.extra: socket ignored
tar: ./etc/pacman.d/gnupg/S.gpg-agent: socket ignored
tar: ./etc/pacman.d/gnupg/S.gpg-agent.browser: socket ignored
sha256:41cd9d9163a17e702384168733a9ca1ade0c6497d4e49a2c641b3eb34251bde1
Success.
[staf@archlinux32 archlinux32]$ 

Rename

A new image is created with the name archlinux.

[staf@archlinux32 archlinux32]$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE
archlinux           latest              18e74c4d823c        About a minute ago   472MB
[staf@archlinux32 archlinux32]$ 

You might want to rename it. You can do this by retag the image and remove the old image name.

[staf@archlinux32 archlinux32]$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED              SIZE
archlinux           latest              18e74c4d823c        About a minute ago   472MB
[staf@archlinux32 archlinux32]$ docker tag stafwag/archlinux:386 18e74c4d823c
Error response from daemon: No such image: stafwag/archlinux:386
[staf@archlinux32 archlinux32]$ docker tag 18e74c4d823c stafwag/archlinux:386             
[staf@archlinux32 archlinux32]$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
archlinux           latest              18e74c4d823c        3 minutes ago       472MB
stafwag/archlinux   386                 18e74c4d823c        3 minutes ago       472MB
[staf@archlinux32 archlinux32]$ docker rmi archlinux
Untagged: archlinux:latest
[staf@archlinux32 archlinux32]$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
stafwag/archlinux   386                 18e74c4d823c        3 minutes ago       472MB
[staf@archlinux32 archlinux32]$ 

Test

[staf@archlinux32 archlinux32]$ docker run --rm -it stafwag/archlinux:386 /bin/sh
sh-5.0# pacman -Syu
:: Synchronizing package databases...
 core is up to date
 extra is up to date
 community is up to date
:: Starting full system upgrade...
 there is nothing to do
sh-5.0# 

** Have fun! **

May 04, 2019

The post Enable the RPC JSON API with password authentication in Bitcoin Core appeared first on ma.ttias.be.

The bitcoin daemon has a very useful & easy-to-use HTTP API built-in, that allows you to talk to it like a simple webserver and get JSON responses back.

By default, it's enabled but it only listens on localhost port 8223, and it's unauthenticated.

$ netstat -alpn | grep 8332
tcp     0   0 127.0.0.1:8332     0.0.0.0:*    LISTEN      31667/bitcoind
tcp6    0   0 ::1:8332           :::*         LISTEN      31667/bitcoind

While useful if you're on the same machine (you can query it locally without username/password), it won't help much if you're querying a remote node.

In order to allow bitcoind to bind on a public-facing IP and have username/password authentication, you can modify the bitcoin.conf.

$ cat .bitcoin/bitcoin.conf
# Expose the RPC/JSON API
server=1
rpcbind=10.0.1.5
rpcallowip=0.0.0.0/0
rpcport=8332
rpcuser=bitcoin
rpcpassword=J9JkYnPiXWqgRzg3vAA

If you restart your daemon with this config, it would try to bind to IP "10.0.1.5" and open the RCP JSON API endpoint on its default port 8332. To authenticate, you'd give the user & password as shown in the config.

If you do not pass the rpcallowip parameter, the server won't bind on the requested IP, as confirmed in the manpage:

-rpcbind=[:port]
Bind to given address to listen for JSON-RPC connections. Do not expose
the RPC server to untrusted networks such as the public internet!
This option is ignored unless -rpcallowip is also passed. Port is
optional and overrides -rpcport. Use [host]:port notation for
IPv6. This option can be specified multiple times (default:
127.0.0.1 and ::1 i.e., localhost)

Keep that note that it's a lot safer to actually pass the allowed IPs and treat it as a whitelist, not as a workaround to listen to all IPs like I did above.

Here's an example of a curl call to query the daemon.

$ curl \
  --user bitcoin \
  --data-binary '{"jsonrpc": "1.0", "id":"curltest", "method": "getnetworkinfo", "params": [] }' \ 
  -H 'content-type: text/plain;' \
  http://10.0.1.5:8332/
Enter host password for user 'bitcoin':

{
  "result":
    {
      "version":180000,
      ...
    }
}

You can now safely query your bitcoin daemon with authentication.

The post Enable the RPC JSON API with password authentication in Bitcoin Core appeared first on ma.ttias.be.

May 02, 2019

The post How to upgrade to the latest Bitcoin Core version appeared first on ma.ttias.be.

This guide assumes you've followed my other guide, where you compile Bitcoin Core from source. These steps will allow you to upgrade a running Bitcoin Core node to the latest version.

Stop current Bitcoin Core node

First, stop the running version of the Bitcoin Core daemon.

$ su - bitcoin
$ bitcoin-cli stop
Bitcoin server stopping

Check to make sure your node is stopped.

$ bitcoin-cli status
error: Could not connect to the server 127.0.0.1:8332

Once that's done, download & compile the new version.

Install the latest version of Bitcoin Core

To do so, you can follow all other steps to self-compile Bitcoin Core.

In those steps, there's a part where you do a git checkout to a specific version. Change that to refer to the latest release. In this example, we'll upgrade to 0.18.0.

$ git clone https://github.com/bitcoin/bitcoin.git
$ cd bitcoin
$ git checkout v0.18.0

And compile Bitcoin Core using the same steps as before.

$ ./autogen.sh
$ ./configure
$ make -j $(nproc)

Once done, you have the latest version of Bitcoin Core.

Start the Bitcoin Core daemon

Start it again as normal;

$ bitcoind --version
Bitcoin Core Daemon version v0.18.0
$ bitcoind -daemon
Bitcoin server starting

Check the logs to make sure everything is OK.

$ tail -f ~/.bitcoin/debug.log
Bitcoin Core version v0.18.0 (release build)
Assuming ancestors of block 0000000000000000000f1c54590ee18d15ec70e68c8cd4cfbadb1b4f11697eee have valid signatures.
Setting nMinimumChainWork=0000000000000000000000000000000000000000051dc8b82f450202ecb3d471
Using the 'sse4(1way),sse41(4way),avx2(8way)' SHA256 implementation
Using RdSeed as additional entropy source
Using RdRand as an additional entropy source
Default data directory /home/bitcoin/.bitcoin
...

And you're now running the latest version.

The post How to upgrade to the latest Bitcoin Core version appeared first on ma.ttias.be.

I published the following diary on isc.sans.edu: “Another Day, Another Suspicious UDF File“:

In my last diary, I explained that I found a malcious UDF image used to deliver a piece of malware. After this, I created a YARA rule on VT to try to spot more UDF files in the wild. It seems like the tool ImgBurn is the attacker’s best friend to generate such malicious images. To find more UDF images, I used the following very simple YARA rule… [Read more]

[The post [SANS] Another Day, Another Suspicious UDF File has been first published on /dev/random]

May 01, 2019

The post I forgot how to manage a server appeared first on ma.ttias.be.

Something embarrassing happened to me the other day. I was playing around with a new server on Digital Ocean and it occurred to me: I had no idea how to manage it.

This is slightly awkward because I've been a sysadmin for over 10yrs, the largest part of my professional career.

The spoils of configuration management

The thing is, I've been writing and using config management for the last 6-ish years. I've shared many blogposts regarding Puppet, some of its design patterns, even pretended to hold all the knowledge by sharing my lessons learned after 3yr of using Puppet.

And now I've come to the point where I no longer know how to install, configure or run software without Puppet.

My config management does this for me. Whether it's Puppet, Ansible, Chef, ... all of the boring parts of being a sysadmin have been hidden behind management tools. Yet here I am, trying to quickly configure a personal server, without my company-managed config management to aid me.

Boy did I feel useless.

I had to Google the correct SSH config syntax to allow root logins, but only via public keys. I had to Google for iptables rule syntax and using ufw to manage them. I forgot where I had to place the configs for supervisor for running jobs, let alone how to write the config.

I know these configs in our tools. In our abstractions. In our automation. But I forgot what it's like in Linux itself.

A pitfall to remember

I previously blogged about 2 pitfalls I had already known about: trying to automate a service you don't fully understand or blindly trusting the automation of someone else, not understanding what it's doing under the hood.

I should add a third one to my pitfall list: I'm forgetting the basic and core tools used to manage a Linux server.

Is this a bad thing though? I'm not sure yet. Maybe the lower level knowledge isn't that valuable, as long as we have our automation standby to take care of this? It frees us from having to think about too many things and allows us to focus on the more important aspects of being a sysadmin.

But it sure felt strange Googling things I had to Google almost a decade ago.

The post I forgot how to manage a server appeared first on ma.ttias.be.

April 30, 2019

The Drupal Association announced today that Heather Rocker has been selected as its next Executive Director.

This is exciting news because it concludes a seven month search since Megan Sanicki left.

We looked long and hard for someone who could help us grow the global Drupal community by building on its diversity, working with developers and agency partners, and expanding our work with new audiences such as content creators and marketers.

The Drupal Association (including me) believes that Heather can do all of that, and is the best person to help lead Drupal into its next phase of growth.

Heather earned her engineering degree from Georgia Tech. She has dedicated much of her career to working with women in technology, both as the CEO of Girls, Inc. of Greater Atlanta and the Executive Director of Women in Technology.

We were impressed not only with her valuable experience with volunteer organizations, but also her work in the private sector with large customers. Most recently, Heather was part of the management team at Systems Evolution, a team of 250 business consultants, where she specialized in sales operations and managed key client relationships.

She is also a robotics fanatic who organizes and judges competitions for children. So, maybe we’ll see some robots roaming around DrupalCon in the future!

As you can tell, Heather will bring a lot of great experience to the Drupal community and I look forward to partnering with her.

Last but not least, I want to thank Tim Lehnen for serving as our Interim Executive Director. He did a fantastic job leading the Drupal Association through this transition.

Comment le simple fait d’avoir un compte Facebook m’a rendu injoignable pendant 5 ans pour plusieurs dizaines de lecteurs de mon blog

Ne pas être sur Facebook ou le quitter est souvent sujet au débat : « Mais comment vont faire les gens pour te contacter ? Comment vas-tu rester en contact ? ». Tout semble se réduire au choix cornélien : préserver sa vie privée ou bien être joignable par le commun des mortels.

Je viens de me rendre compte qu’il s’agit d’un faux débat. Tout comme Facebook offre l’illusion de popularité et d’audience à travers les likes, la disponibilité en ligne est illusoire. Pire ! J’ai découvert qu’être sur Facebook m’avait rendu moins joignable pour toute une catégorie de lecteurs de mon blog !

Il y’a une raison toute simple qui me pousse à garder un compte Facebook Messenger : c’est l’endroit où les quelques apnéistes belges organisent leurs sorties. Si je n’y suis pas, je rate les sorties, aussi simple que ça. Du coup, j’ai installé l’application Messenger Lite, dans le seul but de pouvoir aller plonger. Or, en fouillant dans les options de l’app, j’ai découvert une sous-rubrique bien cachée intitulée « Invitations filtrées ».

Là, j’y ai trouvé plusieurs dizaines de messages qui m’ont été envoyés depuis 2013. Plus de 5 années de messages dont j’ignorais l’existence ! Principalement des réactions, pas toujours positives, à mes billets de blogs. D’autres étaient plus factuels. Ils émanaient d’organisateurs de conférences, de personnes que j’avais croisées et qui souhaitaient rester en contact. Tous ces messages, sans exception, auraient eu leur place dans ma boîte mail.

Je ne savais pas qu’ils existaient. À aucun moment Facebook ne m’a signalé l’existence de ces messages, ne m’a donné la chance d’y répondre alors que certains datent d’une époque où j’étais très actif sur ce réseau, où l’app était installée sur mon téléphone.

À toutes ces personnes, Facebook a donné l’illusion qu’elles m’avaient contacté. Que j’étais joignable. À tous ceux qui ont un jour posté un commentaire sous un de mes billets publiés automatiquement, Facebook a donné l’impression d’être en contact avec moi.

À moi, personnage public, Facebook a donné l’illusion qu’on pouvait me joindre, que ceux qui n’utilisaient pas l’email pouvaient me contacter.

Tous, nous avons été trompés.

Il est temps de faire tomber le voile. Facebook n’offre pas un service, il offre une illusion de service. Une illusion qui est peut-être ce que beaucoup cherchent. L’illusion d’avoir des amis, d’avoir une activité sociale, une reconnaissance, un certain succès. Mais si vous ne cherchez pas l’illusion, alors il est temps de fuir Facebook. Ce n’est pas facile, car l’illusion est forte. Tout comme les adorateurs de la Bible prétendent qu’elle est la vérité ultime « car c’est écrit dans la Bible », les utilisateurs de Facebook se sentent entendus, écoutés, car « Facebook me dit que j’ai été vu ».

C’est d’ailleurs le seul et unique but de Facebook. Nous faire croire que nous sommes connectés, peu importe que ce soit vrai ou pas.

Pour chaque contenu posté, l’algorithme Facebook va tenter de trouver les quelques utilisateurs qui ont une probabilité maximale de commenter. Et encourager les autres à cliquer sur le like sans même lire ce qui s’y passe, juste parce que la photo est jolie ou que tel utilisateur like spontanément les posts de tel autre. Au final, une conversation entre 5 individus ponctuée de 30 likes donnera l’impression d’un retentissement national. Exception faite des célébrités, qui récolteront des dizaines de milliers de likes et de messages parce que ce sont des célébrités, peu importe la plateforme.

Facebook nous donne une petite impression de célébrité et de gloriole grâce à quelques likes, Facebook nous donne l’impression de faire partie d’une tribu, d’avoir des relations sociales.

Il est indéniable que Facebook a également des effets positifs, permet des échanges qui n’existeraient pas sans cela. Mais, pour paraphraser Cal Newport dans son livre Digital Minimalism : est-ce que le prix que nous payons n’est pas trop élevé pour les bénéfices que nous en retirons ?

Je rajouterais : tirons-nous vraiment des bénéfices ? Ou bien l’illusion de ceux-ci ?

Photo by Aranka Sinnema on Unsplash

Je suis @ploum, conférencier et écrivain électronique. Si vous avez apprécié ce texte, n'hésitez pas à me soutenir sur Tipeee, Patreon, Paypal, Liberapay ou en millibitcoins 34pp7LupBF7rkz797ovgBTbqcLevuze7LF. Vos soutiens réguliers, même symboliques, sont une réelle motivation et reconnaissance. Merci !

Ce texte est publié sous la licence CC-By BE.