Planet Grep

Planet'ing Belgian FLOSS people

Planet Grep is maintained by Wouter Verhelst. All times are in UTC.

October 18, 2017

Hoe lok je de gepassioneerde computernerds?

  • Zorg ervoor dat ze opleiding krijgen. Ook in zaken die niet technisch zijn. Laat toe dat ze zich verdiepen in dieptechnische zaken. Bv. low level softwareontwikkeling, electronica, en zo verder. Combineer hun (bestaande) kennis met nieuwe toepassingen. Een gepassioneerde (computer)nerd wil een leven lang bijleren en vooral: al hun kennis combineren met andere ideeën;
  • Laat toe dat ze publiek laten zien wie ze zijn en wat ze kunnen. Laat zij die dat graag doen toe dat ze op bv. radio, Internet en TV komen vertellen hoe hun werk maatschappelijk relevant is. Spreek duidelijk af wat wel niet niet geheim moet blijven, uiteraard;
  • Zorg ervoor dat ze met regelmaat naar een hackercon of een andere conference kunnen gaan. Uiteraard zowizo bv. FOSDEM (niet echt een hackercon, maar ga er toch maar met z’n allen naartoe). Maar bv. de CCC conferences in Duitsland, SHA2017 in Nederland, en zo verder. Wees daar in ieder geval, zonder schroom, aanwezig;
  • Organiseer misschien een eigen hackercon in België. Waarom niet?
  • Maak het niet te gemakkelijk om toe te treden. Dat je er 200 nodig hebt wil niet zeggen dat de eerste de beste goed genoeg zijn;
  • Zorg ervoor dat ze goed verdienen. Begrijp dat de privé hen meer biedt dan de overheid;
  • Publiceer met regelmaat (hun) code als open source op bv. github. Bv. een Wireshark plugin of log analysetools die onze overheid gebruikt? Laat ze helpen met andere open source projecten. Kijk bv. naar hoe we onze eID software (FireFox plugins, e.d.) publiceren;
  • We hebben veel kennis van encryptie in onze universiteiten (Rijndael), stuur ze op cursus daarover bij onze cryptografen;
  • Zorg ervoor dat onze diensten géén fouten maken tegen de Belgische wetgeving. Alle echte goei zijn zo idealistisch als Edward Snowden en willen goed doen voor de samenleving. M.a.w. De wet, de privacy commissie en het Comité I doen er toe.

Veel success. Ik ben erg benieuwd.

October 17, 2017

Hack.lu is ongoing in Luxembourg, already the thirteen edition! I arrived yesterday to attend a pre-conference event: the MISP summit. Today the regular talks were scheduled. It seems that more attendees joined this edition. The number of talks scheduled is impressive this year: 11 talks today and 12 talks on Wednesday and Thursday… Here is my wrap-up of the first day!

The first talk was not technical but very informative: “Myths and realities of attribution manipulation” presented by Félix Aimé & Ronan Mouchoux from Kaspersky. Many companies put more and more efforts in infowar instead of simple malware research. This affects many topics: cyber espionage, mass opinion manipulation or sabotage. The key is to perform attribution by putting a name on a cyber attack. You can see it as putting a tag on an attack. Note that sometimes, attribution suffers from a lack of naming convention like in the AV industry. Not easy to recognise the different actors. To perform this huge task, a lot of time and skills are required. They are many indicators available but many of them can be manipulated (ex: the country of origin, the C2, …). After a definition of attribution and the associated risks, Félix & Ronan reviewed some interesting examples:

  • The case of Turkey.TR domains that were DDoS after the Russian planes crashed
  • The case of Belgium accused to have done an airstrike against the locality of Hassadjek. A few days later, some Belgian media websites were DDoS’d.
As a conclusion to the talk, I like the quote: “You said fileless malware? APT actors try now to be less actor”.

The second slot was assigned to Sébastien (blotus) Blot, Thibault (buixor) Koechlin, Julien (jvoisin) Voisin who presented their solution to improve the security of PHP websites: Snuffleupagus (don’t ask me to pronounce it ;-). The complete title was: “Snuffleupagus – Killing bugclasses in PHP 7, virtual-patching the rest”. The speakers are working for a company provided hosting services and many of their customers are using PHP websites. Besides the classic security controls (OS-level hardening, custom IDS, WAF, …) they searched for a tool to improve the security of PHP. Suhosin is a nice solution but it does not support PHP7. So they decided to write their own tool: Snuffleupagus. They reviewed how to protect PHP with very nice features like the disable_function() feature. Some examples:

sp.disable_function.function(“system”).filename(“foo.php”).allow();
sp.disable_function.function(“system”).filename(“foo.php”).hash(“xxxx”).allow();

You can also restrict parameters passed to functions:

… param(“command”).value_r(“[$|…”).drop();

Then, the speakers demonstrated real vulnerabilities in a well-known tool written in PHP and how their solution could mitigate the vulnerabilities. This is a really nice project still in development but already used by many websites from the Alexa top-ranking list! The project is available here.

After a coffee break, Bouke van Leathem presented his project: “Randori”. In Japanse, Randori is a form of practice in which a designated aikidoka defends against multiple attackers in quick succession. To make it short, it’s the principle of action-reaction: You scan me, I scan you. Randori is a low interaction honeypot with a vengeance as defined by Bouke. The main idea is to reuse the credentials tested by the attackers against themselves. Bouke explained how it developed his honeypot, mainly the pam_randori PAM module. Collected credentials are re-used, no more no less, no code is executed on the remote system. Based on the collected information, Bouke explained in the second part of his talk, how he generated useful statistics to build botnet maps. One of the tools he used for this purpose is ssdeep. Note that the tool can be used in different ways: from an incident responder or ethical hacker perspectives. This project is very interesting and is also available here.

Before the lunch break, we had a keynote. The slot was assigned to Sarah Jamie Lewis and had the title: “Queer Privacy & Building Consensual Systems”. She started with a nice definition of privacy: “Privacy is the right to share information about you… only with people you trust”. Sarah wrote a book (with the same name as her keynote) and used it to illustrate her keynote. She read samples stories about Kath, Ada, Morgan. All those people had privacy issues and have to protect themselves. During the keynote, Sarah looked really affected by those stories but was it the right place to read some samples? I’m not sure. It looks to be a book that you have to read at home, relaxed and not at a security conference (just my $0.02). About privacy, as usual, the facts reviewed during the keynote were the same: our privacy is always threatened and there is a clear lack of solution.

After the lunch, a first lightning talk session was organized followed by Raúl B. Netto’s presentation: “ManaTI: Web Assistance for the Threat Analyst, supported by Domain Similarity”. ManaTI is a project to use machine learning techniques to assist an intuitive threat analyst to help in the discovery of security issues. I missed this talk because I was out with friends.

Then Paul Rascagnères, a regular speaker at hack.lu, came to present tools and techniques to help in debugging malware code written in .Net. This framework is the key component of many Microsoft tools like Powershell. With a nice integration with the operating system, it is also used by bad guys to produce malicious code. Paul started by explained some .Net techniques used by malware (like Assembly.load()). The next part of the talk focused on PYKD, a Python extension for the WinDBG debugger. In a demo, Paul demonstrated how easy it is to use PYKD to debug malicious code.

The next talk was my preferred for this first day: “Device sensors meet the web – a story of sadness and regret” by Lukasz Olejnik. The idea behind this talk was to demonstrate how our privacy can be affected by connected devices or, simply, our browsers. All devices today handle plenty of personal data but web technologies were not designed with privacy in mind. With the modern web, a browser on your smartphone can take advantage of many sensors or connectivity (USB, NFC or Bluetooth). Modern devices have an API that can be queried by web browsers. The first example that Lukasz gave was the batteries. The power level can be queried from a browser. That’s a nice feature indeed but what about privacy issues? Firebox, by abusing the high precision readout can get useful information about the user behaviour. There are also evil scenarios: Just imagine that somebody is looking for a taxi and his /her battery is almost dead. The idea is to go back asap to home. If the taxi reservation page proposes 2 prices: 10€ for a 10 minutes drive and 5€ for a 30 minutes drive, guess which one will be chosen by the customer? Another example, even crazier, was the (ab)use of the light sensor in mobile phones. Lucasz demonstrated how it is possible to steal the browser history via the light sensor: The display emits light that reflects on objects and can be read/decoded. Scary! And other examples are multiple: tracking, behaviour, fingerprinting, etc… How to mitigate this? Not easy, ask permission to the user to access the data, disable the API, purge it from dangerous calls? Finally, Lucasz gave the last example with web payments (in one click) that also have security issues. This was a very nice talk with plenty of examples that should really open our eyes!

After the afternoon coffee break, Maxime Clementz and Antoine Goichot came on stage to present: “Malicious use of Microsoft Local Administrator Password Solution”. The local admin problem is not new with Microsoft operating systems. This account must be present and, within old environments, the password was often the same across all devices in the domain. This makes lateral movement so easy! To solve this issues, Microsoft implemented LAPS or “Local Administrator Password Solution”. How does it work? Random passwords are generated for the local admin. The goal of the talk was to explain how to perform privilege escalation within an environment that has LAPS deployed. In fact, this tools is not new. It was an open source project that was integrated into Microsoft Windows, a client-side extension (CSE). It’s just a DLL called AdmPwd.dll. First observation: the DLL is not signed and does not implement integrity checks. The idea of the PoC was to create a rogue DLL that ignores the temporary password expiration data and write generated passwords in a simple text file. It worked very well. Their recommendation to mitigate this kind of attack: validate the integrity/signature of the DLL.

The next presentation was about cars: “The Bicho: An Advanced Car Backdoor Maker” by Sheila Ayelen Berta. If we see more and more talks about connected cars, this time, it focused on “regular” cars that just have a CAN bus. Sheila explained the tools and hardware that helps to inject commands on a CAN bus. To achieve this, she used a platform called CANspy to sniff messages on a CAN bus. Then, via another tool called “Car Backdoor Maker 1.0”, she was able to generate CAN bus message. Basically, it’s a replay attack. A website has been created to list all CAB messages discovered: opencandb.online. The payload is injected using a microcontroller connected to the CAN bus. It also has GPS capabilities that allow sending the CAN bus message depending on the cat localisation! The payload generator is available here.

Then, we came back to the issues regarding sharing information. Becky Kazansky presented: “Countering Security Threats by Sharing Information: Emerging Civil Society Practices”. I skipped this talk.

Finally, the first day finished with Parth Suhkla who presented “Intel AMT: Using & Abusing the Ghost in the Machine”. The presentation started with an overview of the AMT technology. It means “Active Management Technology” and is an out-of-band, management platform, embedded into Intel chipsets. The goal is to offer remote management capabilities without any OS. You can imagine that this feature looks juicy to attackers! Parth reviewed the core features of AMT and how it works. One important step is the provisioning options (can be performed via a local agent, remotely, via USB or the BIOS). There was already vulnerabilities discovered in AMT like the INTEL-SA-00075 that covered a privilege escalation issue. AMT was also used by the PLATINIUM attacker group who used the Serial Over LAN as a back channel. In the second part, Parth explained his research: how to abuse AMT? The requirements of the attack were:

  • Control the AMT
  • Implement persistence
  • Be stealthy
He reviewed all the possible scenarios with a rating (complex, easy, …). For each attack, if also explained how to mitigate and detect such attacks. Some interesting ideas:
  • Detect usual AMT ports in the network traffic
  • Query the ME interface for AMT status (easy on Windows, no tool for Linux)
  • Verify the book chain
  • Encrypt disk drives with the TPM chipset
  • Protect your BIOS (you already did it right?)
The last part covered the forensics investigations related to AMT. Again, an interesting talk.
That’s all for today! Note that talks have been recorded and are already available on Youtube! After our classic “Belgian dinner”, it’s time to take some hours of sleep, tomorrow 12 talks are scheduled! Stay tuned for more wrap-ups!

[The post Hack.lu 2017 Wrap-Up Day 1 has been first published on /dev/random]

October 16, 2017

The post Compile PHP from source: error: utf8_mime2text() has new signature appeared first on ma.ttias.be.

It's been a while, but I had to recompile a PHP from source and ran into this problem during the ./configure stage.

$ ./configure
...
checking for IMAP Kerberos support... no
checking for IMAP SSL support... yes
checking for utf8_mime2text signature... new
checking for U8T_DECOMPOSE...
configure: error: utf8_mime2text() has new signature, but U8T_CANONICAL is missing.
This should not happen. Check config.log for additional information.

To resolve that utf8_mime2text() has new signature, but U8T_CANONICAL is missing error, on CentOS you can install the libc-client-devel package.

$ yum install libc-client-devel

After that, your ./configure should go through.

The post Compile PHP from source: error: utf8_mime2text() has new signature appeared first on ma.ttias.be.

In this post I demonstrate an effective way to create iterators and generators in PHP and provide an example of a scenario in which using them makes sense.

Generators have been around since PHP 5.5, and iterators have been around since the Planck epoch. Even so, a lot of PHP developers do not know how to use them well and cannot recognize situations in which they are helpful. In this blog post I share insights I have gained over the years, that when sharing, always got an interested response from colleague developers. The post goes beyond the basics, provides a real world example, and includes a few tips and tricks. To not leave out those unfamiliar with Iterators the post starts with the “What are Iterators” section, which you can safely skip if you can already answer that question.

What are Iterators

PHP has an Iterator interface that you can implement to represent a collection. You can loop over an instance of an Iterator just like you can loop over an array:

function doStuff(Iterator $things) {
    foreach ($things as $thing) { /* ... */ }
}

Why would you bother implementing an Iterator subclass rather than just using an array? Let’s look at an example.

Imagine you have a directory with a bunch of text files. One of the files contains an ASCII NyanCat (~=[,,_,,]:3). It is the task of our code to find which file the NyanCat is hiding in.

We can get all the files by doing a glob( $path . '*.txt' ) and we can get the contents for a file with a file_get_contents. We could just have a foreach going over the glob result that does the file_get_contents. Luckily we realize this would violate separation of concerns and make the “does this file contain NyanCat” logic hard to test since it will be bound to the filesystem access code. Hence we create a function that gets the contents of the files, and ones with our logic in it:

function getContentsOfTextFiles(): array {
    // glob and file_get_contents
}

function findTextWithNyanCat(array $texts) {
    foreach ($texts as $text) { if ( /* ... */ ) { /* ... */ } }
}

function findNyanCat() {
    findTextWithNyanCat(getContentsOfTextFiles());
}

While this approach is decoupled, a big drawback is that now we need to fetch the contents of all files and keep all of that in memory before we even start executing any of our logic. If NyanCat is hiding in the first file, we’ll have fetched the contents of all others for nothing. We can avoid this by using an Iterator, as they can fetch their values on demand: they are lazy.

class TextFileIterator implements Iterator {
    /* ... */
    public function current() {
        // return file_get_contents
    }
    /* ... */
}

function findTextWithNyanCat(Iterator $texts) {
    foreach ($texts as $text) { if ( /* ... */ ) { /* ... */ } }
}

function findNyanCat() {
    findTextWithNyanCat(new TextFileIterator());
}

Our TextFileIterator gives us a nice place to put all the filesystem code, while to the outside just looking like a collection of texts. The function housing our logic, findTextWithNyanCat, does not know that the text comes from the filesystem. This means that if you decide to get texts from the database, you could just create a new DatabaseTextBlobIterator and pass it to the logic function without making any changes to the latter. Similarly, when testing the logic function, you can give it an ArrayIterator.

function testFindTextWithNyanCat() {
    /* ... */
    findTextWithNyanCat(new ArrayIterator(['test text', '~=[,,_,,]:3']));
    /* ... */
}

I wrote more about basic Iterator functionality in Lazy iterators in PHP and Python and Some fun with iterators. I also blogged about a library that provides some (Wikidata specific) iterators and a CLI tool build around an Iterator. For more on how generators work, see the off-site post Generators in PHP.

PHP’s collection type hierarchy

Let’s start by looking at PHP’s type hierarchy for collections as of PHP 7.1. These are the core types that I think are most important:

  •  iterable
    • array
    • Traversable
      • Iterator
        • Generator
      • IteratorAggregate

At the very top we have iterable, the supertype of both array and Traversable. If you are not familiar with this type or are using a version of PHP older than 7.1, don’t worry, we don’t need it for the rest of this blog post.

Iterator is the subtype of Traversable, and the same goes for IteratorAggregate. The standard library iterator_ functions such as iterator_to_array all take a Traversable. This is important since it means you can give them an IteratorAggregate, even though it is not an Iterator. Later on in this post we’ll get back to what exactly an IteratorAggregate is and why it is useful.

Finally we have Generator, which is a subtype of Iterator. That means all functions that accept an Iterator can be given a Generator, and, by extension, that you can use generators in combination with the Iterator classes in the Standard PHP Library such as LimitIterator and CachingIterator.

IteratorAggregate + Generator = <3

Generators are a nice and easy way to create iterators. Often you’ll only loop over them once, and not have any problem. However beware that generators create iterators that are not rewindable, which means that if you loop over them more than once, you’ll get an exception.

Imagine the scenario where you pass in a generator to a service that accepts an instance of Traversable:

$aGenerator = function() { /* ... yield ... */ };
$aService->doStuff($aGenerator());

public function doStuff(Traversable $things) {
    foreach ($things as $thing) { /* ... */ }
}

The service class in which doStuff resides does not know it is getting a Generator, it just knows it is getting a Traversable. When working on this class, it is entirely reasonable to iterate though $things a second time.

public function doStuff(Traversable $things) {
    foreach ($things as $thing) { /* ... */ }
    foreach ($things as $thing) { /* ... */ } // Boom if Generator!
}

This blows up if the provided $things is a Generator, because generators are non-rewindable. Note that it does not matter how you iterate through the value. Calling iterator_to_array with $things has the exact same result as using it in a foreach loop. Most, if not all, generators I have written, do not use resources or state that inherently prevents them from being rewindable. So the double-iteration issue can be unexpected and seemingly silly.

There is a simple and easy way to get around it though. This is where IteratorAggregate comes in. Classes implementing IteratorAggregate must implement the getIterator() method, which returns a Traversable. Creating one of these is extremely trivial:

class AwesomeWords implements \IteratorAggregate {
    public function getIterator() {
        yield 'So';
        yield 'Much';
        yield 'Such';
    }
}

If you call getIterator, you’ll get a Generator instance, just like you’d expect. However, normally you never call this method. Instead you use the IteratorAggregate just as if it was an Iterator, by passing it to code that expects a Traversable. (This is also why usually you want to accept Traversable and not just Iterator.) We can now call our service that loops over the $things twice without any problem:

$aService->doStuff(new AwesomeWords()); // no boom!

By using IteratorAggregate we did not just solve the non-rewindable problem, we also found a good way to share our code. Sometimes it makes sense to use the code of a Generator in multiple classes, and sometimes it makes sense to have dedicated tests for the Generator. In both cases having a dedicated class and file to put it in is very helpful, and a lot nicer than exposing the generator via some public static function.

For cases where it does not make sense to share a Generator and you want to keep it entirely private, you might need to deal with the non-rewindable problem. For those cases you can use my Rewindable Generator library, which allows making your generators rewindable by wrapping their creation function:

$aGenerator = function() { /* ... yield ... */ };
$aService->doStuff(new RewindableGenerator($aGenerator));

A real-world example

A few months ago I refactored some code part of the Wikimedia Deutschland fundraising codebase. This code gets the filesystem paths of email templates by looking in a set of specified directories.

private function getMailTemplatesOnDisk( array $mailTemplatePaths ): array {
    $mailTemplatesOnDisk = [];

    foreach ( $mailTemplatePaths as $path ) {
        $mailFilesInFolder = glob( $path . '/Mail_*' );
        array_walk( $mailFilesInFolder, function( & $filename ) {
            $filename = basename( $filename ); // this would cause problems w/ mail templates in sub-folders
        } );
        $mailTemplatesOnDisk = array_merge( $mailTemplatesOnDisk, $mailFilesInFolder );
    }

    return $mailTemplatesOnDisk;
}

This code made the class bound to the filesystem, which made it hard to test. In fact, this code was not tested. Furthermore, this code irked me, since I like code to be on the functional side. The array_walk mutates its by-reference variable and the assignment at the end of the loop mutates the return variable.

This was refactored using the awesome IteratorAggregate + Generator combo:

class MailTemplateFilenameTraversable implements \IteratorAggregate {
	public function __construct( array $mailTemplatePaths ) {
		$this->mailTemplatePaths = $mailTemplatePaths;
	}

	public function getIterator() {
		foreach ( $this->mailTemplatePaths as $path ) {
			foreach ( glob( $path . '/Mail_*' ) as $fileName ) {
				yield basename( $fileName );
			}
		}
	}
}

Much easier to read/understand code, no state mutation whatsoever, good separation of concerns, easier testing and reusability of this collection building code elsewhere.

See also: Use cases for PHP generators (off-site post).

Tips and Tricks

Generators can yield key value pairs:

yield "Iterators" => "are useful";
yield "Generators" => "are awesome";
// [ "Iterators" => "are useful", "Generators" => "are awesome" ]

You can use yield in PHPUnit data providers.

You can yield from an iterable.

yield from [1, 2, 3];
yield from new ArrayIterator([4, 5]);
// 1, 2, 3, 4, 5

// Flattens iterable[] into Generator
foreach ($collections as $collection) {
    yield from $collection;
}

Thanks for Leszek Manicki and Jan Dittrich for reviewing this blog post.

October 15, 2017

The post Get shell in running Docker container appeared first on ma.ttias.be.

This has saved me more times than I can count, having the ability to debug a running container the way you would in a "normal" VM.

First, see which containers are running;

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  [...] NAMES
925cc10d55df        66cc85c3f275        "gitlab-runner-ser..."   [...] runner-f500bed1-project-3888560-concurrent-0-mysql-0-wait-for-service
0ab431ea0bcf        3e3878acd190        "docker-entrypoint..."   [...] runner-f500bed1-project-3888560-concurrent-0-mysql-0
4d9de6c0fba1        nginx:alpine        "nginx -g 'daemon ..."   [...] nginx-container

To get a shell (Bash) on a container of choice, run this;

$ docker exec -i -t nginx-container /bin/bash

The nginx-container determines which container you want to enter, it's the name in the last column of the docker ps output.

Alternatively, use the container ID;

$ docker exec -i -t 4d9de6c0fba1 /bin/bash

Don't use docker attach, as that'll give you funky results if the initial command that's started in a Docker container is something like MongoDB or Redis, the instance will be killed.

The post Get shell in running Docker container appeared first on ma.ttias.be.

October 13, 2017

I can’t remember why I started to write conference wrap-ups but it started in 2009 when I attended my first Hack.lu! I had a quick look at my blog archives and, until today, I wrote 184 wrap-ups!  The initial idea was probably to bring some material to colleagues who did not have the chance to attend the conference in Luxembourg. Quickly I got some very positive feedbacks from my existing blog readers and it started to attract more and more people. 

Wrap-Up Feedback

Why am I still writing such kind of articles today? For multiple reasons… The first one is personal: It helps me to learn new stuff. The exercise to keep the focus on a speaker and to take notes on the fly is complex. You need to listen, understand and summarize in real time. Usually, I’m writing very synthetic notes and I force myself to beautify the text the same day (otherwise, I’m quickly losing details). Often, my wrap-ups are published during the night.

The second one is for the community… If I’ve some content, why not share it? Honestly, based on the number of infosec events I’m attending,  I consider myself as a lucky guy. With my wrap-ups, I can share a (very) small piece of information that I collected. They are published “as is” without any restriction and review (read: errors can always be present!). I know that some people reuse them even if they attended the same conference. They need to report some content internally in their organization 😉 They are free but be fair to keep a link to the original article.

It won’t change and, next week, I’ll be in Luxembourg for hack.lu. Immediately after, I’ll fly to Budapest for Hacktivity. Hack.lu is one of my preferred events not only for the quality of the schedule but also for the relaxed atmosphere. I meet some friends once a year at hack.lu! My first participation was in 2008 and this edition promises to be awesome with a bunch of interesting talks. Here is my pre-selection:

  • Randori, a low interaction honeypot with a vengeance (Bouke van Laethem)
  • Device sensors meet the web – a story of sadness and regret (Lukasz Olejnik)
  • The Bicho: An Advanced Car Backdoor Maker (Sheila Ayelen Berta, Claudio Caracciolo)
  • Keynterceptor: Press any key to continue (Niels van Dijkhuizen)
  • Sigma – Generic Signatures for Log Events (Thomas Patzke)
  • Front door Nightmares. When smart is not secure (ObiWan666)

Then, let’s go to Hacktivity. Contrariwise, it will be my first experience with this event. The conference has a very good reputation. A lot of nice topics and here is my pre-selection:

  • REST API, pentester’s perspective (Mateusz Olejarka)
  • Exploiting USB/IP in Linux (Ignat Korchagin)
  • Hacking drones and buying passwords in the Darknet (Tobias Schrödel)
  • A heaven for Hackers: Breaking Log/SIEM Products (Mehmet Ince)
  • BlueBorne Explained: Exploiting Android devices over the air (Ben Seri Gregory Vishnepolsky)

You can expect a massive amount of Tweets and daily wrap-ups during the week! Stay tuned and thanks again for reading my delusions…

[Note: You can follow the upcoming conferences that I will attend on the right side of this page in the “Upcoming Events” section]

 

 

 

[The post Wrap-Ups Storm Ahead! has been first published on /dev/random]

October 12, 2017

Children aren’t worried about the future. Young people aren’t worried about the future; they’re worried about us: us leading them into the future we envision

Jack Ma — Oct 2017, keynote speech at Alibaba Cloud’s Computing Conference in Hangzhou

I published the following diary on isc.sans.org: “Version control tools aren’t only for Developers“.

When you start to work on a big project or within a team of developers, it is very useful to use a version control system. The most known are probably ’svn’ or ‘git’. For developers, such tools are a great help to perform tasks like… [Read more]

[The post [SANS ISC] Version control tools aren’t only for Developers has been first published on /dev/random]

October 11, 2017

Four months ago, I shared that Acquia was on the verge of a shift equivalent to the decision to launch Acquia Fields and Drupal Gardens in 2008. As we entered Acquia's second decade, we outlined a goal to move from content management to data-driven customer journeys. Today, Acquia announced two new products that support this mission: Acquia Journey and Acquia Digital Asset Manager (DAM).

Last year on my blog, I shared a video that demonstrated what is possible with cross-channel user experiences and Drupal. We showed a sample supermarket chain called Gourmet Market. Gourmet Market wants its customers to not only shop online using its website, but to also use Amazon Echo or push notifications to do business with them. The Gourmet Market prototype showed an omnichannel customer experience that is both online and offline, in store and at home, and across multiple digital touchpoints. The Gourmet Market demo video was real, but required manual development and lacked easy customization. Today, the launch of Acquia Journey and Acquia DAM makes building these kind of customer experiences a lot easier. It marks an important milestone in Acquia's history, as it will accelerate our transition from content management to data-driven customer journeys.

A continuous journey across multiple digital touch points and devices

Introducing Acquia Journey

I've written a great deal about the Big Reverse of the Web, which describes the transition from "pull-based" delivery of the web, meaning we visit websites, to a "push-based" delivery, meaning the web comes to us. The Big Reverse forces a major re-architecture of the web to bring the right information, to the right person, at the right time, in the right context.

The Big Reverse also ushers in the shift from B2C to B2One, where organizations develop a one-to-one relationship with their customers, and contextual and personalized interactions are the norm. In the future, every organization will have to rethink how it interacts with customers.

Successfully delivering a B2One experience requires an understanding of your user's journey and matching the right information or service to the user's context. This alone is no easy feat, and many marketers and other digital experience builders often get frustrated with the challenge of rebuilding customer experiences. For example, although organizations can create brilliant campaigns and high-value content, it's difficult to effectively disseminate marketing efforts across multiple channels. When channels, data and marketing software act in different silos, it's nearly impossible to build a seamless customer experience. The inability to connect customer profiles and journey maps with various marketing tools can result in unsatisfied customers, failed conversion rates, and unrealized growth.

An animation showing Acquia's journey building solution

Acquia Journey delivers on this challenge by enabling marketers to build data-driven customer journeys. It allows marketers to easily map, assemble, orchestrate and manage customer experiences like the one we showed in our Gourmet Market prototype.

It's somewhat difficult to explain Acquia Journey in words — probably similar to trying to explain what a content management system does to someone who has never used one before. Acquia Journey provides a single interface to define and evaluate customer journeys across multiple interaction points. It combines a flowchart-style journey mapping tool with unified customer profiles and an automated decision engine. Rules-based triggers and logic select and deliver the best-next action for engaging customers.

One of the strengths of Acquia Journey is that it integrates many different technologies, from marketing and advertising technologies to CRM tools and commerce platforms. This makes it possible to quickly assemble powerful and complex customer journeys.

Implementing getBestNextExperience() creates both customer and business value

Acquia Journey will simplify how organizations deliver the "best next experience" for the customer. Providing users with the experience they not only want, but expect will increase conversion rates, grow brand awareness, and accelerate revenue. The ability for organizations to build more relevant user experiences not only aligns with our customers' needs but will enable them to make the biggest impact possible for their customers.

Acquia's evolving product offering also puts control of user data and experience back in the hands of the organization, instead of walled gardens. This is a step toward uniting the Open Web.

Introducing Acquia Digital Asset Manager (DAM)

Digital asset management systems have been around for a long time, and were originally hosted through on-premise servers. Today, most organizations have abandoned on-premise or do-it-yourself DAM solutions. After listening to our customers, it became clear that large organizations are seeking a digital asset management solution that centralizes control of creative assets for the entire company.

Many organizations lack a single-source of truth when it comes to managing digital assets. This challenge has been amplified as the number of assets has rapidly increased in a world with more devices, more channels, more campaigns, and more personalized and contextualized experiences. Acquia DAM provides a centralized repository for managing all rich media assets, including photos, videos, PDFs, and other corporate documents. Creative and marketing teams can upload and manage files in Acquia DAM, which can then be shared across the organization. Graphic designers, marketers and web managers all have a hand in translating creative concepts into experiences for their customers. With Acquia DAM, every team can rely on one dedicated application to gather requirements, share drafts, consolidate feedback and collect approvals for high-value marketing assets.

On top of Drupal's asset and media management capabilities, Acquia DAM provides various specialized functionality, such as automatic transcoding of assets upon download, image and video mark-up during approval workflows, and automated tagging for images using machine learning and image recognition.

A screenshot of Acquia's Digital Asset Management solutionBy using a drag-and-drop interface on Acquia DAM, employees can easily publish approved assets in addition to searching the repository for what they need.

Acquia DAM seamlessly integrates with both Drupal 7 and Drupal 8 (using Drupal's "media entities"). In addition to Drupal, Acquia DAM is built to integrate with the entirety of the Acquia Platform. This includes Acquia Lift and Acquia Journey, which means that any asset managed in the Acquia DAM repository can be utilized to create personalized experiences across multiple Drupal sites. Additionally, through a REST API, Acquia DAM can also be integrated with other marketing technologies. For example, Acquia DAM supports designers with a plug in to Adobe Creative Cloud, which integrates with Photoshop, InDesign and Illustrator.

Acquia's roadmap to data-driven customer journeys

Some of the most important market trends in digital for 2017

Throughout Acquia's first decade, we've been primarily focused on providing our customers with the tools and services necessary to scale and succeed with content management. We've been very successful with helping our customers scale and manage Drupal and cloud solutions. Drupal will remain a critical component to our customer's success, and we will continue to honor our history as committed supporters of open source, in addition to investing in Drupal's future.

However, many of our customers need more than content management to be digital winners. The ability to orchestrate customer experiences using content, user data, decisioning systems, analytics and more will be essential to an organization's success in the future. Acquia Journey and Acquia DAM will remove the complexity from how organizations build modern digital experiences and customer journeys. We believe that expanding our platform will be good not only for Acquia, but for our partners, the Drupal community, and our customers.

Acquia's product strategy for 2017 and beyond

October 10, 2017

Suppose that you have a RDO/Openstack cloud already in place, but that you'd want to automate some operations : what can you do ? On my side, I already mentioned that I used puppet to deploy initial clouds, but I still prefer Ansible myself when having to launch ad-hoc tasks, or even change configuration[s]. It's particulary true for our CI environment where we run "agentless" so all configuration changes happen through Ansible.

The good news is that Ansible has already some modules for Openstack but it has some requirements and a little bit of understanding before being able to use those.

First of all, all the ansible os_ modules need "shade" on the host included in the play, and that will be responsible of all os_ modules launch. At the time of writing this post, it's not yet available on mirror.centos.org, (a review is open so that will be soon available directly) but you can find the pkg on our CBS builders

Once installed, a simple os_image task was directly failing, despite the fact that auth: was present, and that's due to a simple reason : Ansible os_ modules still want to use v2 API, while it's now defaulting to v3 in Pike release. There is no way to force ansible itself to use v3, but as it uses shade behind the scene, there is a way to force this through os-client-config

That means that you just have to use a .yaml file (does that sound familiar for ansible ?) that will contain everything you need to know about specific cloud, and then just in ansible declare which cloud you're configuring.

That clouds.yaml file can be under $current_directory, ~/.config/openstack or /etc/openstack so it's up to you to decide where you want to temporary host it, but I selected /etc/openstack/ :

- name: Ensuring we have required pkgs for ansible/openstack
  yum:
    name: python2-shade
    state: installed

- name: Ensuring local directory to hold the os-client-config file
  file:
    path: /etc/openstack
    state: directory
    owner: root
    group: root

- name: Adding clouds.yaml for os-client-config for further actions
  template:
    src: clouds.yaml.j2
    dest: /etc/openstack/clouds.yaml
    owner: root
    group: root
    mode: 0700

Of course such clouds.yaml file is itself a jinja2 template distributed by ansible on the host in the play before using the os_* modules :

clouds:
  {{ cloud_name }}:
    auth:
      username: admin
      project_name: admin
      password: {{ openstack_admin_pass }}
      auth_url: http://{{ openstack_controller }}:5000/v3/
      user_domain_name: default
      project_domain_name: default
    identity_api_version: 3

You just have to adapt to your needs (see doc for this) but the interesting part is the identity_api_version to force v3.

Then, you can use all that in a simple way through ansible tasks, in this case adding users to a project :

- name: Configuring OpenStack user[s]
  os_user:
    cloud: "{{ cloud_name }}"
    default_project: "{{ item.0.name }}"
    domain: "{{ item.0.domain_id }}"
    name: "{{ item.1.login }}"
    email: "{{ item.1.email }}"
    password: "{{ item.1.password }}"           
  with_subelements:
    - "{{ cloud_projects }}"
    - users  
  no_log: True

From a variables point of view, I decided to just have a simple structure to host project/users/roles/quotas like this :

cloud_projects:
  - name: demo
    description: demo project
    domain_id: default
    quota_cores: 20
    quota_instances: 10
    quota_ram: 40960
    users:
      - login: demo_user
        email: demo@centos.org
        password: Ch@ngeM3
        role: admin # can be _member_ or admin
      - login: demo_user2
        email: demo2@centos.org
        password: Ch@ngeMe2

Now that it works, you can explore all the other os_* modules and I'm already using those to :

  • Import cloud images in glance
  • Create networks and subnets in neutron
  • Create projects/users/roles in keystone
  • Change quotas for those projects

I'm just discovering how powerful those tools are, so I'll probably discover much more interesting things to do with those later.

October 09, 2017

BruCON 0x09 is over! It’s time to have a look at the data captured during the last Thursday and Friday. As the previous years, the setup was almost the same: An Internet pipe with a bunch of access-points, everything interconnected through a pfSense firewall. The guest network (dedicated to attendees) traffic is captured and processed by a SecurityOnion server + basic full packet capture. We also used our classic wall-of-sheep to track the web browsing activity of our beloved visitors.

Let’s start with a few raw numbers. With the end of the 3G/4G roaming costs in Europe since June, most European visitors avoid the usage of wireless networks and prefer to remain connected via their mobile phone. In a few numbers:

  • 206 Gigabytes of PCAP files
  • 50.450 pictures collected by the wall-of-sheep
  • 19 credentials captured
  • 500+ unique devices connected to the WiFi network
  • 150 PE files downloaded (Windows executables)
  • 3 blocked users
  • 1 rogue DHCP server

We saw almost the same amount of traffic than the previous years (even if we had more people attending the conference!). What about our visitors?
Unique Wi-Fi Clients by OS over Time

Strange that we had some many “unknown” device. Probably due to an outdated MAC address prefixes databases.

Top 10 Applications by Usage - Summary

Good to see that SSL is the top protocol detected! UDP reached the third-position due to the massive use of VPN connections. Which is also good!

Our visitors communicated with 118K+ IP addresses from all over the word:

Worldwide Connections

Here is the top-20 of DNS requests logged:

Rank

FQDN Hits

1

api.dataplicity.com

59310

2

www.google.com

20097

3

softwareupdate.vmware.com

9050

4

auth.gfx.ms

6766

5

swscan.apple.com

6706

6

v10.vortex-win.data.microsoft.com

5300

7

www.googleapis.com

5252

8

www.icanhazip.com

4402

9

www.google.be

3831

10

clients4.google.com

3721

11

play.google.com

3562

12

win10.ipv6.microsoft.com

3459

13

outlook.office365.com

3267

14

ssl.gstatic.com

3130

15

settings-win.data.microsoft.com

3111

16

pingsl.avast.com

2884

17

safebrowsing-cache.google.com

2841

18

avast.com.edgesuite.net

2533

19

graph.facebook.com

2164

20

0x13.nl

1990

As most of the traffic captured was web-based, I had a look at the different tools/applications used to access web resources. Here is the top-20:

Rank

FQDN

1

Firefox

2

Chrome

3

Microsoft-CryptoAPI

4

Microsoft

5

Safari

6

Dalvik

7

trustd

8

MSIE

9

cloudd

10

Debian

11

Windows-Update-Agent

12

iPhone

13

Unspecified

14

Microsoft-WNS

15

CaptiveNetworkSupport

16

serer-bag

17

MICROSOFT_DEVICE_METADATA_RETRIEVAL_CLIENT (1)

18

Spotify

19

Unknown

20

Microsoft-Delivery-Optimization

(1) https://docs.microsoft.com/en-us/windows-hardware/drivers/install/device-metadata-retrieval-client

I uploaded the 200+ gigabytes of PCAP data into my Moloch instance and searched for interesting traffic. What has been found:

  • One visitor polled his network devices (172.16.x.x) during the two days (5995 SNMP connections detected)
  • Two visitors performed RDP sessions to two external IP addresses
  • Two visitors generated SIP (VoIP) traffic with two remote servers
  • 29 remote IMAP servers were identified (strange, no POP3! 🙂
  • SSH connections were established with 36 remote servers (no telnet!)

Finally, our wall-of-sheep captured web traffic during the whole conference:

Wall of Sheep

Of course, we had some “p0rn denial of service attacks” but it’s part of the game right? See you for the 0x0A (10th edition) next year with, crossing fingers, more fun on the network!

 

[The post BruCON Network 0x09 Wrap-Up has been first published on /dev/random]

I published the following diary on isc.sans.org: “Base64 All The Things!“.

Here is an interesting maldoc sample captured with my spam trap. The attached file is “PO# 36-14673.DOC” and has a score of 6 on VT. The file contains Open XML data that refers to an invoice.. [Read more]

[The post [SANS ISC] Base64 All The Things! has been first published on /dev/random]

Initially I started creating a general post about PHP Generators, a feature introduced in PHP 5.5. However since I keep failing to come up with good examples for some cool ways to use Generators, I decided to do this mini post focusing on one such cool usage.

PHPUnit data providers

A commonly used PHPUnit feature is data providers. In a data provider you specify a list of argument lists, and the test methods that use the data provider get called once for each argument list.

Often data providers are created with an array variable in which the argument lists get stuffed. Example (including poor naming):

/**
 * @dataProvider provideUserInfo
 */
function testSomeStuff( string $userName, int $userAge ) {}

function provideUserInfo() {
    $return = [];

    $return[] = [ 'Such Name', 42 ];
    $return[] = [ 'Very Name', 23 ];
    $return['Named parameter set'] = [ 'So Name', 1337 ];

    return $return;
}

The not so nice thing here is that you have a variable (explicit state) and you modify it (mutable state). A more functional approach is to just return an array that holds the argument lists directly. However if your argument list creation is more complex than in this example, requiring state, this might not work. And when such state is required, you end up with more complexity and a higher chance that the $return variable will bite you.

Using yield

What you might not have realized is that data providers do not need to return an array. They need to return an iterable, so they can also return an Iterator, and by extension, a Generator. This means you can write the above data provider as follows:

function provideUserInfo() {
    yield [ 'Such Name', 42 ];
    yield [ 'Very Name', 23 ];
    yield 'Named parameter set' => [ 'So Name', 1337 ];
}

No explicit state to be seen!

Stay tuned for more generator goodness if I can overcome my own laziness (hint hint :))

October 06, 2017

Hermes

Since its founding in 1837, Hermès has defined luxury. Renowned as an iconic brand within the fashion industry, Hermès is now setting the trend for how customers shop online. This week, Hermès launched its new site in Drupal!

Hermès married the abilities of Drupal as a CMS and Magento as an eCommerce engine to provide their customers with highly engaging shopping experiences. Hermès' new site is a great example of how iconic brands can use Drupal to power ambitious digital experiences. If you are in the mood for some retail therapy, check out https://www.hermes.com!

The post Antwerp WordPress User Group offering public speaking course appeared first on ma.ttias.be.

At the next WordPress Antwerp meetup, there will be a presentation and workshop on how to do public speaking, how to complete a CFP (Call For Presentation) and increase your chances to be accepted as a conference speaker.

I believe one of the biggest enhancers of my career has been public speaking & being a good communicator. In order to give others the same abilities I want to raise awareness for the efforts the WordPress Antwerp team are putting into this. It's rare I see a user group organise something like this and I'd love to see more of it.

To get started, sign up at the Meetup page of WordPress Antwerp & get started with public speaking!

(Sorry English readers, it's a local meetup that'll be Dutch/Flemish only)

The post Antwerp WordPress User Group offering public speaking course appeared first on ma.ttias.be.

October 05, 2017

At work, I help maintain a smartcard middleware that is provided to Belgian citizens who want to use their electronic ID card to, e.g., log on to government websites. This middleware is a piece of software that hooks into various browsers and adds a way to access the smartcard in question, through whatever APIs the operating system and the browser in question provide for that purpose. The details of how that is done differ between each browser (and in the case of Google Chrome, for the same browser between different operating systems); but for Firefox (and Google Chrome on free operating systems), this is done by way of a PKCS#11 module.

For Firefox 57, mozilla decided to overhaul much of their browser. The changes are large and massive, and in some ways revolutionary. It's no surprise, therefore, that some of the changes break compatibility with older things.

One of the areas in which breaking changes were made is in the area of extensions to the browser. Previously, Firefox had various APIs available for extensions; right now, all APIs apart from the WebExtensions API are considered "legacy" and support for them will be removed from Firefox 57 going forward.

Since installing a PKCS#11 module manually is a bit complicated, and since the legacy APIs provided a way to do so automatically provided the user would first install an add-on (or provided the installer of the PKCS#11 module sideloads it), most parties who provide a PKCS#11 module for use with Firefox will provide an add-on to automatically install it. Since the alternative involves entering the right values in a dialog box that's hidden away somewhere deep in the preferences screen, the add-on option is much more user friendly.

I'm sure you can imagine my dismay when I found out that there was no WebExtensions API to provide the same functionality. So, after asking around a bit, I filed bug 1357391 to get a discussion started. While it took some convincing initially to get people to understand the reasons for wanting such an API, eventually the bug was assigned the "P5" priority -- essentially, a "we understand the need and won't block it, but we don't have the time to implement it. Patches welcome, though" statement.

Since having an add-on was something that work really wanted, and since I had the time, I got the go-ahead from management to look into implementing the required code myself. I made it obvious rather quickly that my background in Firefox was fairly limited, though, and so was assigned a mentor to help me through the process.

Having been a Debian Developer for the past fifteen years, I do understand how to develop free software. Yet, the experience was different enough that still learned some new things about free software development, which was somewhat unexpected.

Unfortunately, the process took much longer than I had hoped, which meant that the patch was not ready by the time Firefox 57 was branched off mozilla's "central" repository. The result of that is that while my patch has been merged into what will eventually become Firefox 58, it looks strongly as though it won't make it into Firefox 57. That's going to cause some severe headaches, which I'm not looking forward to; and while I can certainly understand the reasons for not wanting to grant the exception for the merge into 57, I can't help but feeling like this is a missed opportunity.

Anyway, writing code for the massive Open Source project that mozilla is has been a load of fun, and in the process I've learned a lot -- not only about Open Source development in general, but also about this weird little thing that Javascript is. That might actually be useful for this other project that I've got running here.

In closing, I'd like to thank Tomislav 'zombie' Jovanovic for mentoring me during the whole process, without whom it would have been doubtful if I would even have been ready by now. Apologies for any procedural mistakes I've made, and good luck in your future endeavours! :-)

October 04, 2017

The post Laravel Forge + Envoyer + Managed Hosting = Nucleus appeared first on ma.ttias.be.

I've been enjoying using Laravel a lot lately, a modern PHP framework that comes with queues, a CLI component, decent standards and an incredibly large package ecosystem, not the least by the Spatie guys who publish a ton of their work online.

What has always fascinated me by the Laravel ecosystem is that the creator, Taylor Otwell, saw the bigger picture of application development. It's not just about writing code, it's about running infrastructure to support it (Laravel Forge), deploying code reliably (Laravel Envoyer), writing maintainable tests (Laravel Dusk), ... Everything is neatly packaged and available.

Forge: a managed hosting alternative

With Forge, everyone can create a Laravel-optimized server on providers like Digital Ocean, Linode or Ocean in mere minutes. A VM gets deployed, a config is written and you can SSH and git clone to get started.

While I see the appeal, the sysadmin in me wonders;

  • Who monitors those servers? If MySQL crashes at 11PM, who fixes it?
  • Who takes care of the updates? The security ones get auto-applied (a sane default), but who takes care of package updates?
  • Who handles the security of the machines? Do you know what's running? Do you know its configs? What are you exposing? What versions are you running?
  • Who takes care of the back-ups of the databases and files? How regularly are they stored? Are they copied offsite?
  • How quickly can you get up and running again if a server crashes? Or if it accidentally gets deleted?

So in general: who manages that Forge server?

I fear, for many sites deployed via Forge, there isn't anyone actively managing that server. That's a shame, because even though Forge's default are OK'ish, one day you'll wish your site/server was actually managed by a team that understands hosting and servers. And that's where we come in.

At Nucleus, we specialize in managed hosting & custom made server setups. And we've got a pre-built template specific for running Laravel applications. We manage all the configs, we take care of back-ups, the 24/7 support and interventions, the monitoring & graphing of your CPU/memory/disk capacity, monthly reporting of uptime, etc.

If you're looking for Managed Laravel Hosting, come have a chat. Our Laravel-optimized servers come pre-configured with all you need like PHP 7.1, Redis, the schedule:run cron, supervisor workers, a pre-generated .env config, a deploy script, pre-installed composer/yarn, SSH access, ... well, all you need to reliably run Laravel.

Deploy with the ease of Envoyer, tailored to our servers

I'll admit, our server setup is slightly different than Forge's. I think it's better in a couple of critical ways, though, which is why we've tailored the deploy mechanisme to our setup.

For starters, we run CentOS over the latest Ubuntu, for stability. But we combine it with modern packages, so you get PHP 7.1, MariaDB 10.2, Redis 4 & all other up-to-date packages you'd expect.

We also run multiple PHP-FPM master pools for better OPCache efficiency,  multiple Redis instances, tight firewalling, an opinionated (but proven) directory layout, ... all things that slightly influence your deployment. To make that easier, we publish a simple Laravel package to help take care of your deploys.

To install, run these 2 commands in your Laravel application;

$ composer require nucleus/laravel-deploy
$ php artisan vendor:publish --provider=Nucleus\\Deploy\\DeployServiceProvider

After that, deploying to your Nucleus server(s) is as simple as:

$ php artisan deploy

That's it.

The deploy reads a few parameters from your .env configuration (like host, username, your git repository location etc.) and handles the rest.

It uses the nucleus/laravel-deploy package, whose source is up on Github. Feedback is more than welcome! It's only a 1.0 version now, we plan to extend the functionality with HipChat/Slack hooks, better notifications, multi-server support & whatever fancy things we can come up with.

Don't like the way it deploys? Change it. It's a Laravel Blade template, easy to read, easy to extend. It's based on Spatie's deploy script, tuned to our stack.

Ease of use + stability + quality = Nucleus

OK, sounds like some marketing BS, I agree. ;-)

If you're developing a Laravel application and you're looking for reliable, quality hosting by a team of experts who -- quite literally -- speak your language, poke me on Twitter as @mattiasgeniar or have a look at the nucleus.be website. I'd love to have a chat with you to see how we can help support your business and how we can improve our Laravel-focussed hosting offering.

The post Laravel Forge + Envoyer + Managed Hosting = Nucleus appeared first on ma.ttias.be.

October 03, 2017

We are pleased to announce the developer rooms that will be organised at FOSDEM 2018. Developer rooms are assigned to self-organising groups to work together on open source projects, to discuss topics relevant to a broader subset of the community, etc. The individual developer room organisers will issue their calls for participation in the next few days. We will update this table with links to the calls for participation. Both days, 3 February & 4 February 2018 Topic Call for Participation CfP deadline Embedded, Mobile and Automotive - TBA Virtualization and IaaS announcement 2017-12-01 Saturday 3 February 2018 Topic舰
Mijn vriendin is EU-burger. Geen enkel niveau van overheid in België kent haar burgerlijke staat, dus die is "onbepaald".

De stad Leuven stuurt haar een brief. Ze heeft niet graag mensen met burgerlijke staat "onbepaald" in haar registers staan. Dat geeft miserie, onder meer als je wettelijk wil gaan samenwonen. Hoe moeilijk kan het zijn, anno 2017, om van "onbepaald" "ongehuwd" te maken?

Dag 1. Mijn vriendin luistert braaf en schiet in actie. Ze vraagt bij de Letse overheid haar gegevens uit het bevolkingsregister op. Elektronisch. 11 pagina's. Op pagina 1 in enkele lijntjes alle nodige gegevens om "ongehuwd" te worden voor de Belgische overheid. In het Lets weliswaar.

Burgerlijke stand Leuven is enkel op afspraak bereikbaar. Voor de meeste zaken moet je meerdere keren langsgaan omdat er iets niet helemaal volgens het boekje is. We emailen dus, om zeker te zijn dat dit document goed genoeg is.

Dag 6. De (vriendelijke!) ambtenaar burgerlijke stand antwoordt op onze email. Hij weigert in eerste instantie dit document. "Het document dat u doorstuurt lijkt niet op de documenten in onze bronnen.
Onze bronnen stellen dat de documenten afgegeven dienen te worden door het ministerie van Binnenlandse zaken van Letland. Daarnaast zal, om de aanpassing mogelijk te maken, het document vertaald dienen te worden naar het Nederlands door een beëdigd vertaler. Bijgevoegd vindt u een lijst met vertalers."
Dag 7. Het elektronisch uittreksel bevolkingsregister is weldegelijk afgegeven door het ministerie van Binnenlandse zaken van Letland. Hun website vermeldt dat ook duidelijk. Ik stuur een email terug dus, met citaat en vertaling naar het Nederlands van de relevante delen van de website van het ministerie van Binnenlandse zaken van Letland.
Plus de contactgegevens van een beëdigd vertaler Lets-Nederlands, want de lijst van beëdigd vertalers die hij had opgestuurd bevatte er geen.
Ik bel ook even, en krijg de belofte dat een en ander "vandaag nog" aan het diensthoofd wordt voorgelegd.

Dag 13. Ik stuur de dienst burgerlijke stand van de stad een herinnering, want nog geen antwoord op mijn bericht. Dezelfde dag nog antwoord van het diensthoofd burgerlijke stand. "Wij kunnen dit document aanvaarden (zonder legalisatie, Letland en België zijn beide partij bij het Verdrag van Brussel van 1987), maar met beëdigde vertaling naar het Nederlands."

Ik vraag ook  een offerte op voor beëdigde vertaling van het hele document Herinner u: alle nuttige info staat op de eerste paar lijntjes, maar op mijn herhaalde vraag of het voldoende was dat te vertalen was er nog geen antwoord. Soms is gemoedsrust ook iets waard...

Dag 14. Antwoord op onze offerteaanvraag voor beëdigde vertaling: 80€ ex. BTW, inclusief legalisatie bij de rechtbank. Ah, maar ik herinner me het antwoord van het diensthoofd burgerlijke stand: "Wij kunnen dit document aanvaarden (zonder legalisatie), maar met beëdigde vertaling naar het Nederlands." Ik vraag en krijg dus een kleine korting omdat legalisatie niet nodig is...

Dag 17. De elektronische vertaling is klaar. Nu nog wachten op een scan met de handtekening van de beëdigd vertaler erop.


Dag 19. Het weekend is voorbij. De dienst burgerlijke stand werkt weer. Ik bel maar even om te vragen of ik een scan mag doorsturen, of dat we een afspraak moeten maken (1 week wachttijd) en persoonlijk overhandigen. We moeten het zowiezo persoonlijk komen afgeven. En het origineel uittreksel bevolkingsuittreksel mag dan wel vrij van legalisatie mag zijn, de beëdigde vertaling is dat niet. Die moet gelegaliseerd worden. Door de rechtbank van eerste aanleg waar de beëdigd vertaler de eed heeft afgelegd. Niet Leuven dus, maar Antwerpen.

We emailen de beëdigd vertaler met de vraag toch legalisatie te voorzien.

De wachttijd op het stadhuis voor dit soort afspraak is miniaal een week. Ik maak dus alvast 2 afspraken voor begin volgende week. Als er een niet nodig blijkt te zijn kan ik ze nog steeds annuleren. Het alternatief is mogelijk een week extra vertraging.

Dag 20. Vlot antwoord van de beëdigd vertaler. Voor 25€ ex. BTW extra (15€ duurder dan de oorspronkelijke quote inclusief legalisatie) kan ze de legalisatie voorzien. Een billijk voorstel waar we graag op ingaan.

Wordt vervolgd...

Een speldekopje hoop voor de toekomst is een EU-verordening uit juli 2016 http://eur-lex.europa.eu/legal-content/NL/TXT/HTML/?uri=CELEX:32016R1191 . Er komt vlottere gegevensuitwisseling tussen de EU-lidstaten. Misschien. Binnen enkele jaren. Op papier. In een heel beperkt aantal gevallen.

Liefste Europa, liefste EU-lidstaten,

Een beetje meer ambitie graag! Er is geen enkel, maar dan ook geen enkel excuus om dit zo ongelofelijk traag en omslachtig te laten verlopen. Los dit op. Niet in 2021, 2024 of later, maar nog dit jaar!

October 02, 2017

Drupal react

Last week at DrupalCon Vienna, I proposed adding a modern JavaScript framework to Drupal core. After the keynote, I met with core committers, framework managers, JavaScript subsystem maintainers, and JavaScript experts in the Drupal community to discuss next steps. In this blog post, I look back on how things have evolved, since the last time we explored adding a new JavaScript framework to Drupal core two years ago, and what we believe are the next steps after DrupalCon Vienna.

As a group, we agreed that we had learned a lot from watching the JavaScript community grow and change since our initial exploration. We agreed that today, React would be the most promising option given its expansive adoption by developers, its unopinionated and component-based nature, and its well-suitedness to building new Drupal interfaces in an incremental way. Today, I'm formally proposing that the Drupal community adopt React, after discussion and experimentation has taken place.

Two years ago, it was premature to pick a JavaScript framework

Three years ago, I developed several convictions related to "headless Drupal" or "decoupled Drupal". I believed that:

  1. More and more organizations wanted a headless Drupal so they can use a modern JavaScript framework to build application-like experiences.
  2. Drupal's authoring and site building experience could be improved by using a more modern JavaScript framework.
  3. JavaScript and Node.js were going to take the world by storm and that we would be smart to increase the amount of JavaScript expertise in our community.

(For the purposes of this blog post, I use the term "framework" to include both full MV* frameworks such as Angular, and also view-only libraries such as React combined piecemeal with additional libraries for managing routing, states, etc.)

By September 2015, I had built up enough conviction to write several long blog posts about these views (post 1, post 2, post 3). I felt we could accomplish all three things by adding a JavaScript framework to Drupal core. After careful analysis, I recommended that we consider React, Ember and Angular. My first choice was Ember, because I had concerns about a patent clause in Facebook's open-source license (since removed) and because Angular 2 was not yet in a stable release.

At the time, the Drupal community didn't like the idea of picking a JavaScript framework. The overwhelming reactions were these: it's too early to tell which JavaScript framework is going to win, the risk of picking the wrong JavaScript framework is too big, picking a single framework would cause us to lose users that favor other frameworks, etc. In addition, there were a lot of different preferences for a wide variety of JavaScript frameworks. While I'd have preferred to make a bold move, the community's concerns were valid.

Focusing on Drupal's web services instead

By May of 2016, after listening to the community, I changed my approach; instead of adding a specific JavaScript framework to Drupal, I decided we should double down on improving Drupal's web service APIs. Instead of being opinionated about what JavaScript framework to use, we would allow people to use their JavaScript framework of choice.

I did a deep dive on the state of Drupal's web services in early 2016 and helped define various next steps (post 1, post 2, post 3). I asked a few of the OCTO team members to focus on improving Drupal 8's web services APIs; funded improvements to Drupal core's REST API, as well as JSON API, GraphQL and OpenAPI; supported the creation of Waterwheel projects to help bootstrap an ecosystem of JavaScript front-end integrations; and most recently supported the development of Reservoir, a Drupal distribution for headless Drupal. There is also a lot of innovation coming from the community with lots of work on the Contenta distribution, JSON API, GraphQL, and more.

The end result? Drupal's web service APIs have progressed significantly the past year. Ed Faulkner of Ember told us: "I'm impressed by how fast Drupal made lots of progress with its REST API and the JSON API contrib module!". It's a good sign when a core maintainer of one of the leading JavaScript frameworks acknowledges Drupal's progress.

The current state of JavaScript in Drupal

Looking back, I'm glad we decided to focus first on improving Drupal's web services APIs; we discovered that there was a lot of work left to stabilize them. Cleanly integrating a JavaScript framework with Drupal would have been challenging 18 months ago. While there is still more work to be done, Drupal 8's available web service APIs have matured significantly.

Furthermore, by not committing to a specific framework, we are seeing Drupal developers explore a range of JavaScript frameworks and members of multiple JavaScript framework communities consuming Drupal's web services. I've seen Drupal 8 used as a content repository behind Angular, Ember, React, Vue, and other JavaScript frameworks. Very cool!

There is a lot to like about how Drupal's web service APIs matured and how we've seen Drupal integrated with a variety of different frameworks. But there is also no denying that not having a JavaScript framework in core came with certain tradeoffs:

  1. It created a barrier for significantly leveling up the Drupal community's JavaScript skills. In my opinion, we still lack sufficient JavaScript expertise among Drupal core contributors. While we do have JavaScript experts working hard to maintain and improve our existing JavaScript code, I would love to see more experts join that team.
  2. It made it harder to accelerate certain improvements to Drupal's authoring and site building experience.
  3. It made it harder to demonstrate how new best practices and certain JavaScript approaches could be leveraged and extended by core and contributed modules to create new Drupal features.

One trend we are now seeing is that traditional MV* frameworks are giving way to component libraries; most people seem to want a way to compose interfaces and interactions with reusable components (e.g. libraries like React, Vue, Polymer, and Glimmer) rather than use a framework with a heavy focus on MV* workflows (e.g. frameworks like Angular and Ember). This means that my original recommendation of Ember needs to be revisited.

Several years later, we still don't know what JavaScript framework will win, if any, and I'm willing to bet that waiting two more years won't give us any more clarity. JavaScript frameworks will continue to evolve and take new shapes. Picking a single one will always be difficult and to some degree "premature". That said, I see React having the most momentum today.

My recommendations at DrupalCon Vienna

Given that it's been almost two years since I last suggested adding a JavaScript framework to core, I decided to bring the topic back in my DrupalCon Vienna keynote presentation. Prior to my keynote, there had been some renewed excitement and momentum behind the idea. Two years later, here is what I recommended we should do next:

  • Invest more in Drupal's API-first initiative. In 2017, there is no denying that decoupled architectures and headless Drupal will be a big part of our future. We need to keep investing in Drupal's web service APIs. At a minimum, we should expand Drupal's web service APIs and standardize on JSON API. Separately, we need to examine how to give API consumers more access to and control over Drupal's capabilities.
  • Embrace all JavaScript frameworks for building Drupal-powered applications. We should give developers the flexibility to use their JavaScript framework of choice when building front-end applications on top of Drupal — so they can use the right tool for the job. The fact that you can front Drupal with Ember, Angular, Vue, React, and others is a great feature. We should also invest in expanding the Waterwheel ecosystem so we have SDKs and references for all these frameworks.
  • Pick a framework for Drupal's own administrative user interfaces. Drupal should pick a JavaScript framework for its own administrative interface. I'm not suggesting we abandon our stable base of PHP code; I'm just suggesting that we leverage JavaScript for the things that JavaScript is great at by moving relevant parts of our code from PHP to JavaScript. Specifically, Drupal's authoring and site building experience could benefit from user experience improvements. A JavaScript framework could make our content modeling, content listing, and configuration tools faster and more application-like by using instantaneous feedback rather than submitting form after form. Furthermore, using a decoupled administrative interface would allow us to dogfood our own web service APIs.
  • Let's start small by redesigning and rebuilding one or two features. Instead of rewriting the entirety of Drupal's administrative user interfaces, let's pick one or two features, and rewrite their UIs using a preselected JavaScript framework. This allows us to learn more about the pros and cons, allows us to dogfood some of our own APIs, and if we ultimately need to switch to another JavaScript framework or approach, it won't be very painful to rewrite or roll the changes back.

Selecting a JavaScript framework for Drupal's administrative UIs

In my keynote, I proposed a new strategic initiative to test and research how Drupal's administrative UX could be improved by using a JavaScript framework. The feedback was very positive.

As a first step, we have to choose which JavaScript framework will be used as part of the research. Following the keynote, we had several meetings at DrupalCon Vienna to discuss the proposed initiative with core committers, all of the JavaScript subsystem maintainers, as well as developers with real-world experience building decoupled applications using Drupal's APIs.

There was unanimous agreement that:

  1. Adding a JavaScript framework to Drupal core is a good idea.
  2. We want to have sufficient real-use experience to make a final decision prior to 8.6.0's development period (Q1 2018). To start, the Watchdog page would be the least intrusive interface to rebuild and would give us important insights before kicking off work on more complex interfaces.
  3. While a few people named alternative options, React was our preferred option, by far, due to its high degree of adoption, component-based and unopinionated nature, and its potential to make Drupal developers' skills more future-proof.
  4. This adoption should be carried out in a limited and incremental way so that the decision is easily reversible if better approaches come later on.

We created an issue on the Drupal core queue to discuss this more.

Conclusion

Drupal supporting different JavaScript front endsDrupal should support a variety of JavaScript libraries on the user-facing front end while relying on a single shared framework as a standard across Drupal administrative interfaces.

In short, I continue to believe that adopting more JavaScript is important for the future of Drupal. My original recommendation to include a modern JavaScript framework (or JavaScript libraries) for Drupal's administrative user interfaces still stands. I believe we should allow developers to use their JavaScript framework of choice to build front-end applications on top of Drupal and that we can start small with one or two administrative user interfaces.

After meeting with core maintainers, JavaScript subsystem maintainers, and framework managers at DrupalCon Vienna, I believe that React is the right direction to move for Drupal's administrative interfaces, but we encourage everyone in the community to discuss our recommendation. Doing so would allow us to make Drupal easier to use for site builders and content creators in an incremental and reversible way, keep Drupal developers' skills relevant in an increasingly JavaScript-driven world, move us ahead with modern tools for building user interfaces.

Special thanks to Preston So for contributions to this blog post and to Matt Grill, Wim Leers, Jason Enter, Gábor Hojtsy, and Alex Bronstein for their feedback during the writing process.

I published the following diary on isc.sans.org: “Investigating Security Incidents with Passive DNS“.

Sometimes when you need to investigate a security incident or to check for suspicious activity, you become frustrated because the online resource that you’re trying to reach has already been cleaned. We cannot blame system administrators and webmasters who are just doing their job. If some servers or websites remains compromised for weeks, others are very quickly restored/patched/cleaned to get rid of the malicious content. It’s the same for domain names. Domains registered only for malicious purposes can be suspended to prevent further attacks. If the domain is not suspended, the offending DNS record is simply removed… [Read more]

[The post [SANS ISC] Investigating Security Incidents with Passive DNS has been first published on /dev/random]

September 28, 2017

Recently we got our hands on some aarch64 (aka ARMv8 / 64Bits) nodes running in a remote DC. On my (already too long) TODO/TOTEST list I had the idea of testing armhfp VM on top of aarch64. Reason is that when I need to test our packages, using my own Cubietruck or RaspberryPi3 is time consuming : removing the sdcard, reflashing with the correct CentOS 7 image and booting/testing the pkg/update/etc ...

So is that possible to just automate this through available aarch64 node as hypervisor ? Sure ! and it's just pretty straightforward if you have already played with libvirt. Let's so start with a CentOS 7 aarch64 minimal setup and then :

yum install qemu-kvm-tools qemu-kvm virt-install libvirt libvirt-python libguestfs-tools-c
systemctl enable libvirtd --now

That's pretty basic but for armhfp we'll have to do some extra steps : qemu normally tries to simulate a bios/uefi boot, which armhfp doesn't support, and qemu doesn't emulate the mandatory uboot to just chainload to the RootFS from the guest VM.

So here is just what we need :

  • Import the RootFS from an existing image
curl http://mirror.centos.org/altarch/7/isos/armhfp/CentOS-Userland-7-armv7hl-Minimal-1708-CubieTruck.img.xz|unxz >/var/lib/libvirt/images/CentOS-Userland-7-armv7hl-Minimal-1708-CubieTruck.img
  • Convert image to qcow2 (that will give us more flexibility) and extend it a little bit
qemu-img convert -f raw -O qcow2 /var/lib/libvirt/images/CentOS-Userland-7-armv7hl-Minimal-1708-CubieTruck.img /var/lib/libvirt/images/CentOS-Userland-7-armv7hl-Minimal-1708-guest.qcow2
qemu-img resize /var/lib/libvirt/images/CentOS-Userland-7-armv7hl-Minimal-1708-guest.qcow2 +15G
  • Extract kernel+initrd as libvirt will boot that directly for the VM
mkdir /var/lib/libvirt/armhfp-boot
virt-copy-out -a /var/lib/libvirt/images/CentOS-Userland-7-armv7hl-Minimal-1708-guest.qcow2 /boot/ /var/lib/libvirt/armhfp-boot/

So now that we have a RootFS, and also kernel/initrd, we can just use virt-install to create the VM (pointing to existing backend qcow2) :

virt-install \
 --name centos7_armhfp \
 --memory 4096 \
 --boot kernel=/var/lib/libvirt/armhfp-boot/boot/vmlinuz-4.9.40-203.el7.armv7hl,initrd=/var/lib/libvirt/armhfp-boot/boot/initramfs-4.9.40-203.el7.armv7hl.img,kernel_args="console=ttyAMA0 rw root=/dev/sda3" \
 --disk /var/lib/libvirt/images/CentOS-Userland-7-armv7hl-Minimal-1708-guest.qcow2 \
 --import \
 --arch armv7l \
 --machine virt \

And here we go : we have a armhfp VM that boots really fast (compared to a armhfp board using a microsd card of course)

At this stage, you can configure the node, etc.. The only thing you have to remember is that of course kernel will be provided from outside the VM, so just extract it from an updated VM to boot on that kernel. Let's show how to do that, as in the above example, we configured the VM to run with 4Gb of ram, but only 3 are really seen inside (remember the 32bits mode and so the need for PAE on i386 ?)

So let's use this example to show how to switch kernel : From the armhfp VM :

# Let extend first as we have bigger disk
growpart /dev/sda 3
resize2fs /dev/sda3
yum update -y
yum install kernel-lpae
systemctl poweroff # we'll modify libvirt conf file for new kernel

Back to the hypervisor we can again extract needed files :

virt-copy-out -a /var/lib/libvirt/images/CentOS-Userland-7-armv7hl-Minimal-1708-guest.qcow2 /boot/vmlinuz-4.9.50-203.el7.armv7hl+lpae /var/lib/libvirt/armhfp-boot/boot/
virt-copy-out -a /var/lib/libvirt/images/CentOS-Userland-7-armv7hl-Minimal-1708-guest.qcow2 /boot/initramfs-4.9.50-203.el7.armv7hl+lpae.img /var/lib/libvirt/armhfp-boot/boot/

And just virsh edit centos7_armhfp so that kernel and armhfp are pointing to correct location:

<kernel>/var/lib/libvirt/armhfp-boot/boot/vmlinuz-4.9.50-203.el7.armv7hl+lpae</kernel>
<initrd>/var/lib/libvirt/armhfp-boot/boot/initramfs-4.9.50-203.el7.armv7hl+lpae.img</initrd>

Now that we have a "gold" image, we can even use exiting tools to provision quickly other nodes on that hypervisor ! :

time virt-clone --original centos7_armhfp --name armhfp_guest1 --file /var/lib/libvirt/images/armhfp_guest1.qcow2
Allocating 'armhfp_guest1.qcow2'                                               |  18 GB  00:00:02     

Clone 'armhfp_guest1' created successfully.

real    0m2.809s
user    0m0.473s
sys 0m0.062s

time virt-sysprep --add /var/lib/libvirt/images/armhfp_guest1.qcow2 --operations defaults,net-hwaddr,machine-id,net-hostname,ssh-hostkeys,udev-persistent-net --hostname guest1

virsh start armhfp_guest1

As simple as that. Of course, in the previous example we were just using the default network from libvirt, and not any bridge, but you get the idea : all the rest with well-known concept for libvirt on linux.

When you are performing penetration tests for your customers, you need to build your personal arsenal. Tools, pieces of hardware and software are collected here and there depending on your engagements to increase your toolbox. To perform Wireless intrusion tests, I’m a big fan of the WiFi Pineapple. I’ve one for years (model MK5). It’s not the very latest but it still does a good job. But, recently, after a discussion with a friend, I bought a new wireless toy: the WiNX!

The device is very small (3.5 x 3 CM) based on an ESP-WROOM32 module. It comes with a single interface: a micro USB port to get some power and provide the serial console. No need to have a TCP/IP stack or a browser to manage it, you can just connect it to any device that has a USB port and a terminal emulator (minicom, putty, screen, …). It could be not very user-friendly to some of you but I really like this! The best solution that I found until now is to use the Arduino IDE and its serial monitor tool. You can type your commands in the dedicated field and get the results in the main window:

Capture-WiNX

The device can be flashed with different versions of the firmware that offer the following core features. You can use the WiNX as:

  •  a WiFi scanner
  • a WiFi sniffer
  • a WiFi honeypot

Of course, my preferred mode is the honeypot. If the firmware comes with default example of captive portals, it’s very easy to design your own. The only restrictions are the size of the HTML page (must be less than 150KB) and it must include all the components (CSS, images – Base64 encoded). The form may contain your own fields (ex: add a token, CAPTCHA, CC number, etc) and must just post to the “/”, the web server, to see all the fields logged on the internal storage.

Here is an example of a deceptive page that I made for testing purposes:

Exki Portal Sample

To use the device, you just need to plug it into a computer and it boots in a few seconds. Even better, you can use with a power bank and leave it in a discreet place! Cheap, small, easy to manage, I’d definitively recommend adding this gadget to your arsenal!

 

[The post WiNX: The Ultra-Portable Wireless Attacking Platform has been first published on /dev/random]

I published the following diary on isc.sans.org: “The easy way to analyze huge amounts of PCAP data“.

When you are investigating a security incident, there are chances that, at a certain point, you will have to dive into network traffic analysis. If you’re lucky, you’ll have access to a network capture. Approximatively one year ago, I wrote a quick diary to explain how to implement a simple FPC or “Full Packet Capture” solution based on a Docker container. It’s nice to capture all the traffic in PCAP files but then? PCAP files are not convenient to process and they consume a lot of disk space (depending on the captured traffic of course)… [Read more]

 

[The post [SANS ISC] The easy way to analyze huge amounts of PCAP data has been first published on /dev/random]

September 27, 2017

Ce jeudi 19 octobre 2017 à 19h se déroulera la 62ème séance montoise des Jeudis du Libre de Belgique.

Le sujet de cette séance : Appliquer les licences libres à la création numérique

Thématique : Graphisme|création|licences

Public : Tout public

L’animateur conférencier : Robert Viseur

Lieu de cette séance : Campus technique (ISIMs) de la Haute Ecole en Hainaut, Avenue V. Maistriau, 8a, Salle Académique, 2e bâtiment (cf. ce plan sur le site de l’ISIMs, et ici sur la carte Openstreetmap).

La participation sera gratuite et ne nécessitera que votre inscription nominative, de préférence préalable, ou à l’entrée de la séance. Merci d’indiquer votre intention en vous inscrivant via la page http://jeudisdulibre.fikket.com/. La séance sera suivie d’un verre de l’amitié. L’organisation bénéficie du soutien de la Fédération Wallonie-Bruxelles dans le cadre de la Saison des Cultures Numériques, et en particulier de la Quinzaine Numérique @ Mons (cf. le calendrier complet des activités), du 28 septembre au 30 novembre 2017.

Les Jeudis du Libre à Mons bénéficient aussi du soutien de nos partenaires : CETIC, OpenSides, MeaWeb et Phonoid.

Si vous êtes intéressé(e) par ce cycle mensuel, n’hésitez pas à consulter l’agenda et à vous inscrire sur la liste de diffusion afin de recevoir systématiquement les annonces.

Pour rappel, les Jeudis du Libre se veulent des espaces d’échanges autour de thématiques des Logiciels Libres. Les rencontres montoises se déroulent chaque troisième jeudi du mois, et sont organisées dans des locaux et en collaboration avec des Hautes Écoles et Facultés Universitaires montoises impliquées dans les formations d’informaticiens (UMONS, HEH et Condorcet), et avec le concours de l’A.S.B.L. LoLiGrUB, active dans la promotion des logiciels libres.

Description : Les créations numériques, dans tous les domaines, qu’ils soient artistique, scientifique, éducatif ou autres se doivent de préciser leur licence en matière de propriété intellectuelle. Cela conditionne fortement, à court et long terme, la diffusion des œuvres, et en cette matière, ne rien spécifier ou le faire tardivement ne peut qu’amener des difficultés aux auteurs.

Les licences libres, ou de libre diffusion, de type Creative Commons ou autres, permettent de s’en affranchir. La conférence présentera ces licences ainsi que les enjeux, en particulier légaux et artistiques, qui y sont associés. Des exemples concrets d’usages seront présentés (p.ex. Copyleft Attitude et Wikimedia Commons).

Group photo

Yesterday, I shared my State of Drupal presentation at DrupalCon Vienna. In addition to sharing my slides, I wanted to provide some more detail on how Drupal is evolving, who Drupal is for, and what I believe we should focus on.

Drupal is growing and changing

I started my keynote by explaining that Drupal is growing. Over the past year, we've witnessed a rise in community engagement, which has strengthened Drupal 8 adoption.

This is supported by the 2017 Drupal Business Survey; after surveying 239 executives from Drupal agencies, we can see that Drupal 8 has become the defacto release for them and that most of the Drupal businesses report to be growing.

Drupal agencies report that most Drupal projects are using Drupal 8 now
Most Drupal agencies have growing businesses

While the transition from Drupal 7 to Drupal 8 is not complete, Drupal 8's innovation continues to accelerate. We've seen the contributed modules ecosystem mature; in the past year, the number of stable modules has more than doubled. Additionally, there are over 4,000 modules in development.

We saw a 2x increase in stable contributed modules

In addition to growth, both the vendor and technology landscapes around Drupal are changing. In my keynote, I noted three primary shifts in the vendor landscape. Single blogs, portfolio sites and brochure sites, which represent the low end of the market, are best served by SaaS tools. On the other side of the spectrum, a majority of enterprise vendors are moving beyond content management into larger marketing suites. Finally, the headless CMS market segment is growing rapidly, with some vendors growing at a rate of 500% year over year.

There are also significant changes in the technology landscape surrounding Drupal, as a rising number of Drupal agencies have also started using modern JavaScript technologies. For example, more than 50% of Drupal agencies are also using Node.js to support the needs of their customers.

What other technologies do Drupal agencies use?

While evolving vendor and technology landscapes present many opportunities for Drupal, it can also introduce uncertainty. After listening to many people in the Drupal community, it's clear that all these market and technology trends, combined with the long development and adoption cycle of Drupal 8, has left some wondering what this all means for Drupal, and by extension also for them.

Drupal is no longer for simple sites

Over the past year, I've explained why I believe Drupal is for ambitious digital experiences, in both my DrupalCon Baltimore keynote and on my blog. However, I think it would be valuable to provide more detail on what I mean by "ambitious digital experiences". It's important that we all understand who Drupal is for, because it drives our strategy, which in turn allows us to focus our efforts.

Today, I believe that Drupal is no longer for simple sites. Instead, Drupal's sweetspot is sites or digital experiences that require a certain level of customization or flexibility — something I refer to as "richness".

Drupal is no longer for simple sites, but for sites with medium-to-high richness

Ambitious is much more than just enterprise

This distinction is important because I often find that the term "ambitious" becomes conflated with "enterprise". While I agree that Drupal is a great fit for the enterprise, I personally never loved that categorization. It's not just large organizations that use Drupal. Individuals, small startups, universities, museums and nonprofits can be equally ambitious in what they'd like to accomplish and Drupal can be an incredible solution for them.

An example of this could be a small business that manages 50 rental properties. While they don't have a lot of traffic (reach), they require integrations with an e-commerce system, a booking system, and a customer support tool to support their business. Their allotted budget is $50,000 or less. This company would not be considered an enterprise business; however, Drupal would be a great fit for this use case. In many ways, the "non-enterprise ambitious digital experiences" represent the majority of the Drupal ecosystem. As I made clear in my presentation, we don't want to leave those behind.

Drupal is for digital ambitious experiences

Addressing the needs of smaller organizations

The Drupal ecosystem majority are organizations with sites that require medium-to-high richness, which SaaS builders cannot support. However, they also don't need to scale at the level of enterprise companies. As the Drupal community continues to consider how we can best support this majority, a lot of smaller Drupal agencies and end-users have pointed out that they would benefit from the following two things:

  1. Powerful site building tools. They want easy-to-use site building tools that are simple to learn, and don't require dozens of contributed modules to be installed and configured. They would also prefer to avoid writing a lot of custom code because their clients have smaller budgets. Great examples of tools that would improve site building are Drupal's upcoming layout builder, workspaces and media library. To make some of Drupal's own administrative UIs more powerful and easier to use, I proposed that we add a modern JavaScript to core.
  2. Easier updates and maintenance. While each Drupal 8 site benefits from continuous innovation, it also needs to be updated more often. The new Drupal 8 release cycle has monthly patch releases and 6-month minor releases. In addition, organizations have to juggle ad-hoc updates from contributed modules. In addition, site updates has often become more complex because our dependency on third-party libraries and because not everyone can use Composer. Many smaller users and agencies would benefit tremendously from auto-updates because maintaining and updating their Drupal 8 sites can be too manual, too complex and too expensive.

The good news is that we have made progress in both improving site builder tools and simplifying updates and maintenance. Keep an eye on future blog posts about these topics. In the meantime, you can watch a recording of my keynote (starting at 22:10), or you can download a copy of my slides (56 MB).

September 26, 2017

A few days ago, Jason "perfinion" Zaman stabilized the 2.7 SELinux userspace on Gentoo. This release has quite a few new features, which I'll cover in later posts, but for distribution packagers the main change is that the userspace now has many more components to package. The project has split up the policycoreutils package in separate packages so that deployments can be made more specific.

Let's take a look at all the various userspace packages again, learn what their purpose is, so that you can decide if they're needed or not on a system. Also, when I cover the contents of a package, be aware that it is based on the deployment on my system, which might or might not be a complete installation (as with Gentoo, different USE flags can trigger different package deployments).

libsepol - manipulating SELinux binary policies

The first package, known in Gentoo as sys-libs/libsepol, is the library that enables manipulating the SELinux binary policies. This is a core library, and is the first SELinux userspace package that is installed on a system.

It contains one command, chkcon, which allows users to validate if a specific security context exists within a binary policy file:

~$ chkcon policy.29 user_u:user_r:mozilla_t:s0
user_u:user_r:mozilla_t:s0 is valid

The package does contain two manpages of old commands which are no longer available (or I'm blind, either way, they're not installed and not found in the SELinux userspace repository either) such as genpolusers and genpolbools.

libselinux - the main SELinux handling library

The libselinux library, known in Gentoo as sys-libs/libselinux, is the main SELinux library. Almost all applications that are SELinux-aware (meaning they not only know SELinux is a thing, but are actively modifying their behavior with SELinux-specific code) will link to libselinux.

Because it is so core, the package also provides the necessary bindings for different scripting languages besides the standard shared objects approach, namely Python (as many SELinux related tooling is written in Python) and Ruby.

Next to the bindings and libraries, libselinux also offers quite a few executables to query and manipulate SELinux settings on the system, which are shortly described on the SELinux userspace wiki but repeated here for convenience. Most of these are meant for debugging purposes, as they are simple wrappers toward the libselinux provided functions, but some of them are often used by administrations.

  • avcstat gives statistics about the in-kernel access vector cache, such as number of lookups, hits and misses
  • compute_create queries the kernel security server for a transition decision
  • compute_av queries the kernel security server for an access vector decision
  • compute_relabel queries the kernel security server for a relabel decision
  • compute_member queries the kernel security server for a labeling decision on a polyinstantiated object
  • getconlist uses the security\_compute\_user() function, and orders the resulting list based on the default\_contexts file and per-user context files
  • getdefaultcon is like getconlist but only returns the first context
  • compute_user queries the kernel security server fo a set of reachable user contexts from a source context
  • getfilecon gets the context of a file by path
  • getpidcon gets the context of a process by PID
  • getseuser queries the seuser file for the resulting SELinux user and contxt for a particular linux login and login context
  • getsebool gets the current state of a SELinux boolean in the SELinux security server
  • matchpathcon queries the active filecontext file for how a particular path should be labeled
  • policyvers queries the kernel security server for the maximum policy version supported
  • getenforce gets the enforcing state of the kernel access vector cache
  • sefcontext_compile generates binary filecontext files, optimized for fast querying
  • selabel_lookup looks up what the target default context is for various classes (supporting the X related SELinux types, database types, etc.)
  • selabel_digest calculates the SHA1 digest of spec files, and returns a list of the specfiles used to calculate the digest. This is used by Android.
  • selabel_partial_match determines if a direct or partial match is possible on a file path
  • selabel_lookup_best_match obtains the best matching SELinux security context for file-based operations
  • selinux_check_securetty_context checks whether a SELinux tty security context is defined as a securetty context
  • selinux_check_access checks if the source context has the access permission for the specified class on the target context
  • selinuxexeccon reports the SELinux context for an executable
  • selinuxenabled returns if SELinux is enabled or not
  • setfilecon sets the context of a path
  • setenforce sets the enforcing state of the kernel access vector cache
  • togglesebool toggles a SELinux boolean, but only runtime (so it does not persist across reboots)

checkpolicy - policy compiler

The checkpolicy package, known in Gentoo as sys-apps/checkpolicy, provides two main applications, checkpolicy and checkmodule. Both applications are compilers (unlike what the name implies) which build a binary SELinux policy. The main difference between these two is that one builds a policy binary, whereas the other one builds a SELinux module binary.

Developers don't often call these applications themselves, but use the build scripts. For instance, the semodule_package binary would be used to combine the binary policy with additional files such as file contexts.

libsemanage - facilitating use of SELinux overall

The libsemanage library, known in Gentoo as sys-libs/libsemanage, contains SELinux supporting functions that are needed for any regular SELinux use. Whereas libselinux would be used everywhere, even for embedded systems, libsemanage is generally not for embedded systems but is very important for Linux systems in overall.

Most SELinux management applications that administrators come in contact with will be linked with the libsemanage library. As can be expected, the semanage application as offered by the selinux-python package is one of them.

The only application that is provided by libsemanage is the semanage_migrate_store, used to migrate the policy store from the /etc/selinux to the /var/lib/selinux location. This was done with the introduction of the 2.4 userspace.

selinux-python - Python-based command-line management utilities

The selinux-python package, known in Gentoo as sys-apps/selinux-python, is one of the split packages that originally where part of the policycoreutils package. It contains the majority of management utilities that administrators use for handling SELinux on their systems.

The most known application here is semanage, but it contains quite a few others as well:

  • sepolgen generates an initial SELinux policy module template, and is short for the sepolicy generate command
  • audit2why translates SELinux audit messages into a description of why the access was denied. It is short for the audit2allow -w command.
  • audit2allow generates SELinux policy allow/dontaudit rules from logs of denied operations
  • sepolgen-ifgen generates an overview of available interfaces. This overview is used by audit2allow to guess the right interface to use when allowing or dontauditing certain operations.
  • sepolicy is the SELinux policy inspection tool, allowing to query various aspects of a SELinux configuration (namely booleans, communication flows, interfaces, network information and transition information). It also provides the ability to generate skeleton policies (as described with sepolgen) and manual pages.
  • chcat changes a file's SELinux security category
  • sepolgen-ifgen-attr-helper generates an overview of attributes and attribute mappings. This overview is used by audit2allow to guess the right attribute to use when allowing or dontauditing certain operations.
  • semanage is a SELinux policy management tool, allowing a multitude of operations against the SELinux policy and the configuration. This includes definition import/export, login mappings, user definitions, ports and interface management, module handling, file contexts, booleans and more.

semodule-utils - Developing SELinux modules

The semodule-utils package, known in Gentoo as sys-apps/semodule-utils, is another split package that originally was part of the policycoreutils package. In it, SELinux policy module development utilities are provided. The package is not needed for basic operations such as loading and unloading modules though.

  • semodule_expand expands a SELinux base module package into a kernel binary policy file
  • semodule_deps shows the dependencies between SELinux policy packages
  • semodule_link links SELinux policy module packages together into a single SELinux policy module
  • semodule_unpackage extracts a SELinux module into the binary policy and its associated files (such as file context definitions)
  • semodule_package combines a modular binary policy file with its associated files (such as file context definitions) into a module package

mcstrans - Translate context info in human readable names

The mcstrans package, known in Gentoo as sys-apps/mcstrans, is another split package that originally was part of the policycoreutils package. In it, the MCS translation daemon is hosted. This daemon translates the SELinux-specific context ranges, like s0-s0:c0.c1024 to a human-readable set, like SystemLow-SystemHigh.

This is a purely cosmetic approach (as SELinux internally always uses the sensitivity and category numbers) but helps when dealing with a large number of separate categories.

restorecond - Automatically resetting file contexts

The restorecond package, known in Gentoo as sys-apps/restorecond, is another split package that originally was part of the policycoreutils package. It contains the restorecond daemon, which watches over files and directories and forces the right SELinux label on it.

This daemon was originally intended to resolve a missing feature in SELinux (having more fine-grained rules for label naming) but with the named file transition support, the need for this daemon has diminished a lot.

secilc - SELinux common intermediate language compiler

The secilc package, known in Gentoo as sys-apps/secilc, is the CIL compiler which builds kernel binary policies based on the passed on CIL code. Although the majority of policy development still uses the more traditional SELinux language (and supporting macro's from the reference policy), developers can already use CIL code for policy generation.

With secilc, a final policy file can be generated through the CIL code.

selinux-dbus - SELinux DBus server

The selinux-dbus package (not packaged in Gentoo at this moment) provides a SELinux DBus service which systems can use to query and interact with SELinux management utilities on the system. If installed, the org.selinux domain is used for various supported operations (such as listing SELinux modules, through org.selinux.semodule_list).

selinux-gui - Graphical SELinux settings manager

The selinux-gui package (not packaged in Gentoo at this moment) provides the system-config-selinux application which offers basic SELinux management support in a graphical application. It supports boolean handling, file labeling, user mapping, SELinux user management, network port definitions and module handling. As such, it can be seen as the graphical helper utility for the semanage command.

selinux-sandbox - Sandbox utility utilizing SELinux sandbox domains

The selinux-sandbox package (not packaged in Gentoo at this moment) is a set of scripts to facilitate the creation of SELinux sandboxes. With these utilities, which not only use SELinux sandbox domains like sandbox_t but also Linux namespaces, end users can launch applications in a restricted environment.

policycoreutils - Core SELinux management utilities

The policycoreutils package, known in Gentoo as sys-apps/policycoreutils, contains basic SELinux tooling which is necessary to handle SELinux in a regular environment. Supported utilities are:

  • newrole to switch a user session from one role to another
  • secon to query the SELinux context of a file, program or user input
  • genhomedircon to regenerate home directory context files, necessary when new users are defined on the system
  • setfiles to set SELinux file security contexts on resources
  • semodule to list, load and unload SELinux modules
  • run_init to launch an init script in the right domain
  • open_init_pty to run a program under a pseudo terminal with the right context set
  • sestatus to query current policy status
  • setsebool to set and, if wanted, persist a SELinux boolean value
  • selinuxconfig to display the current active configuration paths
  • restorecon to set SELinux file security contexts on resources
  • load_policy to load the SELinux policy, generally called from initramfs systems if the init system is not SELinux-aware
  • restorecon_xattr manages the security.restorecon_last extended attribute which is set by setfiles or restorecon

Gentoo also adds in two additional scripts: rlpkg to reset file contexts on files provided by a Gentoo package selocal to easily handle small SELinux rule additions to the active policy

There are even more

Attentive readers will notice that the setools package is not discussed here. This package is not provided by the SELinux userspace project, but is an important package for SELinux policy developers as it contains the sesearch command - an often used command to query the active policy.

The above list is thus a picture of the SELinux userspace utilities, which is becoming quite a big application set now that some functionality is split off from the policycoreutils package.

September 20, 2017

Today, Acquia announced that it expanded Acquia Cloud to support Node.js, the popular open-source JavaScript runtime. This is a big milestone for Acquia as it is the first time we have extended our cloud beyond Drupal. I wanted to take some time to explain the evolution of Acquia's open-source stack and why this shift is important for our customers' success.

From client-side JavaScript to server-side JavaScript

JavaScript was created at Netscape in 1995, when Brendan Eich wrote the first version of JavaScript in just 10 days. It took around 10 years for JavaScript to reach enterprise maturity, however. Adoption accelerated in 2004 when Google used JavaScript to build the first release of Gmail. In comparison to e-mail competitors like Yahoo! Mail and Hotmail, Gmail showed what was possible with client-side JavaScript, which enables developers to update pages dynamically and reduces full-page refreshes and round trips to the server. The benefit is an improved user experience that is usually faster, more dynamic in its behavior, and generally more application-like.

In 2009, Google invented the V8 JavaScript engine, which was embedded into its Chrome browser to make both Gmail and Google Maps faster. Ryan Dahl used the V8 run-time as the foundation of Node.js, which enabled server-side JavaScript, breaking the language out of the boundaries of the browser. Node.js is event-driven and provides asynchronous, non-blocking I/O — things that help developers build modern web applications, especially those with real-time capabilities and streamed data. It ushered in the era of isomorphic applications, which means that JavaScript applications can now share code between the client side and server side. The introduction of Node.js has spurred a JavaScript renaissance and contributed to the popularity of JavaScript frameworks such as AngularJS, Ember and React.

Acquia's investment in Headless Drupal

In the web development world, few trends are spreading more rapidly than decoupled architectures using JavaScript frameworks and headless CMS. Decoupled architectures are gaining prominence because architects are looking to take advantage of other front-end technologies, most commonly JavaScript based front ends, in addition to those native to Drupal.

Acquia has been investing in the development of headless Drupal for nearly five years, when we began contributing to the addition of web service APIs to Drupal core. A year ago, we released Waterwheel, an ecosystem of software development kits (SDKs) that enables developers to build Drupal-backed applications in JavaScript and Swift, without needing extensive Drupal expertise. This summer, we released Reservoir, a Drupal distribution for decoupled Drupal. Over the past year, Acquia has helped to support a variety of headless architectures, with and without Node.js. While not always required, Node.js is often used alongside of a headless Drupal application to provide server-side rendering of JavaScript applications or real-time capabilities.

Managed Node.js on Acquia Cloud

Previously, if an organization wanted to build a decoupled architecture with Node.js, it was not able to host the Node.js application on Acquia Cloud. This means that the organization would have to run Node.js with a separate vendor. In many instances, this requires organizations to monitor, troubleshoot and patch the infrastructure supporting the Node.js application of their own accord. Separating the management of the Node.js application and Drupal back end not only introduces a variety of complexities, including security risk and governance challenges, but it also creates operational strain. Organizations must rely on two vendors, two support teams, and multiple contacts to build decoupled applications using Drupal and Node.js.

To eliminate this inefficiency, Acquia Cloud can now support both Drupal and Node.js. Our goal is to offer the best platform for developing and running Drupal and Node.js applications. This means that organizations only need to rely on one vendor and one cloud infrastructure when using Drupal and Node.js. Customers can access Drupal and Node.js environments from a single user interface, in addition to tools that enable continuous delivery, continuous integration, monitoring, alerting and support across both Drupal and Node.js.

Acquia Cloud now supports Node.js for Headless DrupalOn Acquia Cloud, customers can access Drupal and Node.js environments from a single user interface.

Delivering on Acquia's mission

When reflecting on Acquia's first decade this past summer, I shared that one of the original corporate values our small team dreamed up was to "empower everyone to rapidly assemble killer websites". After ten years, we've evolved our mission to "build the universal platform for the world's greatest digital experiences". While our focus has expanded as we've grown, Acquia's enduring aim is to provide our customers with the best tools available. Adding Node.js to Acquia Cloud is a natural evolution of our mission.

The post DNS Research: using SPF to query internal DNS resolvers appeared first on ma.ttias.be.

Using the SPF records to trigger a response from an internal DNS server. Clever way to extract otherwise closed data!

In response to the spread of cache poisoning attacks, many DNS resolvers have gone from being open to closed resolvers, meaning that they will only perform queries on behalf of hosts within a single organization or Internet Service Provider.

As a result, measuring the security of the DNS infrastructure has been made more difficult. Closed resolvers will not respond to researcher queries to determine if they utilize security measures like port randomization or transaction id randomization.

However, we can effectively turn a closed resolver into an open one by sending an email to a mail server (MTA) in the organization. This causes the MTA to make a query on the external researchers' behalf, and we can log the security features of the DNS resolver using information gained by a nameserver and email server under our control.

Source: DNS Research

The post DNS Research: using SPF to query internal DNS resolvers appeared first on ma.ttias.be.

September 18, 2017

The post A proposal for cryptocurrency addresses in DNS appeared first on ma.ttias.be.

By now it's pretty clear that the idea of a cryptocurrency probably isn't going away. It might not be Bitcoin or Litecoin, it might not have the same value as it does today, but the concept of cryptocurrency is here to stay: digital money.

Just like the beginning of IP addresses, using them raw was fine at first. But with crypto, you get long hexadecimal strings that truly no one can remember by heart. It's far from user friendly.

It's like trying to remember that 2a03:a800:a1:1952::ff is the address for this site. Doesn't work very well, does it? It's far easier to say ma.ttias.be than the cryptic representation of IPv6.

I think we need something similar for cryptocurrencies. Something independent and -- relatively -- secure. So here's my proposal I came up with in the car on the way home.

Example: cryptocurrency in DNS

Here's the simplest example I can give.

$ dig ma.ttias.be TXT | sort
ma.ttias.be.	3600	IN    TXT   "ico:10 btc:1AeCyEczAFPVKkvausLSQWP1jcqkccga9m"
ma.ttias.be.	3600	IN    TXT   "ico:10 ltc:Lh1TUmh2WP4LkCeDTm3kMX1E7NQYSKyMhW"
ma.ttias.be.	3600	IN    TXT   "ico:20 eth:0xE526E2Aecb8B7C77293D0aDf3156262e361BfF0e"
ma.ttias.be.	3600	IN    TXT   "ico:30 xrp:rDsbeomae4FXwgQTJp9Rs64Qg9vDiTCdBv"

Cryptocurrency addresses get published as TXT records to a domain of your choosing. Want to receive a payment? Simple say "send it to ma.ttias.be", the client will resolve that TXT record and the accompanying addresses and use the priority field as a guideline for choosing which address to pick first.

Think MX records, but implemented as TXT. The lower the priority, the more preferred it is.

The TXT format explained

A TXT format can contain pretty much anything, so it needs some standardization in order for this to work. Here's my proposal.

ico:[priority] space [currency]:[address]

Let's pick the first result as an example and tear it down.

$ dig ma.ttias.be TXT | sort | head -n 1
ma.ttias.be.	3600	IN    TXT   "ico:10 btc:1AeCyEczAFPVKkvausLSQWP1jcqkccga9m"

Translates to;

  • ico:: a prefix in the TXT record to denote that this ia currency, much like SPF knows the "v=spv1" prefix. This is a fixed value.
  • 10: the priority. The lower, the bigger its preference.
  • btc: preferred currency is btc, or Bitcoin.
  • 1AeCyEczAFPVKkvausLSQWP1jcqkccga9m: the btc address to accept payments.

Simple, versatile format.

The priority allows for round robin implementations, if you wish to diversify your portfolio. Adding multiple cryptocurrency allows the sender the freedom to choose which currency he/she prefers, while still honoring your priority.

Technically, I published 2 records with a priority of 10. It's up to the sender to determine which currency he/she prefers, if it's available to them. If it isn't, they can move down the chain & try other addresses published.

It means only addresses on which you want to receive currency should ever be posted as DNS records.

DNSSEC required

To avoid DNS cache poisoning or other man-in-the-middle attacks, DNSSEC would have to be a hard requirement in order to guarantee integrity.

This should not be optional.

If smart people every end up implementing something like this, an additional GPG/PKI like solution might be added for increased security, by signing the addresses once more.

Currency agnostic

This isn't a solution for Bitcoin, Litecoin or Ripple. It's a solution for all of them. And all the new currencies to come.

It's entirely currency agnostic and can be used for any virtual currency.

Multi tenancy

If you want multiple users on the same domain, you could solve this via subdomains. Ie "john.domain.tld", "mary.domain.tld", ...

It makes the format of the TXT record plain and simple and uses basic DNS records for delegation of accounts.

Why not a dedicated resource record?

For the same reason the SPF resource record went away and was replaced by a TXT alternative: availability.

Every DNS server and client already understands TXT records. If we have to wait for both servers, clients and providers to implement something like a ICO resource record, it'll take ages. Just look at the current state of CAA records, only a handful of providers offer it, even though it's a mandatory CA thing already.

There are already simpler naming schemes for cryptocurrency!

Technically, yes, but they all have a deep flaw: you have to trust someone else's system.

There's BitAlias, onename, ens for ethereum, okTurtles, ... and they all build on top of their own, custom system.

But it turns out, we already have a name-translation-system called DNS, we'd be far better of implementing a readable cryptocurrency variant in DNS than in someone else's closed system.

The validator regex

With the example given above, it can easily be validated with the following regex.

ico:([0-9]+) ([a-z]{3,}):([a-zA-Z0-9]+)

And it translates to;

  • group #1: the priority
  • group #2: the currency
  • group #3: the address

A complete validator with the dig DNS client would translate to;

$ dig ma.ttias.be TXT | sort | grep -P 'ico:([0-9]+) ([a-z]{3}):([a-zA-Z0-9]+)'
ma.ttias.be.	3600	IN    TXT   "ico:10 btc:1AeCyEczAFPVKkvausLSQWP1jcqkccga9m"
ma.ttias.be.	3600	IN    TXT   "ico:10 ltc:Lh1TUmh2WP4LkCeDTm3kMX1E7NQYSKyMhW"
ma.ttias.be.	3600	IN    TXT   "ico:20 eth:0xE526E2Aecb8B7C77293D0aDf3156262e361BfF0e"
ma.ttias.be.	3600	IN    TXT   "ico:30 xrp:rDsbeomae4FXwgQTJp9Rs64Qg9vDiTCdBv"

Now, who's going to make this an RFC? I certainly won't, I've got too many things to do already.

The post A proposal for cryptocurrency addresses in DNS appeared first on ma.ttias.be.

I published the following diary on isc.sans.org: “Getting some intelligence from malspam“.

Many of us are receiving a lot of malspam every day. By “malspam”, I mean spam messages that contain a malicious document. This is one of the classic infection vectors today and aggressive campaigns are started every week. Usually, most of them are blocked by modern antivirus or anti-spam but these files could help us to get some intelligence about the topic used by attackers to fool their victims. By checking the names of malicious files (often .rar, .gip or .7r archives), we found classic words like ‘invoice’, ‘reminder’, ‘urgent’, etc… [Read more]

[The post [SANS ISC] Getting some intelligence from malspam has been first published on /dev/random]

September 17, 2017

The post Chrome to force .dev domains to HTTPS via preloaded HSTS appeared first on ma.ttias.be.

tl;dr: one of the next versions of Chrome is going to force all domains ending on .dev (and .foo) to be redirected to HTTPs via a preloaded HTTP Strict Transport Security (HSTS) header.


This very interesting commit just landed in Chromium:

Preload HSTS for the .dev gTLD.

This adds the following line to Chromium's preload lists;

{ "name": "dev", "include_subdomains": true, "mode": "force-https" },
{ "name": "foo", "include_subdomains": true, "mode": "force-https" },

It forces any domain on the .dev gTLD to be HTTPs.

Wait, there's a legit .dev gTLD?

Yes, unfortunately.

It's been bought by Google as one of their 100+ new gTLDs. What do they use it for? No clue. But it's going to cause a fair bit of confusion and pain to webdevelopers.

The .dev gTLD has nameservers and is basically like any other TLD out there, we as developers just happen to have chosen that name as a good placeholder for local development, too, overwriting the public DNS.

$ dig +trace dev. NS
dev.			172800	IN	NS	ns-tld4.charlestonroadregistry.com.
dev.			172800	IN	NS	ns-tld5.charlestonroadregistry.com.
dev.			172800	IN	NS	ns-tld3.charlestonroadregistry.com.
dev.			172800	IN	NS	ns-tld2.charlestonroadregistry.com.
dev.			172800	IN	NS	ns-tld1.charlestonroadregistry.com.

Google publishes some of their domains on there, too;

$ dig +trace google.dev A
google.dev.		3600	IN	A	127.0.53.53

So yes, it's a legit TLD.

Consequences of redirecting .dev to HTTPS

A lot of (web) developers use a local .dev TLD for their own development. Either by adding records to their /etc/hosts file or by using a system like Laravel Valet, which runs a dnsmasq service on your system to translate *.dev to 127.0.0.1.

In those cases, if you browse to http://site.dev, you'll be redirect to https://site.dev, the HTTPS variant.

That means your local development machine needs to;

  • Be able to serve HTTPs
  • Have self-signed certificates in place to handle that
  • Have that self-signed certificate added to your local trust store (you can't dismiss self-signed certificates with HSTS, they need to be 'trusted' by your computer)

Such fun.

What should we do?

With .dev being an official gTLD, we're most likely better of changing our preferred local development suffix from .dev to something else.

There's an excellent proposal to add the .localhost domain as a new standard, which would be more appropriate here. It would mean we no longer have site.dev, but site.localhost. And everything at *.localhost would automatically translate to 127.0.0.1, without /etc/hosts or dnsmasq workarounds.

Alternatively, if you're looking for a quick "search and replace" alternative for existing setups, consider the .test gTLD, which is a reserved name by IETF for testing (or development) purposes.

I do hope the Chromium team reconsiders the preloaded HSTS as it's going to have rather big implications for local webdevelopment.

The post Chrome to force .dev domains to HTTPS via preloaded HSTS appeared first on ma.ttias.be.

September 15, 2017

Last week, Equifax, one of the largest American credit agencies, was hit by a cyberattack that may have compromised the personal data of nearly 143 million people, including name, address, social security numbers, birth dates and more. The forfeited information reveals everything required to steal someone's identity or to take out a loan in someone else's name. Considering that the current US population is 321 million, this cyberattack is now considered to be one of the largest and most intrusive breaches in US history.

It's Equifax that is to blame, not open-source

As Equifax began to examine how the breach occurred, many unsubstantiated reports and theories surfaced in an attempt to pinpoint the vulnerability. One such theory targeted Apache Struts as the software responsible for the breach. Because Apache Struts is an open-source framework used for developing Java applications, this resulted in some unwarranted open-source shaming.

Yesterday, Equifax confirmed that the security breach was due to an Apache Struts vulnerability. However, here is what is important; it wasn't because Apache Struts is open-source or because open-source is less secure. Equifax was hacked because the firm failed to patch a well-known Apache Struts flaw that was disclosed months earlier in March. Running an old, insecure version of software — open-source or proprietary — can and will jeopardize the security of any site. It's Equifax that is to blame, not open-source.

The importance of keeping software up-to-date

The Equifax breach is a good reminder of why organizations need to remain vigilant about properly maintaining and updating their software, especially when security vulnerabilities have been disclosed. In an ideal world, software would update itself the moment a security patch is released. WordPress, for example, offers automatic updates in an effort to promote better security, and to streamline the update experience overall. It would be interesting to consider automatic security updates for Drupal (just for patch releases, not for minor or major releases).

In absence of automatic updates, I would encourage users to work with PaaS companies that keep not only your infrastructure secure, but also your Drupal application code. Too many organizations underestimate the effort and expertise it takes to do it themselves.

At Acquia, we provide customers with automatic security patching of both the infrastructure and Drupal code. We monitor our customers' sites for intrusion attempts, DDoS attacks, and other suspicious activity. If you prefer to do the security patching yourself, we offer continuous integration or continuous delivery tools that enable you to get security patches into production in minutes rather than weeks or months. We take pride in assisting our customers to keep their sites current with the latest patches and upgrades; it's good for our customers and helps dispel the myth that open-source software is more susceptible to security breaches.

Last week, Equifax, one of the largest American credit agencies, was hit by a cyberattack that may have compromised the personal data of nearly 143 million people, including name, address, social security numbers, birth dates and more. The forfeited information reveals everything required to steal someone's identity or to take out a loan in someone else's name. Considering that the current US population is 321 million, this cyberattack is now considered to be one of the largest and most intrusive breaches in US history.

It's Equifax that is to blame, not open-source

As Equifax began to examine how the breach occurred, many unsubstantiated reports and theories surfaced in an attempt to pinpoint the vulnerability. One such theory targeted Apache Struts as the software responsible for the breach. Because Apache Struts is an open-source framework used for developing Java applications, this resulted in some unwarranted open-source shaming.

Yesterday, Equifax confirmed that the security breach was due to an Apache Struts vulnerability. However, here is what is important; it wasn't because Apache Struts is open-source or because open-source is less secure. Equifax was hacked because the firm failed to patch a well-known Apache Struts flaw that was disclosed months earlier in March. Running an old, insecure version of software — open-source or proprietary — can and will jeopardize the security of any site. It's Equifax that is to blame, not open-source.

The importance of keeping software up-to-date

The Equifax breach is a good reminder of why organizations need to remain vigilant about properly maintaining and updating their software, especially when security vulnerabilities have been disclosed. In an ideal world, software would update itself the moment a security patch is released. WordPress, for example, offers automatic updates in an effort to promote better security, and to streamline the update experience overall. It would be interesting to consider automatic security updates for Drupal (just for patch releases, not for minor or major releases).

In absence of automatic updates, I would encourage users to work with PaaS companies that keep not only your infrastructure secure, but also your Drupal application code. Too many organizations underestimate the effort and expertise it takes to do it themselves.

At Acquia, we provide customers with automatic security patching of both the infrastructure and Drupal code. We monitor our customers' sites for intrusion attempts, DDoS attacks, and other suspicious activity. If you prefer to do the security patching yourself, we offer continuous integration or continuous delivery tools that enable you to get security patches into production in minutes rather than weeks or months. We take pride in assisting our customers to keep their sites current with the latest patches and upgrades; it's good for our customers and helps dispel the myth that open-source software is more susceptible to security breaches.

September 14, 2017

The post Laravel Horizon: requires ext-posix, missing from CentOS appeared first on ma.ttias.be.

Here's what I ran into when I tried to install a project that required laravel/horizon via Composer.

$ composer install
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Your requirements could not be resolved to an installable set of packages.

  Problem 1
    - Installation request for laravel/horizon v1.0.3 -> satisfiable by laravel/horizon[v1.0.3].
    - laravel/horizon v1.0.3 requires ext-posix * -> the requested PHP extension posix is missing from your system.
...

The error message requires ext-posix * -> the requested PHP extension posix is missing from your system is actually confusing. On CentOS, there's no PHP package called 'posix', even though the PHP module is called POSIX.

$ php -m | grep posix

(If that doesn't return any results, the posix extension is missing.)

On CentOS, the package you're looking for is called process, as it contains a set of functions/methods to help with creating child processes, sending signals, parsing ID/GIDs, ...

If you're using the IUS repositories on CentOS/Red Hat, you can install them via;

$ yum install php71u-process

Afterwards, if you run composer again, it'll work. To verify if the posix extension is installed properly, run php -m again.

 $ php -m | grep posix
posix

Now, the posix extension is installed.

The post Laravel Horizon: requires ext-posix, missing from CentOS appeared first on ma.ttias.be.

I published the following diary on isc.sans.org: “Another webshell, another backdoor!“.

I’m still busy to follow how webshells are evolving… I recently found another backdoor in another webshell called “cor0.id”. The best place to find webshells remind pastebin.com[1]. When I’m testing a webshell, I copy it in a VM located on a “wild Internet” VLAN in my home lab with, amongst other controls, full packet capture enabled. This way, I can spot immediately is the VM is trying to “phone home” to some external hosts. This was the case this time! [Read more]

 

[The post [SANS ISC] Another webshell, another backdoor! has been first published on /dev/random]

The post Presentation: Code Obfuscation, PHP shells & more appeared first on ma.ttias.be.

"What hackers do once they get past your code."

I gave this talk a while back and refreshed it a bit for DrupalCamp Antwerpen last Friday. It's a very fun topic where I get to show the results of a compromised website or server, from a hosting point of view.

The slides are pretty self-explanatory, but last time I checked there was also a video recording of the talk. If that makes it online, I'll make sure to add it here.

The post Presentation: Code Obfuscation, PHP shells & more appeared first on ma.ttias.be.

September 13, 2017

Last year, Matthew Tift and I examined Drupal.org's commit data to understand who develops Drupal, how much of that work is sponsored, and where that sponsorship comes from. We published our analysis in a blog post called "Who Sponsors Drupal Development?". A year later, I wanted to present an update. This year's report will also cover additional data, including gender and geographical diversity, and project sponsorship.

Understanding how an open-source project works is important because it establishes a benchmark for project health and scalability. Scaling an open-source project is a difficult task. As an open-source project's rate of adoption grows, the number of people that benefit from the project also increases. Often the open-source project also becomes more complex as it expands, which means that the economic reward of helping to improve the project decreases.

A recent article on the Bitcoin and Ethereum contributor communities illustrates this disparity perfectly. Ethereum and Bitcoin have market capitalizations valued at $30 billion and $70 billion, respectively. However, both projects have fewer than 40 meaningful contributors, and contribution isn't growing despite the rising popularity of cryptocurrency.

Number of Bitcoin contributors between 2010 and 2017According to Bitcoin's GitHub data, Bitcoin has less than 40 active contributors.
Number of Ethereum contributors between 2014 and 2017According to Ethereum's GitHub data, Ethereum has less than 20 active contributors.

Drupal, by comparison, has a diverse community of contributors. In the 12-month period between July 1, 2016 to June 30, 2017 we saw code contributions on Drupal.org from 7,240 different individuals and 889 different companies. This does not mean that Drupal is exempt from the challenges of scaling an open-source project. We hope that this report provides transparency about Drupal project development and encourages more individuals and organizations incentive to contribute. We also will highlight areas where our community can and should do better.

What is the Drupal.org credit system?

In the spring of 2015, after proposing ideas for giving credit and discussing various approaches at length, Drupal.org added the ability for people to attribute their work to an organization or customer in the Drupal.org issue queues. Maintainers of Drupal modules, themes and distributions can award issues credits to people who help resolve issues with code, translations, documentation, design and more.

Example issue credit on drupal orgA screenshot of an issue comment on Drupal.org. You can see that jamadar worked on this patch as a volunteer, but also as part of his day job working for TATA Consultancy Services on behalf of their customer, Pfizer.

Credits are a powerful motivator for both individuals and organizations. Accumulating credits provides individuals with a way to showcase their expertise. Organizations can utilize credits to help recruit developers or to increase their visibility in the Drupal.org marketplace.

While the benefits are evident, it is important to note a few of the limitations in Drupal.org's current credit system:

  • Contributing to issues on Drupal.org is not the only way to contribute. Other activities, such as sponsoring events, promoting Drupal, and providing help and mentorship, are important to the long-term health of the Drupal project. Many of these activities are not currently captured by the credit system. For this post, we chose to only look at code contributions.
  • We acknowledge that parts of Drupal are developed on GitHub and therefore aren't fully credited on Drupal.org. The actual number of contributions and contributors could be significantly higher than what we report.
  • Even when development is done on Drupal.org, the credit system is not used consistently; because using the credit system is optional, a lot of code committed on Drupal.org has no or incomplete contribution credits.
  • Not all code credits are the same. We currently don't have a way to account for the complexity and quality of contributions; one person might have worked several weeks for just one credit, while another person might receive a credit for ten minutes of work. In the future, we should consider issuing credit data in conjunction with issue priority, patch size, etc. We can also reduce the need for trivial credits by automating patch rerolls and automating coding style fixes.

Who is working on Drupal?

For our analysis we looked at all the issues that were marked "closed" or "fixed" in the 12-month period from July 1, 2016 to June 30, 2017. What we learned is that there were 23,238 issues marked "closed" or "fixed", a 22% increase from the 19,095 issues in the 2015-2016 period. Those 23,238 issues had 42,449 issue credits, a 30% increase from the 32,711 issue credits recorded in the previous year. Issue credits against Drupal core remained roughly the same year over year, meaning almost all of this growth came from increased activity in contributed projects. This is no surprise. Drupal development is cyclical, and during this period of the Drupal 8 development cycle, most of the Drupal community has been focused on porting modules from Drupal 7 to Drupal 8. Of the 42,449 issue credits reported this year, 20% (8,619 credits) were for Drupal core, while 80% (33,830 credits) went to contributed themes, modules and distributions.

Compared to the previous year, we also saw an increase in both the number of people contributing and the number of organizations contributing. Drupal.org received code contributions from 7,240 different individuals and 889 different organizations.

Contributions by individuals vs organizationsThe number of individual contributors is up 28% year over year and the number of organizations contributing is up 26% year over year.

While the number of individual contributors rose, a relatively small number of individuals still do the majority of the work. Approximately 47% of individual contributors received just one credit. Meanwhile, the top 30 contributors (the top 0.4%) account for over 17% of the total credits, indicating that these individuals put an incredible amount of time and effort in developing Drupal and its contributed projects:

RankUsernameIssues
1jrockowitz537
2dawehner421
3RenatoG408
4bojanz351
5Berdir335
6mglaman334
7Wim Leers332
8alexpott329
9DamienMcKenna245
10jhodgdon242
11drunken monkey238
12naveenvalecha196
13Munavijayalakshmi192
14borisson_191
15yongt9412189
16klausi185
17Sam152184
18miro_dietiker182
19Pavan B S180
20ajay_reddy176
21phenaproxima172
22sanchiz162
23slashrsm161
24jhedstrom155
25xjm151
26catch147
27larowlan145
28rakesh.gectcr141
29benjy139
30dhruveshdtripathi138

Out of the top 30 contributors featured, 19 were also recognized as top contributors in our 2015-2016 report. These Drupalists' dedication and continued contribution to the project has been crucial to Drupal's development. It's also exciting to see 11 new names on the list. This mobility is a testament to the community's evolution and growth.

Next, we looked at both the gender and geographic diversity of Drupal.org code contributors. While these are only two examples of diversity, this is the only available data that contributors can choose to share on their Drupal.org profiles. The reported data shows that only 6% of the recorded contributions were made by contributors that identify as female, which indicates a steep gender gap. Like in most open-source projects, the gender imbalance in Drupal is profound and underscores the need to continue fostering diversity and inclusion in our community.

Contributions by genderThe gender representation behind the issue credits. Only 6% of the recorded contributions are by women.
When measuring geographic diversity, we saw individual contributors from 6 different continents and 116 different countries:
Contributions by continent
Contributions by countryThe top 20 countries from which contributions originate. The data is compiled by aggregating the countries of all individual contributors behind each commit. Note that the geographical location of contributors doesn't always correspond with the origin of their sponsorship. Wim Leers, for example, works from Belgium, but his funding comes from Acquia, which has the majority of its customers in North America.

How much of the work is sponsored?

Drupal is used by more than one million websites. The vast majority of the individuals and organizations behind these Drupal websites never participate in the development of the project. They might use the software as it is or might not feel the need to help drive its development. We have to provide more incentive for these individuals and organizations to contribute back to the project.

Issue credits can be marked as "volunteer" and "sponsored" simultaneously (shown in jamadar's screenshot near the top of this post). This could be the case when a contributor does the minimum required work to satisfy the customer's need, in addition to using their spare time to add extra functionality.

While Drupal started out as a 100% volunteer-driven project, today the majority of the code on Drupal.org is sponsored by organizations. Only 11% of the commit credits that we examined in 2016-2017 were "purely volunteer" credits (4,498 credits), in stark contrast to the 46% that were "purely sponsored". In other words, there were four times as many "purely sponsored" credits as "purely volunteer" credits.

A few comparisons with the 2015-2016 data:

  • The credit system is used more. Between July 1, 2015 and June 30, 2016, 37% of all credits had no attribution while in the more recent period between July 1, 2016 to June 30, 2017, only 28% of credits lacked attribution. More people have become aware of the credit system, the attribution options, and their benefits.
  • Sponsored credits are growing faster than volunteer credits. Both "purely volunteer" and "purely sponsored" credits grew, but "purely sponsored" credits grew faster. There are two reasons why this could be the case: (1) more contributions are sponsored and (2) organizations are more likely to use the credit system compared to volunteers.
Contributions by volunteer vs sponsored

No data is perfect, but it feels safe to conclude that most of the work on Drupal is sponsored. At the same time, the data shows that volunteer contribution remains very important to Drupal. Maybe most importantly, while the number of volunteers and sponsors has grown year over year in absolute terms, sponsored contributions appear to be growing faster than volunteer contributions. This is consistent with how open source projects grow and scale.

Who is sponsoring the work?

Now that we have established that most of the work on Drupal is sponsored, we want to study which organizations contribute to Drupal. While 889 different organizations contributed to Drupal, approximately 50% of them received four credits or fewer. The top 30 organizations (roughly the top 3%) account for about 48% of the total credits, which implies that the top 30 companies play a crucial role in the health of the Drupal project. The graph below shows the top 30 organizations and the number of credits they received between July 1, 2016 and June 30, 2017:

Top 30 organizations contributing to DrupalThe top 30 contributing organizations based on the number of Drupal.org commit credits.

While not immediately obvious from the graph above, different types of companies are active in Drupal's ecosystem:

Category Description
Traditional Drupal businesses Small-to-medium-sized professional services companies that make money primarily using Drupal. They typically employ fewer than 100 employees, and because they specialize in Drupal, many of these professional services companies contribute frequently and are a huge part of our community. Examples are Chapter Three (shown on graph) and Lullabot (shown on graph).
Digital marketing agencies Larger full-service agencies that have marketing-led practices using a variety of tools, typically including Drupal, Adobe Experience Manager, Sitecore, WordPress, etc. They tend to be larger, with the larger agencies employing thousands of people. Examples are Wunderman and Mirum.
System integrators Larger companies that specialize in bringing together different technologies into one solution. Example system agencies are Accenture, TATA Consultancy Services, Capgemini and CI&T.
Technology and infrastructure companies Examples are Acquia (shown on graph), Lingotek, BlackMesh, Rackspace, Pantheon and Platform.sh.
End-users Examples are Pfizer (shown on graph) or NBCUniversal.

A few observations:

  • Almost all of the sponsors in the top 30 are traditional Drupal businesses. Companies like MD Systems (12 employees), Valuebound (34 employees), Chapter Three (27 employees), Commerce Guys (7 employees) and PreviousNext (20 employees) are, despite their size, critical to Drupal's success.

    It's worth highlighting MD Systems, which ranks second in the list of the top 30 contributing organizations, and is the number-one contributor among traditional Drupal businesses. What distinguishes MD Systems from most others is that it has embedded contribution into its corporate philosophy. For every commercial project, MD Systems invests 20% of that project's value back into Drupal. They believe that using commercial projects as the foundation for community contribution leads to more meaningful and healthier contributions for Drupal and a lower total cost of ownership for their customers. This is different from other organizations, where employees are allotted a number of hours per month to contribute outside of customer-facing projects. There is no denying that MD Systems has had a tremendous impact on the Drupal community with contributions that are both frequent and impactful.

  • Compared to these traditional Drupal businesses, Acquia has nearly 800 employees and several full-time Drupal contributors. Acquia's Office of the CTO (OCTO) works to resolve some of the most complex issues on Drupal.org, many of which are not recognized by the credit system (e.g. release management, communication, sprint organizing, and project coordination). However, I believe that Acquia should contribute even more due to our comparative size.
  • No digital marketing agencies show up in the top 30, though some of them are starting to contribute. It is exciting that an increasing number of digital marketing agencies are delivering beautiful experiences using Drupal. As a community, we need to work to ensure that each of these firms are contributing back to the project with the same commitment that we see from firms like Chapter Three, MD Systems or CI&T.
  • The only system integrator in the top 30 is CI&T, which ranked 6th with 664 credits. As far as system integrators are concerned, CI&T is a smaller player with approximately 2,500 employees. However, we do see various system integrators outside of the top 30, including Globant, Capgemini, Sapient and TATA Consultancy Services. Each of these system integrators reported 30 to 70 credits in the past year. Finally, Wipro began contributing this year with 2 credits. We expect, or hope, to see system integrators contribute more and more, especially given the number of Drupal developers they employ. Many have sizable Drupal practices with hundreds of Drupal developers, yet contributing to open source is relatively new and often not well-understood.
  • Infrastructure and software companies play an important role in our community, yet only Acquia appears in the top 30. While Acquia has a professional services division, 75% of the contributions come from the product organization (including the Office of the CTO and the Acquia Lightning team). Other contributing infrastructure companies include Pantheon and Platform.sh, which are both venture-backed platform-as-a-service companies that originated from the Drupal community. Pantheon has 17 credits and Platform.sh has 47 credits. Amazee Labs, who is building an infrastructure business, reported 51 credits. Rackspace is a public company hosting thousands of Drupal sites; they have 48 credits. Lingotek offers cloud-based translation management software and has 94 credits.
  • We saw two end-users in the top 30 corporate sponsors: Pfizer (251 credits, up from 158 credits the year before) and the German company bio.logis (212 credits). Other notable customers outside of the top 30 were Workday, Wolters Kluwer, Burda Media, University of Colorado Boulder, YMCA and OpenY, CARD.com and NBCUniversal.
Contributions by technology companiesSponsored code contributions to Drupal.org from technology and infrastructure companies. The chart does not reflect sponsored code contributions on GitHub, Drupal event sponsorship, and the many forms of value that these companies add to Drupal and other open-source communities.

We can conclude that technology and infrastructure companies, digital marketing agencies, system integrators and end-users are not meaningfully contributing code to Drupal.org today. How can we explain this disparity in comparison to traditional Drupal businesses who contribute the most? We believe the biggest reasons are:

  1. Drupal's strategic importance. A variety of the traditional Drupal agencies have been involved with Drupal for 10 years and almost entirely depend on Drupal to support their business. Given both their expertise and dependence on Drupal, they are most likely to look after Drupal's development and well-being. These organizations are typically recognized as Drupal experts and are sought out by organizations that want to build a Drupal website. Contrast this with most of the digital marketing agencies and system integrators who are sized to work with a diversified portfolio of content management platforms and who are historically only getting started with Drupal and open source. They deliver digital marketing solutions and aren't necessarily sought out for their Drupal expertise. As their Drupal practices grow in size and importance, this could change. In fact, contributing to Drupal can help grow their Drupal business because it helps their name stand out as Drupal experts and gives them a competitive edge with their customers.
  2. The level of experience with Drupal and open source. Drupal aside, many organizations have little or no experience with open source, so it is important that we motivate and teach them to contribute.
  3. Legal reservations. We recognize that some organizations are not legally permitted to contribute, let alone attribute their customers. We hope that will change as open source continues to get adopted.
  4. Tools and process barriers. Drupal contribution still involves a patch-based workflow on Drupal.org's unique issue queue system. This presents a fairly steep learning curve to most developers, who primarily work with more modern and common tools such as GitHub. Getting the code change proposal uploaded is just the first step; getting code changes accepted into an upstream Drupal project — especially Drupal core — is hard work. Peer reviews, gates such as automated testing and documentation, required sign-offs from maintainers and committers, knowledge of best practices and other community norms are a few of the challenges a contributor must face to get code accepted into Drupal.

Consequently, this data shows that the Drupal community can do more to entice companies to contribute code to Drupal.org. The Drupal community has a long tradition of encouraging organizations to share code rather than keep it behind firewalls. While the spirit of the Drupal project cannot be reduced to any single ideology — not every organization can or will share their code — we would like to see organizations continue to prioritize collaboration over individual ownership. Our aim is not to criticize those who do not contribute, but rather to help foster an environment worthy of contribution. Given the vast amount of Drupal users, we believe continuing to encourage organizations and end-users to contribute could be a big opportunity.

There are substantial benefits and business drivers for organizations that contribute: (1) it improves their ability to sell and win deals and (2) it improves their ability to hire. Companies that contribute to Drupal tend to promote their contributions in RFPs and sales pitches. Contributing to Drupal also results in being recognized as a great place to work for Drupal experts.

The uneasy alliance with corporate contributions

As mentioned above, when community-driven open-source projects grow, there is a bigger need for organizations to help drive their development. It almost always creates an uneasy alliance between volunteers and corporations.

This theory played out in the Linux community well before it played out in the Drupal community. The Linux project is 25 years old and has seen a steady increase in the number of corporate contributors for roughly 20 years. While Linux companies like Red Hat and SUSE rank high on the contribution list, so do non-Linux-centric companies such as Samsung, Intel, Oracle and Google. All of these corporate contributors are (or were) using Linux as an integral part of their business.

The 889 organizations that contribute to Drupal (which includes corporations) is more than four times the number of organizations that sponsor development of the Linux kernel. This is significant because Linux is considered "one of the largest cooperative software projects ever attempted". In fairness, Linux has a different ecosystem than Drupal. The Linux business ecosystem has various large organizations (Red Hat, Google, Intel, IBM and SUSE) for whom Linux is very strategic. As a result, many of them employ dozens of full-time Linux contributors and invest millions of dollars in Linux each year.

What projects have sponsors?

In total, the Drupal community worked on 3,183 different projects (modules, themes and distributions) in the 12-month period between July 1, 2016 to June 30, 2017. To understand where the organizations sponsoring Drupal put their money, I've listed the top 20 most sponsored projects:

RankProject nameIssues
1Drupal core4745
2Drupal Commerce (distribution)526
3Webform361
4Open Y (distribution)324
5Paragraphs231
6Inmail223
7User guide218
8JSON API204
9Paragraphs collection200
10Entity browser196
11Diff190
12Group170
13Metatag157
14Facets155
15Commerce Point of Sale (PoS)147
16Search API143
17Open Social (distribution)133
18Drupal voor Gemeenten (distribution)131
19Solr Search122
20Geolocation field118

Who is sponsoring the top 30 contributors?

Rank Username Issues Volunteer Sponsored Not specified Sponsors
1 jrockowitz 537 88% 45% 9% The Big Blue House (239), Kennesaw State University (6), Memorial Sloan Kettering Cancer Center (4)
2 dawehner 421 67% 83% 5% Chapter Three (328), Tag1 Consulting (19), Drupal Association (12), Acquia (5), Comm-press (1)
3 RenatoG 408 0% 100% 0% CI&T (408)
4 bojanz 351 0% 95% 5% Commerce Guys (335), Adapt A/S (38), Bluespark (2)
5 Berdir 335 0% 93% 7% MD Systems (310), Acquia (7)
6 mglaman 334 3% 97% 1% Commerce Guys (319), Thinkbean, LLC (48), LivePerson, Inc (46), Bluespark (22), Universal Music Group (16), Gaggle.net, Inc. (3), Bluehorn Digital (1)
7 Wim Leers 332 14% 87% 2% Acquia (290)
8 alexpott 329 7% 99% 1% Chapter Three (326), TES Global (1)
9 DamienMcKenna 245 2% 95% 4% Mediacurrent (232)
10 jhodgdon 242 0% 1% 99% Drupal Association (2), Poplar ProductivityWare (2)
11 drunken monkey 238 95% 11% 1% Acquia (17), Vizala (8), Wunder Group (1), Sunlime IT Services GmbH (1)
12 naveenvalecha 196 74% 55% 1% Acquia (152), Google Summer of Code (7), QED42 (1)
13 Munavijayalakshmi 192 0% 100% 0% Valuebound (192)
14 borisson_ 191 66% 39% 22% Dazzle (70), Acquia (6)
15 yongt9412 189 0% 97% 3% MD Systems (183), Acquia (6)
16 klausi 185 9% 61% 32% epiqo (112)
17 Sam152 184 59% 92% 7% PreviousNext (168), amaysim Australia Ltd. (5), Code Drop (2)
18 miro_dietiker 182 0% 99% 1% MD Systems (181)
19 Pavan B S 180 0% 98% 2% Valuebound (177)
20 ajay_reddy 176 100% 99% 0% Valuebound (180), Drupal Bangalore Community (154)
21 phenaproxima 172 0% 99% 1% Acquia (170)
22 sanchiz 162 0% 99% 1% Drupal Ukraine Community (107), Vinzon (101), FFW (60), Open Y (52)
23 slashrsm 161 6% 95% 3% MD Systems (153), Acquia (47)
24 jhedstrom 155 4% 92% 4% Phase2 (143), Workday, Inc. (134), Memorial Sloan Kettering Cancer Center (1)
25 xjm 151 0% 91% 9% Acquia (137)
26 catch 147 3% 83% 16% Third and Grove (116), Tag1 Consulting (6)
27 larowlan 145 12% 92% 7% PreviousNext (133), University of Technology, Sydney (30), amaysim Australia Ltd. (6), Australian Competition and Consumer Commission (ACCC) (1), Department of Justice & Regulation, Victoria (1)
28 rakesh.gectcr 141 100% 91% 0% Valuebound (128)
29 benjy 139 0% 94% 6% PreviousNext (129), Brisbane City Council (8), Code Drop (1)
30 dhruveshdtripathi 138 15% 100% 0% DevsAdda (138), OpenSense Labs (44)

We observe that the top 30 contributors are sponsored by 46 organizations. This kind of diversity is aligned with our desire not to see Drupal controlled by a single organization. These top contributors and organizations are from many different parts of the world and work with customers large and small. Nonetheless, we will continue to benefit from more diversity.

Evolving the credit system

Like Drupal itself, the credit system on Drupal.org is an evolving tool. Ultimately, the credit system will only be useful when the community uses it, understands its shortcomings, and suggests constructive improvements. In highlighting the organizations that sponsor the development of code on Drupal.org, we hope to elicit responses that help evolve the credit system into something that incentivizes business to sponsor more work and enables more people to participate in our community, learn from others, teach newcomers and make positive contributions. Drupal is a positive force for change and we wish to use the credit system to highlight (at least some of) the work of our diverse community, which includes volunteers, companies, nonprofits, governments, schools, universities, individuals, and other groups.

One of the challenges with the existing credit system is it has no way of "weighting" contributions. A typo fix counts just as much as giving multiple detailed technical reviews on a critical core issue. This appears to have the effect of incentivizing organizations' employees to work on "lower-hanging fruit issues", because this bumps their companies' names in the rankings. One way to help address this might be to adjust the credit ranking algorithm to consider things such as issue priority, patch size, and so on. This could help incentivize companies to work on larger and more important problems and save coding standards improvements for new contributor sprints. Implementing a scoring system that ranks the complexity of an issue would also allow us to develop more accurate reports of contributed work.

Conclusion

Our data confirms Drupal is a vibrant community full of contributors who are constantly evolving and improving the software. While we have amazing geographic diversity, we need greater gender diversity. Our analysis of the Drupal.org credit data concludes that most contributions to Drupal are sponsored. At the same time, the data shows that volunteer contribution remains very important to Drupal.

As a community, we need to understand that a healthy open-source ecosystem includes more than traditional Drupal businesses that contribute the most. For example, we don't see a lot of contribution from the larger digital marketing agencies, system integrators, technology companies, or end-users of Drupal — we believe that might come as these organizations build out their Drupal practices and Drupal becomes more strategic for them.

To grow and sustain Drupal, we should support those that contribute to Drupal and find ways to get those that are not contributing involved in our community. We invite you to help us continue to strengthen our ecosystem.

Special thanks to Tim Lehnen and Neil Drumm from the Drupal Association for providing us with the Drupal.org credit system data and for supporting us during our research. I would also like to extend a special thanks to Matthew Tift for helping to lay the foundation for this research, collaborating on last year's blog post, and for reviewing this year's edition. Finally, thanks to Angie Byron, Gábor Hojtsy, Jess (xjm), Preston So, Ted Bowman, Wim Leers and Gigi Anderson for providing feedback during the writing process.

Last year, Matthew Tift and I examined Drupal.org's commit data to understand who develops Drupal, how much of that work is sponsored, and where that sponsorship comes from. We published our analysis in a blog post called "Who Sponsors Drupal Development?". A year later, I wanted to present an update. This year's report will also cover additional data, including gender and geographical diversity, and project sponsorship.

Understanding how an open-source project works is important because it establishes a benchmark for project health and scalability. Scaling an open-source project is a difficult task. As an open-source project's rate of adoption grows, the number of people that benefit from the project also increases. Often the open-source project also becomes more complex as it expands, which means that the economic reward of helping to improve the project decreases.

A recent article on the Bitcoin and Ethereum contributor communities illustrates this disparity perfectly. Ethereum and Bitcoin have market capitalizations valued at $30 billion and $70 billion, respectively. However, both projects have fewer than 40 meaningful contributors, and contribution isn't growing despite the rising popularity of cryptocurrency.

Number of Bitcoin contributors between 2010 and 2017According to Bitcoin's GitHub data, Bitcoin has less than 40 active contributors.
Number of Ethereum contributors between 2014 and 2017According to Ethereum's GitHub data, Ethereum has less than 20 active contributors.

Drupal, by comparison, has a diverse community of contributors. In the 12-month period between July 1, 2016 to June 30, 2017 we saw code contributions on Drupal.org from 7,240 different individuals and 889 different companies. This does not mean that Drupal is exempt from the challenges of scaling an open-source project. We hope that this report provides transparency about Drupal project development and encourages more individuals and organizations incentive to contribute. We also will highlight areas where our community can and should do better.

What is the Drupal.org credit system?

In the spring of 2015, after proposing ideas for giving credit and discussing various approaches at length, Drupal.org added the ability for people to attribute their work to an organization or customer in the Drupal.org issue queues. Maintainers of Drupal modules, themes and distributions can award issues credits to people who help resolve issues with code, translations, documentation, design and more.

Example issue credit on drupal orgA screenshot of an issue comment on Drupal.org. You can see that jamadar worked on this patch as a volunteer, but also as part of his day job working for TATA Consultancy Services on behalf of their customer, Pfizer.

Credits are a powerful motivator for both individuals and organizations. Accumulating credits provides individuals with a way to showcase their expertise. Organizations can utilize credits to help recruit developers or to increase their visibility in the Drupal.org marketplace.

While the benefits are evident, it is important to note a few of the limitations in Drupal.org's current credit system:

  • Contributing to issues on Drupal.org is not the only way to contribute. Other activities, such as sponsoring events, promoting Drupal, and providing help and mentorship, are important to the long-term health of the Drupal project. Many of these activities are not currently captured by the credit system. For this post, we chose to only look at code contributions.
  • We acknowledge that parts of Drupal are developed on GitHub and therefore aren't fully credited on Drupal.org. The actual number of contributions and contributors could be significantly higher than what we report.
  • Even when development is done on Drupal.org, the credit system is not used consistently; because using the credit system is optional, a lot of code committed on Drupal.org has no or incomplete contribution credits.
  • Not all code credits are the same. We currently don't have a way to account for the complexity and quality of contributions; one person might have worked several weeks for just one credit, while another person might receive a credit for ten minutes of work. In the future, we should consider issuing credit data in conjunction with issue priority, patch size, etc. We can also reduce the need for trivial credits by automating patch rerolls and automating coding style fixes.

Who is working on Drupal?

For our analysis we looked at all the issues that were marked "closed" or "fixed" in the 12-month period from July 1, 2016 to June 30, 2017. What we learned is that there were 23,238 issues marked "closed" or "fixed", a 22% increase from the 19,095 issues in the 2015-2016 period. Those 23,238 issues had 42,449 issue credits, a 30% increase from the 32,711 issue credits recorded in the previous year. Issue credits against Drupal core remained roughly the same year over year, meaning almost all of this growth came from increased activity in contributed projects. This is no surprise. Drupal development is cyclical, and during this period of the Drupal 8 development cycle, most of the Drupal community has been focused on porting modules from Drupal 7 to Drupal 8. Of the 42,449 issue credits reported this year, 20% (8,619 credits) were for Drupal core, while 80% (33,830 credits) went to contributed themes, modules and distributions.

Compared to the previous year, we also saw an increase in both the number of people contributing and the number of organizations contributing. Drupal.org received code contributions from 7,240 different individuals and 889 different organizations.

Contributions by individuals vs organizationsThe number of individual contributors is up 28% year over year and the number of organizations contributing is up 26% year over year.

While the number of individual contributors rose, a relatively small number of individuals still do the majority of the work. Approximately 47% of individual contributors received just one credit. Meanwhile, the top 30 contributors (the top 0.4%) account for over 17% of the total credits, indicating that these individuals put an incredible amount of time and effort in developing Drupal and its contributed projects:

RankUsernameIssues
1jrockowitz537
2dawehner421
3RenatoG408
4bojanz351
5Berdir335
6mglaman334
7Wim Leers332
8alexpott329
9DamienMcKenna245
10jhodgdon242
11drunken monkey238
12naveenvalecha196
13Munavijayalakshmi192
14borisson_191
15yongt9412189
16klausi185
17Sam152184
18miro_dietiker182
19Pavan B S180
20ajay_reddy176
21phenaproxima172
22sanchiz162
23slashrsm161
24jhedstrom155
25xjm151
26catch147
27larowlan145
28rakesh.gectcr141
29benjy139
30dhruveshdtripathi138

Out of the top 30 contributors featured, 19 were also recognized as top contributors in our 2015-2016 report. These Drupalists' dedication and continued contribution to the project has been crucial to Drupal's development. It's also exciting to see 11 new names on the list. This mobility is a testament to the community's evolution and growth.

Next, we looked at both the gender and geographic diversity of Drupal.org code contributors. While these are only two examples of diversity, this is the only available data that contributors can choose to share on their Drupal.org profiles. The reported data shows that only 6% of the recorded contributions were made by contributors that identify as female, which indicates a steep gender gap. Like in most open-source projects, the gender imbalance in Drupal is profound and underscores the need to continue fostering diversity and inclusion in our community.

Contributions by genderThe gender representation behind the issue credits. Only 6% of the recorded contributions are by women.
When measuring geographic diversity, we saw individual contributors from 6 different continents and 116 different countries:
Contributions by continent
Contributions by countryThe top 20 countries from which contributions originate. The data is compiled by aggregating the countries of all individual contributors behind each commit. Note that the geographical location of contributors doesn't always correspond with the origin of their sponsorship. Wim Leers, for example, works from Belgium, but his funding comes from Acquia, which has the majority of its customers in North America.

How much of the work is sponsored?

Drupal is used by more than one million websites. The vast majority of the individuals and organizations behind these Drupal websites never participate in the development of the project. They might use the software as it is or might not feel the need to help drive its development. We have to provide more incentive for these individuals and organizations to contribute back to the project.

Issue credits can be marked as "volunteer" and "sponsored" simultaneously (shown in jamadar's screenshot near the top of this post). This could be the case when a contributor does the minimum required work to satisfy the customer's need, in addition to using their spare time to add extra functionality.

While Drupal started out as a 100% volunteer-driven project, today the majority of the code on Drupal.org is sponsored by organizations. Only 11% of the commit credits that we examined in 2016-2017 were "purely volunteer" credits (4,498 credits), in stark contrast to the 46% that were "purely sponsored". In other words, there were four times as many "purely sponsored" credits as "purely volunteer" credits.

A few comparisons with the 2015-2016 data:

  • The credit system is used more. Between July 1, 2015 and June 30, 2016, 37% of all credits had no attribution while in the more recent period between July 1, 2016 to June 30, 2017, only 28% of credits lacked attribution. More people have become aware of the credit system, the attribution options, and their benefits.
  • Sponsored credits are growing faster than volunteer credits. Both "purely volunteer" and "purely sponsored" credits grew, but "purely sponsored" credits grew faster. There are two reasons why this could be the case: (1) more contributions are sponsored and (2) organizations are more likely to use the credit system compared to volunteers.
Contributions by volunteer vs sponsored

No data is perfect, but it feels safe to conclude that most of the work on Drupal is sponsored. At the same time, the data shows that volunteer contribution remains very important to Drupal. Maybe most importantly, while the number of volunteers and sponsors has grown year over year in absolute terms, sponsored contributions appear to be growing faster than volunteer contributions. This is consistent with how open source projects grow and scale.

Who is sponsoring the work?

Now that we have established that most of the work on Drupal is sponsored, we want to study which organizations contribute to Drupal. While 889 different organizations contributed to Drupal, approximately 50% of them received four credits or fewer. The top 30 organizations (roughly the top 3%) account for about 48% of the total credits, which implies that the top 30 companies play a crucial role in the health of the Drupal project. The graph below shows the top 30 organizations and the number of credits they received between July 1, 2016 and June 30, 2017:

Top 30 organizations contributing to DrupalThe top 30 contributing organizations based on the number of Drupal.org commit credits.

While not immediately obvious from the graph above, different types of companies are active in Drupal's ecosystem:

Category Description
Traditional Drupal businesses Small-to-medium-sized professional services companies that make money primarily using Drupal. They typically employ fewer than 100 employees, and because they specialize in Drupal, many of these professional services companies contribute frequently and are a huge part of our community. Examples are Chapter Three (shown on graph) and Lullabot (shown on graph).
Digital marketing agencies Larger full-service agencies that have marketing-led practices using a variety of tools, typically including Drupal, Adobe Experience Manager, Sitecore, WordPress, etc. They tend to be larger, with the larger agencies employing thousands of people. Examples are Wunderman and Mirum.
System integrators Larger companies that specialize in bringing together different technologies into one solution. Example system agencies are Accenture, TATA Consultancy Services, Capgemini and CI&T.
Technology and infrastructure companies Examples are Acquia (shown on graph), Lingotek, BlackMesh, Rackspace, Pantheon and Platform.sh.
End-users Examples are Pfizer (shown on graph) or NBCUniversal.

A few observations:

  • Almost all of the sponsors in the top 30 are traditional Drupal businesses. Companies like MD Systems (12 employees), Valuebound (34 employees), Chapter Three (27 employees), Commerce Guys (7 employees) and PreviousNext (20 employees) are, despite their size, critical to Drupal's success.

    It's worth highlighting MD Systems, which ranks second in the list of the top 30 contributing organizations, and is the number-one contributor among traditional Drupal businesses. What distinguishes MD Systems from most others is that it has embedded contribution into its corporate philosophy. For every commercial project, MD Systems invests 20% of that project's value back into Drupal. They believe that using commercial projects as the foundation for community contribution leads to more meaningful and healthier contributions for Drupal and a lower total cost of ownership for their customers. This is different from other organizations, where employees are allotted a number of hours per month to contribute outside of customer-facing projects. There is no denying that MD Systems has had a tremendous impact on the Drupal community with contributions that are both frequent and impactful.

  • Compared to these traditional Drupal businesses, Acquia has nearly 800 employees and several full-time Drupal contributors. Acquia's Office of the CTO (OCTO) works to resolve some of the most complex issues on Drupal.org, many of which are not recognized by the credit system (e.g. release management, communication, sprint organizing, and project coordination). However, I believe that Acquia should contribute even more due to our comparative size.
  • No digital marketing agencies show up in the top 30, though some of them are starting to contribute. It is exciting that an increasing number of digital marketing agencies are delivering beautiful experiences using Drupal. As a community, we need to work to ensure that each of these firms are contributing back to the project with the same commitment that we see from firms like Chapter Three, MD Systems or CI&T.
  • The only system integrator in the top 30 is CI&T, which ranked 6th with 664 credits. As far as system integrators are concerned, CI&T is a smaller player with approximately 2,500 employees. However, we do see various system integrators outside of the top 30, including Globant, Capgemini, Sapient and TATA Consultancy Services. Each of these system integrators reported 30 to 70 credits in the past year. Finally, Wipro began contributing this year with 2 credits. We expect, or hope, to see system integrators contribute more and more, especially given the number of Drupal developers they employ. Many have sizable Drupal practices with hundreds of Drupal developers, yet contributing to open source is relatively new and often not well-understood.
  • Infrastructure and software companies play an important role in our community, yet only Acquia appears in the top 30. While Acquia has a professional services division, 75% of the contributions come from the product organization (including the Office of the CTO and the Acquia Lightning team). Other contributing infrastructure companies include Pantheon and Platform.sh, which are both venture-backed platform-as-a-service companies that originated from the Drupal community. Pantheon has 17 credits and Platform.sh has 47 credits. Amazee Labs, who is building an infrastructure business, reported 51 credits. Rackspace is a public company hosting thousands of Drupal sites; they have 48 credits. Lingotek offers cloud-based translation management software and has 94 credits.
  • We saw two end-users in the top 30 corporate sponsors: Pfizer (251 credits, up from 158 credits the year before) and the German company bio.logis (212 credits). Other notable customers outside of the top 30 were Workday, Wolters Kluwer, Burda Media, University of Colorado Boulder, YMCA and OpenY, CARD.com and NBCUniversal.
Contributions by technology companiesSponsored code contributions to Drupal.org from technology and infrastructure companies. The chart does not reflect sponsored code contributions on GitHub, Drupal event sponsorship, and the many forms of value that these companies add to Drupal and other open-source communities.

We can conclude that technology and infrastructure companies, digital marketing agencies, system integrators and end-users are not meaningfully contributing code to Drupal.org today. How can we explain this disparity in comparison to traditional Drupal businesses who contribute the most? We believe the biggest reasons are:

  1. Drupal's strategic importance. A variety of the traditional Drupal agencies have been involved with Drupal for 10 years and almost entirely depend on Drupal to support their business. Given both their expertise and dependence on Drupal, they are most likely to look after Drupal's development and well-being. These organizations are typically recognized as Drupal experts and are sought out by organizations that want to build a Drupal website. Contrast this with most of the digital marketing agencies and system integrators who are sized to work with a diversified portfolio of content management platforms and who are historically only getting started with Drupal and open source. They deliver digital marketing solutions and aren't necessarily sought out for their Drupal expertise. As their Drupal practices grow in size and importance, this could change. In fact, contributing to Drupal can help grow their Drupal business because it helps their name stand out as Drupal experts and gives them a competitive edge with their customers.
  2. The level of experience with Drupal and open source. Drupal aside, many organizations have little or no experience with open source, so it is important that we motivate and teach them to contribute.
  3. Legal reservations. We recognize that some organizations are not legally permitted to contribute, let alone attribute their customers. We hope that will change as open source continues to get adopted.
  4. Tools and process barriers. Drupal contribution still involves a patch-based workflow on Drupal.org's unique issue queue system. This presents a fairly steep learning curve to most developers, who primarily work with more modern and common tools such as GitHub. Getting the code change proposal uploaded is just the first step; getting code changes accepted into an upstream Drupal project — especially Drupal core — is hard work. Peer reviews, gates such as automated testing and documentation, required sign-offs from maintainers and committers, knowledge of best practices and other community norms are a few of the challenges a contributor must face to get code accepted into Drupal.

Consequently, this data shows that the Drupal community can do more to entice companies to contribute code to Drupal.org. The Drupal community has a long tradition of encouraging organizations to share code rather than keep it behind firewalls. While the spirit of the Drupal project cannot be reduced to any single ideology — not every organization can or will share their code — we would like to see organizations continue to prioritize collaboration over individual ownership. Our aim is not to criticize those who do not contribute, but rather to help foster an environment worthy of contribution. Given the vast amount of Drupal users, we believe continuing to encourage organizations and end-users to contribute could be a big opportunity.

There are substantial benefits and business drivers for organizations that contribute: (1) it improves their ability to sell and win deals and (2) it improves their ability to hire. Companies that contribute to Drupal tend to promote their contributions in RFPs and sales pitches. Contributing to Drupal also results in being recognized as a great place to work for Drupal experts.

The uneasy alliance with corporate contributions

As mentioned above, when community-driven open-source projects grow, there is a bigger need for organizations to help drive their development. It almost always creates an uneasy alliance between volunteers and corporations.

This theory played out in the Linux community well before it played out in the Drupal community. The Linux project is 25 years old and has seen a steady increase in the number of corporate contributors for roughly 20 years. While Linux companies like Red Hat and SUSE rank high on the contribution list, so do non-Linux-centric companies such as Samsung, Intel, Oracle and Google. All of these corporate contributors are (or were) using Linux as an integral part of their business.

The 889 organizations that contribute to Drupal (which includes corporations) is more than four times the number of organizations that sponsor development of the Linux kernel. This is significant because Linux is considered "one of the largest cooperative software projects ever attempted". In fairness, Linux has a different ecosystem than Drupal. The Linux business ecosystem has various large organizations (Red Hat, Google, Intel, IBM and SUSE) for whom Linux is very strategic. As a result, many of them employ dozens of full-time Linux contributors and invest millions of dollars in Linux each year.

What projects have sponsors?

In total, the Drupal community worked on 3,183 different projects (modules, themes and distributions) in the 12-month period between July 1, 2016 to June 30, 2017. To understand where the organizations sponsoring Drupal put their money, I've listed the top 20 most sponsored projects:

RankProject nameIssues
1Drupal core4745
2Drupal Commerce (distribution)526
3Webform361
4Open Y (distribution)324
5Paragraphs231
6Inmail223
7User guide218
8JSON API204
9Paragraphs collection200
10Entity browser196
11Diff190
12Group170
13Metatag157
14Facets155
15Commerce Point of Sale (PoS)147
16Search API143
17Open Social (distribution)133
18Drupal voor Gemeenten (distribution)131
19Solr Search122
20Geolocation field118

Who is sponsoring the top 30 contributors?

Rank Username Issues Volunteer Sponsored Not specified Sponsors
1 jrockowitz 537 88% 45% 9% The Big Blue House (239), Kennesaw State University (6), Memorial Sloan Kettering Cancer Center (4)
2 dawehner 421 67% 83% 5% Chapter Three (328), Tag1 Consulting (19), Drupal Association (12), Acquia (5), Comm-press (1)
3 RenatoG 408 0% 100% 0% CI&T (408)
4 bojanz 351 0% 95% 5% Commerce Guys (335), Adapt A/S (38), Bluespark (2)
5 Berdir 335 0% 93% 7% MD Systems (310), Acquia (7)
6 mglaman 334 3% 97% 1% Commerce Guys (319), Thinkbean, LLC (48), LivePerson, Inc (46), Bluespark (22), Universal Music Group (16), Gaggle.net, Inc. (3), Bluehorn Digital (1)
7 Wim Leers 332 14% 87% 2% Acquia (290)
8 alexpott 329 7% 99% 1% Chapter Three (326), TES Global (1)
9 DamienMcKenna 245 2% 95% 4% Mediacurrent (232)
10 jhodgdon 242 0% 1% 99% Drupal Association (2), Poplar ProductivityWare (2)
11 drunken monkey 238 95% 11% 1% Acquia (17), Vizala (8), Wunder Group (1), Sunlime IT Services GmbH (1)
12 naveenvalecha 196 74% 55% 1% Acquia (152), Google Summer of Code (7), QED42 (1)
13 Munavijayalakshmi 192 0% 100% 0% Valuebound (192)
14 borisson_ 191 66% 39% 22% Dazzle (70), Acquia (6)
15 yongt9412 189 0% 97% 3% MD Systems (183), Acquia (6)
16 klausi 185 9% 61% 32% epiqo (112)
17 Sam152 184 59% 92% 7% PreviousNext (168), amaysim Australia Ltd. (5), Code Drop (2)
18 miro_dietiker 182 0% 99% 1% MD Systems (181)
19 Pavan B S 180 0% 98% 2% Valuebound (177)
20 ajay_reddy 176 100% 99% 0% Valuebound (180), Drupal Bangalore Community (154)
21 phenaproxima 172 0% 99% 1% Acquia (170)
22 sanchiz 162 0% 99% 1% Drupal Ukraine Community (107), Vinzon (101), FFW (60), Open Y (52)
23 slashrsm 161 6% 95% 3% MD Systems (153), Acquia (47)
24 jhedstrom 155 4% 92% 4% Phase2 (143), Workday, Inc. (134), Memorial Sloan Kettering Cancer Center (1)
25 xjm 151 0% 91% 9% Acquia (137)
26 catch 147 3% 83% 16% Third and Grove (116), Tag1 Consulting (6)
27 larowlan 145 12% 92% 7% PreviousNext (133), University of Technology, Sydney (30), amaysim Australia Ltd. (6), Australian Competition and Consumer Commission (ACCC) (1), Department of Justice & Regulation, Victoria (1)
28 rakesh.gectcr 141 100% 91% 0% Valuebound (128)
29 benjy 139 0% 94% 6% PreviousNext (129), Brisbane City Council (8), Code Drop (1)
30 dhruveshdtripathi 138 15% 100% 0% DevsAdda (138), OpenSense Labs (44)

We observe that the top 30 contributors are sponsored by 46 organizations. This kind of diversity is aligned with our desire not to see Drupal controlled by a single organization. These top contributors and organizations are from many different parts of the world and work with customers large and small. Nonetheless, we will continue to benefit from more diversity.

Evolving the credit system

Like Drupal itself, the credit system on Drupal.org is an evolving tool. Ultimately, the credit system will only be useful when the community uses it, understands its shortcomings, and suggests constructive improvements. In highlighting the organizations that sponsor the development of code on Drupal.org, we hope to elicit responses that help evolve the credit system into something that incentivizes business to sponsor more work and enables more people to participate in our community, learn from others, teach newcomers and make positive contributions. Drupal is a positive force for change and we wish to use the credit system to highlight (at least some of) the work of our diverse community, which includes volunteers, companies, nonprofits, governments, schools, universities, individuals, and other groups.

One of the challenges with the existing credit system is it has no way of "weighting" contributions. A typo fix counts just as much as giving multiple detailed technical reviews on a critical core issue. This appears to have the effect of incentivizing organizations' employees to work on "lower-hanging fruit issues", because this bumps their companies' names in the rankings. One way to help address this might be to adjust the credit ranking algorithm to consider things such as issue priority, patch size, and so on. This could help incentivize companies to work on larger and more important problems and save coding standards improvements for new contributor sprints. Implementing a scoring system that ranks the complexity of an issue would also allow us to develop more accurate reports of contributed work.

Conclusion

Our data confirms Drupal is a vibrant community full of contributors who are constantly evolving and improving the software. While we have amazing geographic diversity, we need greater gender diversity. Our analysis of the Drupal.org credit data concludes that most contributions to Drupal are sponsored. At the same time, the data shows that volunteer contribution remains very important to Drupal.

As a community, we need to understand that a healthy open-source ecosystem includes more than traditional Drupal businesses that contribute the most. For example, we don't see a lot of contribution from the larger digital marketing agencies, system integrators, technology companies, or end-users of Drupal — we believe that might come as these organizations build out their Drupal practices and Drupal becomes more strategic for them.

To grow and sustain Drupal, we should support those that contribute to Drupal and find ways to get those that are not contributing involved in our community. We invite you to help us continue to strengthen our ecosystem.

Special thanks to Tim Lehnen and Neil Drumm from the Drupal Association for providing us with the Drupal.org credit system data and for supporting us during our research. I would also like to extend a special thanks to Matthew Tift for helping to lay the foundation for this research, collaborating on last year's blog post, and for reviewing this year's edition. Finally, thanks to Angie Byron, Gábor Hojtsy, Jess (xjm), Preston So, Ted Bowman, Wim Leers and Gigi Anderson for providing feedback during the writing process.

September 11, 2017

In order to further secure access to my workstation, after the switch to Gentoo sources, I now enabled two-factor authentication through my Yubico U2F USB device. Well, at least for local access - remote access through SSH requires both userid/password as well as the correct SSH key, by chaining authentication methods in OpenSSH.

Enabling U2F on (Gentoo) Linux is fairly easy. The various guides online which talk about the pam_u2f setup are indeed correct that it is fairly simple. For completeness sake, I've documented what I know on the Gentoo Wiki, as the pam_u2f article.

September 10, 2017

The post Coming soon: Oh Dear! – Monitoring for the encrypted web appeared first on ma.ttias.be.

I'm excited to announce a new project I'm working on: Oh Dear!

The goal of Oh Dear! is to provide modern monitoring & feedback for sites that run on HTTPS. With Chrome's soon-to-be-released version that marks any input on non-HTTPS pages as "Not Secure", that target audience is huge.

The baseline below I think sums it up very accurately.

Many users only look at their certificate expiration dates when running HTTPS sites and -- hopefully -- renew in time. But that's only a small part of the journey to HTTPS. I've ranted about messing up HTTPS often enough that I don't want to repeat myself anymore.

What does Oh Dear! offer?

From my old rant, the summary from way-back-then still stands today:

Browsers don't care if your HTTPS config is 95% perfect. They'll destroy the visitor's experience if you don't nail it for the full 100%.

There's many things that can go wrong with deploying HTTPS, including;

  • Expired certificates
  • Revoked certificates
  • Missing intermediate certificates in your chain
  • Mixed content on your site
  • Bad or insecure TLS ciphers used in the config
  • Incorrectly configured OCSP stapling
  • Badly pinned keys with HPKP
  • ...

Oh Dear! monitors for each and every one of those things, and more.

Included in Oh Dear! is Certificate Transparency reporting, so you can get notified whenever a new certificate is issued for one of your domains, intentional or otherwise.

Meet the team

Unlike my usual projects, this time I'm working together with smart folks to help make Oh Dear! a success.

The team consists of Dries Vints, Freek Van der Herten & me. We're all active in the Laravel community. Dries & Freek go way back, I only got to know these smart men a little over a year ago.

Join the beta

We're not open to the public yet, but there's a beta program you can subscribe to in order to get access to Oh Dear!.

If you run a website on HTTPS -- and chances are, you do -- don't let a bad certificate or configuration ruin your day. Trust us to monitor it for you and report any errors, before your visitors do.

Go check out our app at ohdearapp.com or follow us on Twitter via @OhDearApp.

The post Coming soon: Oh Dear! – Monitoring for the encrypted web appeared first on ma.ttias.be.

As I threatened to do in my previous post, I'm going to talk about PR 219251 for a bit. The bug report dates from only a few months ago, but the first report (that I can remeber) actually came from Shawn Webb on Twitter, of all places: backtrace

Despite there being a stacktrace it took quite a while (nearly 6 months in fact) before I figured this one out.

It took Reshad Patuck managing to distill the problem down to a small-ish test script to make real progress on this. His testcase meant that I could get core dumps and experiment. It also provided valuable clues because it could be tweaked to see what elements were required to trigger the panic.

This test script starts a (vnet) jail, adds an epair interface to it, sets up pf in the jail, and then reloads the pf rules on the host. Interestingly the panic does not seem to occur if that last step is not included.

Time to take a closer look at the code that breaks:

u_int32_t
pf_state_expires(const struct pf_state *state)
{
        u_int32_t       timeout;
        u_int32_t       start;
        u_int32_t       end;
        u_int32_t       states;

        /* handle all PFTM_* > PFTM_MAX here */
        if (state->timeout == PFTM_PURGE)
                return (time_uptime);
        KASSERT(state->timeout != PFTM_UNLINKED,
            ("pf_state_expires: timeout == PFTM_UNLINKED"));
        KASSERT((state->timeout < PFTM_MAX),
            ("pf_state_expires: timeout > PFTM_MAX"));
        timeout = state->rule.ptr->timeout[state->timeout];
        if (!timeout)
                timeout = V_pf_default_rule.timeout[state->timeout];
        start = state->rule.ptr->timeout[PFTM_ADAPTIVE_START];
        if (start) {
                end = state->rule.ptr->timeout[PFTM_ADAPTIVE_END];
                states = counter_u64_fetch(state->rule.ptr->states_cur);
        } else {
                start = V_pf_default_rule.timeout[PFTM_ADAPTIVE_START];
                end = V_pf_default_rule.timeout[PFTM_ADAPTIVE_END];
                states = V_pf_status.states;
        }
        if (end && states > start && start < end) {
                if (states < end)
                        return (state->expire + timeout * (end - states) /
                            (end - start));
                else
                        return (time_uptime);
        }
        return (state->expire + timeout);
}

Specifically, the following line:

	states = counter_u64_fetch(state->rule.ptr->states_cur);

We try to fetch a counter value here, but instead we dereference a bad pointer. There's two here, so already we need more information. Inspection of the core dump reveals that the state pointer is valid, and contains sane information. The rule pointer (rule.ptr) points to a sensible location, but the data is mostly 0xdeadc0de. This is the memory allocator being helpful (in debug mode) and writing garbage over freed memory, to make use-after-free bugs like this one easier to find.

In other words: the rule has been free()d while there was still a state pointing to it. Somehow we have a state (describing a connection pf knows about) which points to a rule which no longer exists.

The core dump also shows that the problem always occurs with states and rules in the default vnet (i.e. the host pf instance), not one of the pf instances in one of the vnet jails. That matches with the observation that the test script does not trigger the panic unless we also reload the rules on the host.

Great, we know what's wrong, but now we need to work out how we can get into this state. At this point we're going to have to learn something about how rules and states get cleaned up in pf. Don't worry if you had no idea, because before this bug I didn't either.

The states keep a pointer to the rule they match, so when rules are changed (or removed) we can't just delete them. States get cleaned up when connections are closed or they time out. This means we have to keep old rules around until the states that use them expire.

When rules are removed pf_unlink_rule() adds then to the V_pf_unlinked_rules list (more on that funny V_ prefix later). From time to time the pf purge thread will run over all states and mark the rules that are used by a state. Once that's done for all states we know that all rules that are not marked as in-use can be removed (because none of the states use it).

That can be a lot of work if we've got a lot of states, so pf_purge_thread() breaks that up into smaller chuncks, iterating only part of the state table on every run. Let's look at that code:

void
pf_purge_thread(void *unused __unused)
{
        VNET_ITERATOR_DECL(vnet_iter);
        u_int idx = 0;

        sx_xlock(&pf_end_lock);
        while (pf_end_threads == 0) {
                sx_sleep(pf_purge_thread, &pf_end_lock, 0, "pftm", hz / 10);

                VNET_LIST_RLOCK();
                VNET_FOREACH(vnet_iter) {
                        CURVNET_SET(vnet_iter);


                        /* Wait until V_pf_default_rule is initialized. */
                        if (V_pf_vnet_active == 0) {
                                CURVNET_RESTORE();
                                continue;
                        }

                        /*
                         *  Process 1/interval fraction of the state
                         * table every run.
                         */
                        idx = pf_purge_expired_states(idx, pf_hashmask /
                            (V_pf_default_rule.timeout[PFTM_INTERVAL] * 10));

                        /*
                         * Purge other expired types every
                         * PFTM_INTERVAL seconds.
                         */
                        if (idx == 0) {
                                /*
                                 * Order is important:
                                 * - states and src nodes reference rules
                                 * - states and rules reference kifs
                                 */
                                pf_purge_expired_fragments();
                                pf_purge_expired_src_nodes();
                                pf_purge_unlinked_rules();
                                pfi_kif_purge();
                        }
                        CURVNET_RESTORE();
                }
                VNET_LIST_RUNLOCK();
        }

        pf_end_threads++;
        sx_xunlock(&pf_end_lock);
        kproc_exit(0);
}

We iterate over all of our virtual pf instances (VNET_FOREACH()), check if it's active (for FreeBSD-EN-17.08, where we've seen this code before) and then check the expired states with pf_purge_expired_states(). We start at state 'idx' and only process a certain number (determined by the PFTM_INTERVAL setting) states. The pf_purge_expired_states() function returns a new idx value to tell us how far we got.

So, remember when I mentioned the odd V_ prefix? Those are per-vnet variables. They work a bit like thread-local variables. Each vnet (virtual network stack) keeps its state separate from the others, and the V_ variables use a pointer that's changed whenever we change the currently active vnet (say with CURVNET_SET() or CURVNET_RESTORE()). That's tracked in the 'curvnet' variable. In other words: there are as many V_pf_vnet_active variables as there are vnets: number of vnet jails plus one (for the host system).

Why is that relevant here? Note that idx is not a per-vnet variable, but we handle multiple pf instances here. We run through all of them in fact. That means that we end up checking the first X states in the first vnet, then check the second X states in the second vnet, the third X states in the third and so on and so on.

That of course means that we think we've run through all of the states in a vnet while we really only checked some of them. So when pf_purge_unlinked_rules() runs it can end up free()ing rules that actually are still in use because pf_purge_thread() skipped over the state(s) that actually used the rule.

The problem only happened if we reloaded rules in the host jail, because the active ruleset is never free()d, even if there are no states pointing to the rule.

That explains the panic, and the fix is actually quite straightforward: idx needs to be a per-vnet variable, V_pf_purge_idx, and then the problem is gone.

As is often the case, the solution to a fairly hard problem turns out to be really simple.

September 08, 2017

Here we go with a quick wrap-up of the second day. It started smoothly around 09:00 and was dedicated to more technical talks. After some refill of coffee, I was ready to follow all talks presented in the main track.

It started with LiveOverflow who presented “Play CTF“. CTF games (“Capture The Flag”) are present on the schedule of many infosec conferences but can also be independent events. The idea of the talk was original. It started with a short definition: They are security competitions with challenges that you must solve to find the “flag”. Sometimes, I’m playing CTF games but it’s always a dilemma. If you play, you don’t follow tracks (and I can’t write my wrap-ups!). Playing CTF is a great way to learn to hack. That’s what demonstrated Fabian in his talk. CTF’s are also a great to learn new technologies because it’s always changing. (example: many developers switched from PHP to Node.js). Challenges are usually based on typical vulnerabilities but you must be creative to solve them. They are often weird and do not always reflect the real world. So be prepared to fail :). The second part of the talk was more technical with examples of challenges. I like the one based on an issue present in Python 2 and how it compares objects. The second example was a format string vulnerability and finally a Python sandbox evasion. A very nice talk to start the day! If you’re interesting, you can find many CTF’s on a common agenda on ctftime.org.

The second slot was mine. I presented my overview of webshells from an HTTP protection perspective. Here are my slides:

Then, Ryan Lackey presented “The trouble with updates“. This is a fact, to be better protected against software vulnerabilities, patching is the key! Today, most operating systems and software have automatic update features but it’s not always for the good. Indeed, a few times a year, we read some bad news about a patch that broke a system or makes it less stable. But automatic installation also means that some bad code can be automatically injected into a system. What if the patching process is compromized? There was already several papers released about Microsoft WSUS! Some people also recommend to not install patches automatically. Certainly not on production systems. In this case, a best practice is to deploy the patches on a test system first to ensure that everything runs smoothly.

 

The next presentation was about “The status of web browsers VS DOM fuzzing” by Ivan Fratric (from the Google Project Zero). DOM or “Document Object Model” used in web browsers has been an interesting target for a while. A classic technique to find bugs in software is fuzzing. Ivan’s presentation reviewed how modern browsers are protecting themselves against fuzzing. Ivan explained how he developed his fuzzer and successfully used it to discover a lot of vulnerabilities. And guess what? All major browsers suffered from vulnerabilities.

I really expected a lot of the next talk about AutoIT by Vanja Svajcer from Cisco/Talos: “Hiding malware payloads with AutoIT”. A few days ago, I wrote a diary about AutoIT based malware. Vanja started with a nice introduction about AutoIT. This tool exists for years but seems to be back on stage. It’s a BASIC alike scripting language that can perform GUI automation and testing, can load external UDF (“User Defined Function, …). Of course, like any other languages, the code is never released as is, it is usually heavily obfuscated (variables and functions are renamed, junk code is inserted, strings split, etc…). It is even possible to inject the payload into an existing process. After the introduction, Vanja focused on a specific sample and explained how it infects the victim’s computer.

During the lunch break, I attended the lightning call session. A lot of interesting stuff and, amongst others, a quick presentation of Taler was performed by Sva. Taler is an alternative electronic payment system still under development. If you’re interested, check this website.

There was no talk foreseen in the afternoon, just the closing session and the results of the CTF. We spent the rest of the day chatting around some drinks under a nice weather. Yes, networking is also important during security conferences. This wraps up my first visit to FSEC. This is a nice event and the country looks nice. You can add put it on your wish-list of conferences to attend next year!

[The post FSEC 2017 Wrap-Up Day #2 has been first published on /dev/random]

As part of working in Acquia’s Office of the CTO, I’ve been working on the API-First Initiative for the past year and a half! Where are we at? Find out :)

Preview:

Intro


I have two Philips Hue lamps and mostly like them when  they are in the very warm white (2000k) color temperature setting.
The problem is that the setting does not persist when the lamps are switched using a physical power switch. To solve this I wrote the Hue Persistence program last year.
This program runs on the LAN and basically polls and stores the lamp settings and when it it detects a lamp going from off to on it restores the last setting.

After using it for a few weeks it became clear that there is one big problem with it: the Philips Hue hub sometimes accepts a connection from my program but then does not respond, yet keeps the connection open.
As the program is written using synchronous IO, this causes the program to hang forever, and it doesn't work anymore.

Solution


The Hue Persistence program uses the Rust Philips Hue library which using Hyper as a HTTP client to talk to the Philips Hue hub via it's restful API.
This library still uses Hyper 0.10 which does not provide async IO yet.

Recently there has been a lot of work for asynchronous IO in rust using Tokio and Futures. Version 0.11 of Hyper uses this.

First thing I had to do is port the Rust Philips Hue library to use Hyper 0.11 and thus async IO. This turned out to be quite an undertaking. The result of this can be found on the hyper_0.11 branch of my fork of the library. It is still a bit ugly, but works.

So far I've found using Futures for async IO to have the following issues:

  • error handling can be very confusing; error types need to be the same in future composition and you have to explicitly do error type conversions to make things work
  • future composition results in complex types, returning to Box<Future> helps a bit, but it still can be rather confusing (I didn't try the not yet released impl Trait solution)
  • it's not so obvious which future composition methods to use when, docs are sparse
  • branching (if, ...) inside futures is tedious again because of types having to match perfectly
In the end though I got the satisfaction of a working async library, after which I updated the Hue Persistence app to have a timeout on it's requests. This again turned out to be really tedious.
I had to use select2 with a sleep instead of tokio_timer::Timer::timeout() because it was just impossible to get the types to work for the error types when using tokio_timer::Timer, due to the TimeoutError containing Future type as a containing type in the signature of timeout().

Conclusion


My lamps are behaving fine now!

Async IO in Rust using Futures/tokio is interesting but at this point it still feels like "getting the types right" is somewhat in the way of normal coding productivity.
Some ergonomic improvements/language support could surely be used.
To be revisited in the Future (pun intended ;) ).



September 07, 2017

There are more and more infosec events worldwide and it’s always nice to attend new events and meet new people. This time, it is the case with FSEC. First visit to this security conference organized in Varazdin, Croatia. I had the honor to be invited as a speaker. This is already the seventh edition. FSEC was born thanks to the initiative of Tonimir Kisasondi. The event grew years after years and reached today 470 attendees (they reached the maximum capacity the venue). The conference was kicked off with some words by authorities and academic people. I was also impressed by the number of journalists presents to interview Tonimir! If only, we could have the same interest in Belgium! About the event itself, it is based on three tracks that cover large topics from round tables about GDPR to high technical presentations like browser exploration. Of course, there is also a CTF. The venue is a very nice  old theatre:

FSEC VenueCroatian National Theater, Varazdin

As usual, the event started with some keynotes. The first one was assigned to Alan Duric, the CEO of wire.com. The topic was “E2EE” or end-to-end encryption. Alan started with a review of the messaging tools that people used to communicate over the Internet (the “consumer world”). I liked the comparison of encryption with smoking in presence of children. Twenty years ago, parents were smoking in presence of their kids. Not because they were evil or bad parents, just because they were not aware of the risks associated to the smoke. It’s the same with encryption. In the past, people were not aware that their communications could be intercepted. Today communications are everywhere: not only people communicate but also computers (“M2M” or “Machine to machine”). Business critical data must also be protected, IoT, health data, etc… All of these must be federated with interoperability. Most of the data is stored in the cloud today. How to increase the awareness for E2EE requirements? Let’s imagine a deny of service attack against institutions from cars of video recorders. That’s not science-fiction! What if big cloud players are brought down due to DDoS? Our daily life will be affected. Security must be presented and sold in the right wat. It is driven by behavioral economics. Alan descrive three types of companies that are facing different challenges:

  • To protect intelectual property (risk is industrial espionage)
  • To protect client data (risks is coming with the GDPR)
  • Internal information (sensitive) like political parties (risk is cybercrime)

Again many services rely on service providers. Security and privacy are dependant on them and their practices and ability to fight threats. Interesting question from the audience: what to do when E2EE is not implemented? Do we wait or do we go with risks? Business can’t wait, the most important is to know the risks.

The second keynote speaker was Jaya Baloo, CISO of KPN. How KPN approach security? KPN is an old company (like most telco’s, they have to take into account multiple technologies, some of them being quite old). In 2012, they were hacked and realized that the question that you ask to yourself is not “if” but “when” you’ll be targeted. They provide critical services (emergency, airports, etc) Their mission: keep KPN secure, reliable and trusted. Besides a classic security life cycle (“prevent – detect – respond – verify”), they also have an internal red team responsible to test everything. As said Jaya: “They are there to hack all our sh*t!”, the message looks clear. If something is not secure, the project is simply blocked no matter  of what the business says. But the critical aspect for security professionals is to translate the impact to the business… What will happen if case of successful attack? How much will it cost to the organization? Jaya is known to be transparent and delivers good advice like “If your management relies on compliance, you’re in big sh*t!“. She also showed the security dashboard used by the management. Clear, simple, based on four categories: a “DEFCON” level, a threat intelligence, the vulnerabilities discovered by the red team and incidents. Very nice keynote full of good advices and facts!

After the lunch, I went to a session focussing on TLS: “TLS/SSL Transactions and PCI DSS, 3-D Secure, PSD2 & GDPR” presented by Predrag Kovačević. The idea was to review the status of the different TLS/SSL versions and then to see how to use the properly in different scenario’s: in a PCI/DSS environment, 3D-secure and GDPR. The problem was that the talk not in English… A good advice if you have to implement TLS: OWASP has a great cheat sheet about TLS. For the organizers, if a talk is not in English, it could be interesting to notify it on the schedule.

Then, I switched to another room to follow a presentation of the Sentinel SPS solution. This is a home router (for residential users) that has many interesting features all based on free software. It offers many interesting features: Internet access, multimedia features, IoT, telephone services and the “Sentinel services” that are:

  • A classic firewall
  • Network analysis
  • Hash lookup
  • RHA
  • YARA
  • Format parsing and analysis (check archives)
  • Parental controls
  • Remote management

Then, a talk looked interesting: “HiSiicon DVR hack” by Istvan Toth. Istvan made a very didactic presentation and explained how the firmware used in many DVR boxes was vulnerable. The presentation was easy to understand. It was like an hands-on presentation. He demonstrated how to use different tools to perform reverse engineering tasks (based on IDApro & python scripts or gdbserver). Step by step demonstration how to find the buffer overflow in the firmware. Nice presentation except that Istvan was reading every time his documentation 😉

Then, I followed Ivan Voras who presented “Designing a private blockchain… in case you really need one”. Everybody heard about block chains (mainly due to the BitCoin crypto currency) but not everybody knows how it works (and I’m part of this group). Only 10% of the audience looked to know what blockchain is (from a technical point of view). Ivan made a great introduction to the blockchain technology. Consider a block chain as some kind of database and data are dropped into blocks. In case of crypto currencies, one transactions is a block. Each new block signs the block before it. It’s immutable. But blockchains are not only used by crypto currencies, they’re many usages. Ivan developed his own private blockchain implementation that is a private blockchain. Each block is a SQLite database. The goal is to distribute official data in a safe way. His project is called Daisy and is available here.

Then, “Challenges and pitfalls of modern deep learning computing vision models” was presented by Marko Velic & Enes Deumic. Machine learning is a hot topic today and is used everywhere. We can find such technologies in security and surveillance, fraud detection, drones, self-driving cars, … After an introduction, they explained the risks that machine learning might face. Examples were based on self-driving cars. They demonstrated how the detection rate of road signs can be affected by covering some signs with stickers etc… Interesting. Again about machine learning, tgrzinic presented his machine learning framework to analyze malware samples. The framework is called MaliO and based on many free tools like Cuckoo.

The talk “Targeted attacks in 2017” by Felix Aimé.There were so many events that it’s not possible to remember all of them. Felix made a review of the last attacks and the techniques used to compromise the targets. They don’t change often and remains based on:

  • Compromised installation tools
  • Webmail phishing
  • Compromization of suppliers
  • Frontal compromise of servers
  • Classic spear phishing

sva came on stage to speak about PEP or “Pretty Easy Privacy”. I saw her presentation at the RMLL in July. The goal of PEP is got help the user to get rid of the complexity of PGP or S/MIME who are known to not be the best tools in terms of usability.

Finally, the last talk of the day (not the least one) was presented by Arnim Eijkhoudt from KPN about threat intelligence: “From incident to threat response“. I also gave the same message as Felix: “Sharing is caring”. Threat intelligence is:

  • Not a phase
  • Formalization with some caveats
  • Based on standards (STIX, TAXII, Cybox, )
  • Tons of players/feed/tools. All vendors have a feed
  • “Hype” (which could be bad)

The goal is to find the threat common to multiple incidents, to include a risk assessment component. It must be actionable and provides input for hunting. It must, of course, be a continuous process. Arnim explained different aspects of threat intelligence and finished with a real spear phishing case that targeted KPN (but without disclosing too much details ;-). A last advice: do not automatically classify  phishing emails as not malicious or to not submit they to VT. Perform a minimum set of controls to be certain that it’s not a targeted attack!

The day ended with a party but the music was way too loud, I had a dinner outside in a relaxed atmosphere with a friend. See you tomorrow for the next wrap-up!

[The post FSEC 2017 Wrap-Up Day #1 has been first published on /dev/random]

September 06, 2017

Just a quick blog post about an interesting sample that I found today. Usually, modern pieces of malware implement anti-debugging and anti-VM techniques. They perform some checks against the target and when a positive result is found, they silently exit… Such checks might be testing the screen resolution, the activity of a connected user, the presence of files on the desktop, etc. But they also search for interesting processes that could reveal that they are being monitored or debugged. This is achieved via the GetProcessesByName system call. Example:

processName = "tool_executed_by_analyst"
processList = Process.GetProcessesByName(processName)
If processList.Count > 0 Then
    ' Process is running, exit silently...
Else
    ' Process is not running, do our malicious stuff...
End If

This time, the sample did not search for running processes. Instead is a stealthy exit, it just executed a long list of taskkill.exe commands with process names like this:

taskkill.exe /IM <string> /T /F

“/IM” refers to the process image name, “/T” means to terminate all child processes and “/F” means to kill the process forcefully. This is a quite agressive technique!

Some processes are well-known, others were more exotic. Here is the full list:

avpmapp.exe
econceal.exe
escanmon.exe
escanpro.exe
TRAYSSER.EXE
TRAYICOS.EXE
econser.exe
VIEWTCP.EXE
FSHDLL64.exe
fsgk32.exe
fshoster32.exe
FSMA32.EXE
fsorsp.exe
fssm32.exe
FSM32.EXE
trigger.exe
FProtTray.exe
FPWin.exe
FPAVServer.exe
AVK.exe
GdBgInx64.exe
AVKProxy.exe
GDScan.exe
AVKWCtlx64.exe
AVKService.exe
AVKTray.exe
GDKBFltExe32.exe
GDSC.exe
virusutilities.exe
guardxservice.exe
guardxkickoff_x64.exe
iptray.exe
freshclam.exe
freshclamwrap.exe
K7RTScan.exe
K7FWSrvc.exe
K7PSSrvc.exe
K7EmlPxy.EXE
K7TSecurity.exe
K7AVScan.exe
K7CrvSvc.exe
K7SysMon.Exe
K7TSMain.exe
K7TSMngr.exe
nanosvc.exe
nanoav.exe
nnf.exe
nvcsvc.exe
nbrowser.exe
nseupdatesvc.exe
nfservice.exe
cmd.exetaskkill/IMnwscmon.exe
njeeves2.exe
nvcod.exe
nvoy.exe
zlhh.exe
Zlh.exe
nprosec.exe
Zanda.exe
NS.exe
acs.exe
op_mon.exe
PSANHost.exe
PSUAMain.exe
PSUAService.exe
AgentSvc.exe
BDSSVC.EXE
EMLPROXY.EXE
OPSSVC.EXE
ONLINENT.EXE
QUHLPSVC.EXE
SAPISSVC.EXE
SCANNER.EXE
SCANWSCS.EXE
scproxysrv.exe
ScSecSvc.exe
SUPERAntiSpyware.exe
SASCore64.exe
SSUpdate64.exe
SUPERDelete.exe
SASTask.exe
K7RTScan.exe
K7FWSrvc.exe
K7PSSrvc.exe
K7EmlPxy.EXE
K7TSecurity.exe
K7AVScan.exe
K7CrvSvc.exe
K7SysMon.Exe
K7TSMain.exe
K7TSMngr.exe
uiWinMgr.exe
uiWatchDog.exe
uiSeAgnt.exe
PtWatchDog.exe
PtSvcHost.exe
PtSessionAgent.exe
coreFrameworkHost.exe
coreServiceShell.exe
uiUpdateTray.exe
VIPREUI.exe
SBAMSvc.exe
SBAMTray.exe
SBPIMSvc.exe
bavhm.exe
BavSvc.exe
BavTray.exe
Bav.exe
BavWebClient.exe
BavUpdater.exe
MCShieldCCC.exe
MCShieldRTM.exe
MCShieldDS.exe
MCS-Uninstall.exe
SDScan.exe
SDFSSvc.exe
SDWelcome.exe
SDTray.exe
UnThreat.exe
utsvc.exe
FortiClient.exe
fcappdb.exe
FCDBlog.exe
FCHelper64.exe
fmon.exe
FortiESNAC.exe
FortiProxy.exe
FortiSSLVPNdaemon.exe
FortiTray.exe
FortiFW.exe
FortiClient_Diagnostic_Tool.exe
av_task.exe
CertReg.exe
FilMsg.exe
FilUp.exe
filwscc.exe
filwscc.exe
psview.exe
quamgr.exe
quamgr.exe
schmgr.exe
schmgr.exe
twsscan.exe
twssrv.exe
UserReg.exe

[The post Interesting List of Windows Processes Killed by Malicious Software has been first published on /dev/random]

Ever since I saw Magnolia I’ve been a fan of Aimee Mann. Not that I know all her songs, not that I listen to her music that regularly, but every time I do I regret having waited that long.

Anyway, she’s still going strong with high-quality songwriting about the not-so-pleasant aspects of life. This is her in the KEXP studios in July 2017, enjoy!

YouTube Video
Watch this video on YouTube.

DRY, standing for Don’t Repeat Yourself, is a well-known design principle in the software development world.

It is not uncommon for removal of duplication to take center stage via mantras such as “Repetition is the root of all evil”. Yet while duplication is often bad, the well intended pursuit of DRY often leads people astray. To see why, let’s take a step back and look at what we want to achieve by removing duplication.

The Goal of Software

First and foremost, software exists to fulfill a purpose. Your client, which can be your employer, is paying money because they want the software to provide value. As a developer it is your job to provide this value as effectively as possible. This includes tasks beyond writing code to do whatever your client specifies, and might best be done by not writing any code. The creation of code is expensive. Maintenance of code and extension of legacy code is even more so.

Since creation and maintenance of software is expensive, the quality of a developers work (when just looking at the code) can be measured in how quickly functionality is delivered in a satisfactory manner, and how easy to maintain and extend the system is afterwards. Many design discussions arise about trade-offs between those two measures. The DRY principle mainly situates itself in the latter category: reducing maintenance costs. Unfortunately applying DRY blindly often leads to increased maintenance costs.

The Good Side of DRY

So how does DRY help us reduce maintenance costs? If code is duplicated, and it needs to be changed, you will need to find all places where it is duplicated and apply the change. This is (obviously) more difficult than modifying one place, and more error prone. You can forget about one place where the change needs to be applied, you can accidentally apply it differently in one location, or you can modify code that happens to the same at present but should nevertheless not be changed due to conceptual differences (more on this later). This is also known as Shotgun Surgery. Duplicated code tends to also obscure the structure and intent of your code, making it harder to understand and modify. And finally, it conveys a sense of carelessness and lack of responsibility, which begets more carelessness.

Everyone that has been in the industry for a little while has come across horrid procedural code, or perhaps pretend-OO code, where copy-paste was apparently the favorite hammer of its creators. Such programmers indeed should heed DRY, cause what they are producing suffers from the issues we just went over. So where is The Fallacy of DRY?

The Fallacy of DRY

Since removal of duplication is a means towards more maintainable code, we should only remove duplication if that removal makes the code more maintainable.

If you are reading this, presumably you are not a copy-and-paste programmer. Almost no one I ever worked with is. Once you know how to create well designed OO applications (ie by knowing the SOLID principles), are writing tests, etc, the code you create will be very different from the work of a copy-paste-programmer. Even when adhering to the SOLID principles (to the extend that it makes sense) there might still be duplication that should be removed.The catch here is that this duplication will be mixed together with duplication that should stay, since removing it makes the code less maintainable. Hence trying to remove all duplication is likely to be counter productive.

Costs of Unification

How can removing duplication make code less maintainable? If the costs of unification outweigh the costs of duplication, then we should stick with duplication. We’ve already gone over some of the costs of duplication, such as the need for Shotgun Surgery. So let’s now have a look at the costs of unification.

The first cost is added complexity. If you have two classes with a little bit of common code, you can extract this common code into a service, or if you are a masochist extract it into a base class. In both cases you got rid of the duplication by introducing a new class. While doing this you might reduce the total complexity by not having the duplication, and such extracting might make sense in the first place for instance to avoid a Single Responsibility Principle violation. Still, if the only reason for the extraction is reducing duplication, ask yourself if you are reducing the overall complexity or adding to it.

Another cost is coupling. If you have two classes with some common code, they can be fully independent. If you extract the common code into a service, both classes will now depend upon this service. This means that if you make a change to the service, you will need to pay attention to both classes using the service, and make sure they do not break. This is especially a problem if the service ends up being extended to do more things, though that is more of a SOLID issue. I’ll skip going of the results of code reuse via inheritance to avoid suicidal (or homicidal) thoughts in myself and my readers.

DRY = Coupling

— A slide at DDDEU 2017

The coupling increases the need for communication. This is especially true in the large, when talking about unifying code between components or application, and when different teams end up depending on the same shared code. In such a situation it becomes very important that it is clear to everyone what exactly is expected from a piece of code, and making changes is often slow and costly due to the communication needed to make sure they work for everyone.

Another result of unification is that code can no longer evolve separately. If we have our two classes with some common code, and in the first a small behavior change is needed in this code, this change is easy to make. If you are dealing with a common service, you might do something such as adding a flag. That might even be the best thing to do, though it is likely to be harmful design wise. Either way, you start down the path of corrupting your service, which now turned into a frog in a pot of water that is being heated. If you unified your code, this is another point at which to ask yourself if that is still the best trade-off, or if some duplication might be easier to maintain.

You might be able to represent two different concepts with the same bit of code. This is problematic not only because different concepts need to be able to evolve individually, it’s also misleading to have only a single representation in the code, which effectively hides that you are dealing with two different concepts. This is another point that gains importance the bigger the scope of reuse. Domain Driven Design has a strategic pattern called Bounded Contexts, which is about the separation of code that represents different (sub)domains. Generally speaking it is good to avoid sharing code between Bounded Contexts. You can find a concrete example of using the same code for two different concepts in my blog post on Implementing the Clean Architecture, in the section “Lesson learned: bounded contexts”.

DRY is for one Bounded Context

— Eric Evans

Conclusion

Duplication itself does not matter. We care about code being easy (cheap) to modify without introducing regressions. Therefore we want simple code that is easy to understand. Pursuing removal of duplication as an end-goal rather than looking at the costs and benefits tends to result in a more complex codebase, with higher coupling, higher communication needs, inferior design and misleading code.

September 03, 2017

The post 2 interesting things happened in last cron.weekly’s newsletter appeared first on ma.ttias.be.

As usual, I sent out my weekly Linux & open source newsletter this Sunday. Lots of good content in there, you should probably have a read; cron.weekly issue #96: LogDevice, qmail, redis, Linus, HAProxy, libraries, concert, restic & more .

But as it was sent, I received a fair amount of feedback regarding 2 links I shared in the newsletter.

concert has been deprecated

I learned about concert, a tool for managing certificates via Let's Encrypt, via a colleague playing with Minio, the open source S3 compatible server.

I wrote the newsletter on Saturday, and by Sunday morning, this commit had landed;

+***DEPRECATED -- This project is deprecated and not maintained anymore.***
+***It is recommended all users use https://certbot.eff.org/ instead.***
+
Deprecate concert project (#27)

Interesting: between me discovering the project & adding it to the newsletter, it got deprecated.

Things move fast in open source.

`publicfile` HTTP server does not support "?querystring" arguments

Ok, quick summary: I linked to an article about qmail's security guarantee & bounty program, which hasn't been breached since as early as 1997. That's impressive.

So naturally, I wanted to include it in the newsletter. By default, I translate all links a bit, to get better tracking of the source/medium through Google Analytics. It still keeps the URLs readable -- as opposed to an actual click tracker, which would scramble all links. It transforms them like this;

Pretty harmless, usually. I added some logic that I don't try to add query strings to an URL if it already contains any, as not to accidentally break any functionality.

However, if you check the links above: the first one works, the second does not. I didn't see that one coming.

Turns out, that blog/page is hosted on a webserver called 'publicfile', created by D. J. Bernstein, who also created qmail & a lot of articles ultimately defining the internet as we know it today.

However, the webserver doesn't support query string arguments.

*Queue the security vs. flexibility/usability debate*

Lessons learned

A couple of gotcha's;

  1. Maybe visit each link again right before it goes out
  2. At least test the final new URLs it they still work

Those are things I did at the very beginning of the newsletter, but after a while it becomes routine and you start to take things for granted. That's when things get interesting!

The post 2 interesting things happened in last cron.weekly’s newsletter appeared first on ma.ttias.be.

September 02, 2017

Sucuri writes about their pov on performance:

We focus on useful metrics to optimize speed, like total time, not first byte or server response time.

Whereas Neil Patel writes:

When you’re in the weeds of optimizing your site speed, play smart. Instead of trying hard to push down your document complete, start rendering, or fully loaded page load time metrics (all of which are important), focus on the highest-impact metric first: TTFB.

Where TTFB = time to first byte. Needless to say I’m with Neil on this!

Allan twisted my arm in BSDNow episode 209 so I'll have to talk about FreeBSD-EN-17:08.pf.

First things first, so I have to point out that I think Allan misremembered things. The heroic debugging story is PR 219251, which I'll try to write about later.

FreeBSD-EN-17:08.pf is an issue that affected some FreeBSD 11.x systems, where FreeBSD would panic at startup. There were no reports for CURRENT. That looked like this:

Crash information

There's very little to go on here, but we do now the cause of the panic ("integer divide fault"), and that the current process was "pf purge". The pf purge thread is part of the pf housekeeping infrastructure. It's a housekeeping kernel thread which cleans up things like old states and expired fragments.

Here's the core loop:
void
pf_purge_thread(void *unused __unused)
{
        VNET_ITERATOR_DECL(vnet_iter);
        u_int idx = 0;

        sx_xlock(&pf_end_lock);
        while (pf_end_threads == 0) {
                sx_sleep(pf_purge_thread, &pf_end_lock, 0, "pftm", hz / 10);

                VNET_LIST_RLOCK();
                VNET_FOREACH(vnet_iter) {
                        CURVNET_SET(vnet_iter);

                        /*
                         *  Process 1/interval fraction of the state
                         * table every run.
                         */
                        idx = pf_purge_expired_states(idx, pf_hashmask /
                            (V_pf_default_rule.timeout[PFTM_INTERVAL] * 10));

                        /*
                         * Purge other expired types every
                         * PFTM_INTERVAL seconds.
                         */
                        if (idx == 0) {
                                /*
                                 * Order is important:
                                 * - states and src nodes reference rules
                                 * - states and rules reference kifs
                                 */
                                pf_purge_expired_fragments();
                                pf_purge_expired_src_nodes();
                                pf_purge_unlinked_rules();
                                pfi_kif_purge();
                        }
                        CURVNET_RESTORE();
                }
                VNET_LIST_RUNLOCK();
        }

        pf_end_threads++;
        sx_xunlock(&pf_end_lock);
        kproc_exit(0);
}

The lack of mention of pf functions in the backtrace is a hint unto itself. It suggests that the error is probably directly in pf_purge_thread(). It might also be in one of the static functions it calls, because compilers often just inline those so they don't generate stack frames.

Remember that the problem is an "integer divide fault". How can integer divisions be a problem? Well, you can try to divide by zero. The most obvious suspect for this is this code:

	idx = pf_purge_expired_states(idx, pf_hashmask /
		(V_pf_default_rule.timeout[PFTM_INTERVAL] * 10));

However, this variable is both correctly initialised (in pfattach_vnet()) and can only be modified through the DIOCSETTIMEOUT ioctl() call and that one checks for zero.

At that point I had no idea how this could happen, but because the problem did not affect CURRENT I looked at the commit history and found this commit from Luiz Otavio O Souza:

	Do not run the pf purge thread while the VNET variables are not
	initialized, this can cause a divide by zero (if the VNET initialization
	takes to long to complete).

	Obtained from:	pfSense
	Sponsored by:	Rubicon Communications, LLC (Netgate)

That sounds very familiar, and indeed, applying the patch fixed the problem. Luiz explained it well: it's possible to use V_pf_default_rule.timeout before it's initialised, which caused this panic.

To me, this reaffirms the importance of writing good commit messages: because Luiz mentioned both the pf purge thread and the division by zero I was easily able to find the relevant commit. If I hadn't found it this fix would have taken a lot longer.

As I've blogged before, I've been on and off working on SReview, a video review system. Development over the past year has been mostly driven by the need to have something up and running for first FOSDEM 2017, and then DebConf17, and so I've cut corners left and right which made the system, while functional, not quite entirely perfect everywhere. For instance, the backend scripts were done in ad-hoc perl, each reinventing their own wheel. Doing so made it easier for me to experiment with things and figure out where I want them to go, without immediately creating a lot of baggage that is not necessarily something I want to be stuck to. This flexibility has already paid off, in that I've redone the state machine between FOSDEM and DebConf17—and all it needed was to update a few SQL statements here and there. Well, and add a few of them, too.

It was always the intent to replace most of the ad-hoc perl with something better, however, once the time was ripe. One place where historical baggage is not so much of a problem, and where in fact abstracting away the complexity would now be an asset, is in the area of FFmpeg command lines. Currently, these are built by simple string expansion. For example, we do something like this (shortened for brevity):

system("ffmpeg -y -i $outputdir/$slug.ts -pass 1 -passlogfile ...");

inside an environment where the $outputdir and $slug variables are set in a perl environment. That works, but it has some downsides; e.g., adding or removing options based on which codecs we're using is not so easy. It would be much more flexible if the command lines were generated dynamically based on requested output bandwidth and codecs, rather than that they be hardcoded in the file. Case in point: currently there are multiple versions of some of the backend scripts, that only differ in details—mostly the chosen codec on the ffmpeg command line. Obviously this is suboptimal.

Instead, we want a way where video file formats can be autodetected, so that I can just say "create a file that uses encoder etc settings of this other file here". In addition, we also want a way where we can say "create a file that uses encoder etc settings of this other file here, except for these one or two options that I want to fine-tune manually". When I first thought about doing that about a year ago, that seemed complicated and not worth it—or at least not to that extent.

Enter Moose.

The Moose OO system for Perl 5 is an interesting way to do object orientation in Perl. I knew Perl supports OO, and I had heard about Moose, but never had looked into it, mostly because the standard perl OO features were "good enough". Until now.

Moose has a concept of adding 'attributes' to objects. Attributes can be set at object construction time, or can be accessed later on by way of getter/setter functions, or even simply functions named after the attribute itself (the default). For more complicated attributes, where the value may not be known until some time after the object has been created, Moose borrows the concept of "lazy" variables from Perl 6:

package Object;

use Moose;

has 'time' => (
    is => 'rw',
    builder => 'read_time',
    lazy => 1,
);

sub read_time {
    return localtime();
}

The above object has an attribute 'time', which will not have a value initially. However, upon first read, the 'localtime()' function will be called, the result is cached, and then (and on all further calls of the same function), the cached result will be returned. In addition, since the attribute is read/write, the time can also be written to. In that case, any cached value that may exist will be overwritten, and if no cached value exists yet, the read_time function will never be called. (it is also possible to clear values if needs be, so that the function would be called again).

We use this with the following pattern:

package SReview::Video;

use Moose;

has 'url' => (
    is => 'rw',
)

has 'video_codec' => (
    is => 'rw',
    builder => '_probe_videocodec',
    lazy => 1,
);

has 'videodata' => (
    is => 'bare',
    reader => '_get_videodata',
    builder => '_probe_videodata',
    lazy => 1,
);

has 'probedata' => (
    is => 'bare',
    reader => '_get_probedata',
    builder => '_probe',
    lazy => 1,
);

sub _probe_videocodec {
    my $self = shift;
    return $self->_get_videodata->{codec_name};
}

sub _probe_videodata {
    my $self = shift;
    if(!exists($self->_get_probedata->{streams})) {
        return {};
    }
    foreach my $stream(@{$self->_get_probedata->{streams}}) {
        if($stream->{codec_type} eq "video") {
            return $stream;
        }
    }
    return {};
}

sub _probe {
    my $self = shift;

    open JSON, "ffprobe -print_format json -show_format -show_streams " . $self->url . "|"
    my $json = "";
    while(<JSON>) {
        $json .= $_;
    }
    close JSON;
    return decode_json($json);
}

The videodata and probedata attributes are internal-use only attributes, and are therefore of the 'bare' type—that is, they cannot be read nor written to. However, we do add 'reader' functions that can be used from inside the object, so that the object itself can access them. These reader functions are generated, so they're not part of the object source. The probedata attribute's builder simply calls ffprobe with the right command-line arguments to retrieve data in JSON format, and then decodes that JSON file.

Since the passed JSON file contains an array with (at least) two streams—one for video and one for audio—and since the ordering of those streams depends on the file and is therefore not guaranteed, we have to loop over them. Since doing so in each and every attribute of the file we might be interested in would be tedious, we add a videodata attribute that just returns the data for the first found video stream (the actual source also contains a similar one for audio streams).

So, if you create an SReview::Video object and you pass it a filename in the url attribute, and then immediately run print $object->video_codec, then the object will

  • call ffprobe, and cache the (decoded) output for further use
  • from that, extract the video stream data, and cache that for further use
  • from that, extract the name of the used codec, cache it, and then return that name to the caller.

However, if the caller first calls $object->video_codec('h264'), then the ffprobe and most of the caching will be skipped, and instead the h265 data will be returned as video codec name.

Okay, so with a reasonably small amount of code, we now have a bunch of attributes that have defaults based on actual files but can be overwritten when necessary. Useful, right? Well, you might also want to care about the fact that sometimes you want to generate a video file that uses the same codec settings of this other file here. That's easy. First, we add another attribute:

has 'reference' => (
    is => 'ro',
    isa => 'SReview::Video',
    predicate => 'has_reference'
);

which we then use in the _probe method like so:

sub _probe {
    my $self = shift;

    if($self->has_reference) {
        return $self->reference->_get_probedata;
    }
    # original code remains here
}

With that, we can create an object like so:

my $video = SReview::Video->new(url => 'file.ts');
my $generated = SReview::Video->new(url => 'file2.ts', reference => $video);

now if we ask the $generated object what the value of its video_codec setting is without telling it ourselves first, it will use the $video object for its probed data, and use that.

That only misses generating the ffmpeg command line, but that's all fairly straightforward and therefore left as an exercise to the reader. Or you can cheat, and look it up.

We now invite proposals for main track presentations, developer rooms, stands and lightning talks. FOSDEM offers open source and free software developers a place to meet, share ideas and collaborate. Renowned for being highly developer-oriented, the event brings together some 8000+ geeks from all over the world. The eighteenth edition will take place on Saturday 3rd and Sunday 4th February 2018 at the usual location: ULB Campus Solbosch in Brussels. We will record and stream all main tracks, devrooms and lightning talks live. The recordings will be published under the same licence as all FOSDEM content (CC-BY). If, exceptionally,舰

I published the following diary on isc.sans.org: “AutoIT based malware back in the wild“.

One week ago I wrote a diary with an analysis of a malicious RAR archive that contained an AutoIT script. The technique was not new but I was curious to see if this was a one-shot or not. To search for juicy samples, VirusTotal Intelligence or “VTI” is a nice source. Thanks to the “Retro Hunt” feature, it is possible to search for specific samples that were submitted. The search conditions are based on YARA rules… [Read more]

[The post [SANS ISC] AutoIT based malware back in the wild has been first published on /dev/random]

August 31, 2017

Logo Proxmox VECe jeudi 21 septembre 2017 à 19h se déroulera la 61ème séance montoise des Jeudis du Libre de Belgique.

Le sujet de cette séance : Présentation de Proxmox VE

Thématique : sysadmin|virtualisation|stockage

Public : sysadmin|entreprises|étudiants

L’animateur conférencier : Alexandre Derumier (Odiso)

Lieu de cette séance : Université de Mons, Faculté Polytechnique, Site Houdain, Rue de Houdain, 9, auditoire 3 (cf. ce plan sur le site de l’UMONS, ou la carte OSM). Entrée par la porte principale, au fond de la cour d’honneur. Suivre le fléchage à partir de là.

La participation sera gratuite et ne nécessitera que votre inscription nominative, de préférence préalable, ou à l’entrée de la séance. Merci d’indiquer votre intention en vous inscrivant via la page http://jeudisdulibre.fikket.com/. La séance sera suivie d’un verre de l’amitié (le tout sera terminé au plus tard à 22h).

Les Jeudis du Libre à Mons bénéficient aussi du soutien de nos partenaires : CETIC, OpenSides, MeaWeb et Phonoid.

Si vous êtes intéressé(e) par ce cycle mensuel, n’hésitez pas à consulter l’agenda et à vous inscrire sur la liste de diffusion afin de recevoir systématiquement les annonces.

Pour rappel, les Jeudis du Libre se veulent des espaces d’échanges autour de thématiques des Logiciels Libres. Les rencontres montoises se déroulent chaque troisième jeudi du mois, et sont organisées dans des locaux et en collaboration avec des Hautes Écoles et Facultés Universitaires montoises impliquées dans les formations d’informaticiens (UMONS, HEH et Condorcet), et avec le concours de l’A.S.B.L. LoLiGrUB, active dans la promotion des logiciels libres.

Description : Proxmox VE est une distribution Linux qui permet, grâce à des outils libres comme Qemu-kvm et les conteneurs LXC, de mettre en place une plateforme de virtualisation bare-metal avec le clustering et la haute disponibilité. Proxmox VE offre diverses solutions de réseaux et stockage comme le LVM, NFS, iSCSI, Ceph, ZFS, …

La présentation abordera la creation et gestion de machine virtuelles et containers, la gestion du réseau et firewall, la migration à chaud des machines et stockage, le hotplug des périphériques (cpu,ram,disk) des machines, la gestion du stockage (snapshot, clones, backup),la gestion du cluster et de la Haute dispo. Je présenterais également ceph, une solution de stockage cluster, repliqué, qui peux être embarqué sur proxmox.

Une démonstration live sera faite sur les différents points abordés.

short bio : Ingénieur système et réseaux depuis près de 13 ans, Manager de l’equipe infrastructure chez Odiso, la cellule hosting de la société M6. je suis un passionné du logiciel Libre, contributeur sur proxmox depuis 2009 et formateur proxmox pour la France. Nous faisons tourner en production sur Proxmox, +- 3000 vms et containers.

Entertainment tonight

Entertainment Tonight is the number one entertainment news magazine in the world, and has been on the air for over 30 years. Fans around the world rely on Entertainment Tonight to receive news and updates on their favorite celebrities and stars. I recently discovered that the newest star featured on Entertainment Tonight was Drupal 8!

Entertainment Tonight's new Drupal 8 website, ETOnline.com, receives 19 million monthly unique visitors, making it the second most visited entertainment news website.

Chapter Three helped build the site. This project really helped Drupal 8 move forward because they ported many modules to Drupal 8 in the process. Check it out at http://www.etonline.com!

Entertainment tonight

Entertainment Tonight is the number one entertainment news magazine in the world, and has been on the air for over 30 years. Fans around the world rely on Entertainment Tonight to receive news and updates on their favorite celebrities and stars. I recently discovered that the newest star featured on Entertainment Tonight was Drupal 8!

Entertainment Tonight's new Drupal 8 website, ETOnline.com, receives 19 million monthly unique visitors, making it the second most visited entertainment news website.

Chapter Three helped build the site. This project really helped Drupal 8 move forward because they ported many modules to Drupal 8 in the process. Check it out at http://www.etonline.com!

August 30, 2017

Feedback we received after DebConf17 was that the quality of the released videos was rather low. Since I'd been responsible for encoding the videos, I investigated that, and found that I had used the video bitrate defaults of ffmpeg, which are rather low. So, I read some more documentation, and came up with better settings for the encoding to be done properly.

While at it, I found that the reason that VP9 encoding takes forever, as I'd found before, has everything to do with the same low-bitrate defaults, and is not an inherent problem to VP9 encoding. In light of that, I ran a test encode of a video with VP8 as well as VP9 encoding, and found that it shaves off some of the file size for about the same quality, with little to no increase in encoder time.

Unfortunately, the initial VP9 encoder settings that I used produced files that would play in some media players and in some browsers, but not in all of them. A quick email to the WebM-discuss mailinglist got me a reply, explaining that since our cameras used YUV422 encoding, which VP9 supports but some media players do not, and that since VP8 doesn't support that format, the result was VP9 files with YUV422 encoding, and VP8 ones with YUV420. The fix was to add a single extra command-line parameter to force the pixel format to YUV420 and tell ffmpeg to also downsample the video to that.

After adding another few parameters so as to ensure that some metadata would be added to the files as well, I've now told the system to re-encode the whole corpus of DebConf17 videos in both VP8 as well as VP9. It would appear that the VP9 files are, on average, about 1/4th to 1/3rd smaller than the equivalent VP8 files, with no apparent loss in quality.

Since some people might like them for whatever reason, I've not dropped the old lower-quality VP8 files, but instead moved them to an "lq" directory. They are usually about half the size of the VP9 files, but the quality is pretty terrible.

TLDR: if you downloaded videos from the debconf video archive, you may want to check whether it has been re-encoded yet (which would be the case if the state on that page is "done"), and if so, download them anew.